且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

合并cron作业以减少实例数量

更新时间:2022-02-26 08:54:05

是的,时间表重叠的cron作业可能会导致不必要的活动高峰,从而增加实例时间.在分钟开始时,不仅每分钟平均有3个请求,而且应用程序的请求队列中同时有3个请求 .如果您使用自动/基本缩放配置,则GAE可能会决定生成其他实例以保持较低的请求延迟,即使单个实例可以轻松地平均每分钟处理3个请求.

Yes, cron jobs with overlapping schedules can cause potentially unnecessary peaks of activity which can drive up the instance hours. Not only that you'd have an average of 3 requests per minute, you'd have 3 requests in your app's request queue simultaneously, at the beginning of that minute. If you use automatic/basic scaling configs GAE may decide to spawn additional instances to keep the request latency low, even if a single instance could easily handle the average of 3 requests per minute.

侧面说明:您已经收到cron请求来触发作业,您有点不必要地创建了另一个(任务队列)请求来实际执行作业(因此,在您的情况下,每分钟6个请求),为什么不直接从cron处理程序中执行作业呢?

Side note: you already have the cron request coming in to trigger the job, you're kinda unnecessarily creating yet another (task queue) request to actually execute the job (thus amounting to 6 requests per minute in your case), why not just execute the job directly from the cron handler?

现在,如果每个请求需要大约1分钟才能完成交错,将无济于事-@MatrixTai的评论适用.

Now, if each of the requests takes close to ~1 minute to complete staggering them won't help much - @MatrixTai's comment applies.

但是,如果所有3个作业的总执行时间少于1分钟,则错开它们将有助于降低成本.在这种情况下,也许更好的方法是不要再创建3个请求-缺点是您仍然需要仔细配置延迟以确保它们不重叠,而只是顺序执行这3个作业,而不会使新请求排队(这就是我的工作):

But if the combined execution time of all 3 jobs is below 1 minute staggering them will help keep the costs down. Maybe a better approach in this case would be to not create 3 more requests - which has the disadvantage that you still need to carefully configure the delays to ensure they don't overlap, but simply sequentially execute the 3 jobs, without enqueueing new requests (this is what I do):

@app.route('/per_minute_jobs/')
def handle_per_minute_jobs():  # OK if the combined execution time is < 1 min
     execute_job_1()
     execute_job_2()
     execute_job_3()

使用这种方法,您每分钟恰好有1个请求,就其本身而言,它不能像其他方法那样触发生成新实例.

With this approach you have exactly 1 request per minute, which, by itself, can't trigger spawning new instances as the other approaches did.

现在,合并的作业执行可能比典型的(非cron)请求处理花费更长的时间.如果您在同一个GAE服务中同时处理这两个实例,那么即使您的应用程序总体上的平均流量很低,以至于显然仅用一个实例即可处理它们,您仍将至少运行2个实例

Now the combined job execution could take significantly longer than your typical (non-cron) request handling. If you handle them both in the same GAE service, you'll still end up with at least 2 instances running, even if overall your app has low enough average traffic that could apparently be handled with just one instance.

更新:

好吧,由于您必须将工作拆分为后续任务,因此只需使用deferred.defer()的可选countdowneta args来及时错开这些任务并平滑那些活动高峰.

OK, since you have to split the work in subsequent tasks, just use the deferred.defer()'s optional countdown or eta args to stagger those tasks in time and smooth those activity peaks.