每当我在运行supervisor和芹菜的ubuntu实例上运行“sudo shutdown-h now”时,芹菜似乎只是在处理任务的过程中关闭,当查看尾部-f时,我看不到当您在for中运行芹菜时键入ctrl-c时通常会看到的“warm shutdown”消息。艾格劳德。
这对我来说是个大问题,因为服务器随时都可能被终止,所以等待芹菜完成它的任务是至关重要的。这可能需要几秒钟,10分钟或10小时,但关闭信号应该总是让芹菜完成它的工作。
我已经在我的芹菜配置文件中添加了stopasgroup和killasgroup,看起来从supervisor停止芹菜会停止所有子进程,但是不知怎么的,它似乎忽略了“sudo shutdown-h now”发送的关闭信号。
有人知道为什么芹菜不会像平时那样“温关机”吗?
下面是一个worker conf的示例:
[program:celery-worker]
command=/path/to/script-running-celery.sh
directory=/path/to/celery/project/tasks
user=myuser
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 60
killasgroup=true
stopasgroup=true
priority=50
下面是正在运行的实际shell脚本:
#!/bin/bash
export PATH="$PATH:/mylocal-envs/localenv";
cd /path/to/celery/project/tasks;
exec /mylocal-envs/localenv/bin/celery worker -c8 -n my_worker_%h_$(uuid) --without-gossip -A tasks --queues=myqueue --loglevel=INFO -Ofair
你能理解为什么一个关闭命令不能让芹菜在温暖的关闭状态下运行并结束它当前的任务吗?
编辑:只需注意,当不使用supervisor直接运行我的脚本并从另一个ssh窗口关闭时,我仍然会得到相同的行为:工作进程正在处理任务,然后接收关闭信号并关闭ssh会话。以下是运行脚本时从工作程序前台发生的情况:
[2014-05-17 19:29:02,955: INFO/MainProcess] Task tasks.a_task[5143a89d-e077-4145-ba94-0932d0f81d9e] succeeded in 15.523131618s: u''
[2014-05-17 19:29:03,621: INFO/MainProcess] Received task: tasks.a_task[b61a30ba-2fbb-475a-9cb2-ec88d20a8885]
Broadcast message from [email protected]
(/dev/pts/1) at 19:29 ...
The system is going down for halt NOW!
[2014-05-17 19:29:06,846: INFO/MainProcess] Received task: tasks.a_task[9146f9f4-f06c-4563-b8d1-43c8d9367d9b]
[2014-05-17 19:29:08,089: INFO/MainProcess] Task tasks.a_task[b4c7dacc-3f6d-42e5-ab67-f33748f0af14] succeeded in 16.49714411s: u''
Connection to xx.xx.xxx.xx closed by remote host.
Connection to xx.xx.xxx.xx closed.
您仍然可以看到一个任务成功,另一个任务被接收。看着花,我看到了一堆已经开始的任务,现在被困在“开始”或“收到”中。
编辑:在尝试在command=选项中直接运行芹菜而不是运行shell脚本之后,我注意到了相同的行为。如果我在前台通过ssh运行我的脚本(芹菜处于调试模式),那么当点击ctrl-c时,我将得到以下结果:
worker: Hitting Ctrl+C again will terminate all running tasks!
289
290worker: Warm shutdown (MainProcess)
291[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Hub...
292[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Pool...
293[2014-05-18 13:12:49,557: DEBUG/MainProcess] | Worker: Closing Consumer...
294[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Worker: Stopping Consumer...
295[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Connection...
296[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Events...
297[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Mingle...
298[2014-05-18 13:12:49,558: DEBUG/MainProcess] | Consumer: Closing Tasks...
299[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Control...
300[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing Heart...
301[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Closing event loop...
302[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping event loop...
303[2014-05-18 13:12:49,559: DEBUG/MainProcess] | Consumer: Stopping Heart...
304[2014-05-18 13:12:49,564: DEBUG/MainProcess] | Consumer: Stopping Control...
305[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Tasks...
306[2014-05-18 13:12:49,570: DEBUG/MainProcess] Cancelling task consumer...
307[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Mingle...
308[2014-05-18 13:12:49,570: DEBUG/MainProcess] | Consumer: Stopping Events...
309[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Consumer: Stopping Connection...
310[2014-05-18 13:12:49,571: DEBUG/MainProcess] | Worker: Stopping Pool...
311[2014-05-18 13:12:50,608: DEBUG/MainProcess] | Worker: Stopping Hub...
312[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Heart...
313[2014-05-18 13:12:50,612: DEBUG/MainProcess] | Consumer: Shutdown Control...
314[2014-05-18 13:12:50,613: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
315[2014-05-18 13:12:50,614: DEBUG/MainProcess] Cancelling task consumer...
316[2014-05-18 13:12:50,614: DEBUG/MainProcess] Closing consumer channel...
317[2014-05-18 13:12:50,615: DEBUG/MainProcess] | Consumer: Shutdown Events...
318[2014-05-18 13:12:50,616: DEBUG/MainProcess] | Consumer: Shutdown Connection...
319[2014-05-18 13:12:50,617: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
这看起来像是正常的关机过程,我希望看到发生在主管身上。
另一方面,停止使用supervisor会给我以下好处:
[2014-05-18 13:14:28,369: DEBUG/MainProcess] pidbox received method ping() [reply_to:{u'routing_key': u'1bcaf3a6-0c77-33b9-9d46-56e8634e5b1d', u'exchange': u'reply.celery.pidbox'} ticket:161fcbdb-dac0-48c9-ac13-ab7a75bfdc85]
没有一个关闭跟踪像按下ctrl-c时看到的那样。它只是停止。
最佳答案
这似乎是ubuntu(3.0a)附带的supervisor版本中的一个bug。在切换到supervisor的当前稳定版本(在/usr/local/bin中手动安装)之后,我可以看到sigterm确实正在接收的日志。
关于ubuntu - supervisord和 celery 暖关机,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/23714563/