问题描述
全部，我试图强制 RQ工作人员使用主管进行同时执行。我的设置监督设置似乎工作正常，因为rq仪表板显示3个工作程序，3个PID和3个队列（每个工作程序/ PID一个）。 Supervisord的设置如下（仅显示worker 1的设置，在此下面定义了2个worker）：
All, I'm attempting to 'force' RQ workers to perform concurrently using supervisord. My setup supervisord setup seems to work fine, as rq-dashboard is showing 3 workers, 3 PID's and 3 queue (one for each worker/PID). Supervisord setup is as follows (showing only worker 1 setup, 2 more workers are defined below this one):
[program:rqworker1]
command = rqworker 1
process_name = rqworker1-%(process_num)s
numprocs = 1
user = username
autostart = True
stdout_logfile=/tmp/rqworker1.log
stdout_logfile_maxbytes=50MB
问题是当我同时发送3个作业时，总时间运行时间是单个任务的x3（即，总时间与任务数量成线性关系，缩放为x4，x5等）。似乎没有并发可用。我还通过将新作业发送到具有最少启动+排队作业最少的队列中来实现原始负载平衡，该工作正常（观察到作业均匀地分布在队列中）。
The issue is when I send 3 jobs concurrently, the total time to run is x3 that of a single task (namely, total time is linear with number of tasks, this scales to x4,x5, etc..). It seems no concurrency is available. I also implemented a primitive load-balancing by sending new jobs to the queue with minimum started+queued jobs, that works fine (jobs are observed to be spread evenly among queues).
为什么此设置不允许并发？
Why would this setup not allow concurrency?
是否缺少任何有关设置的考虑？
Any considerations regarding the setup i'm missing?
请注意，rq-gevent-worker软件包（其中由于我迁移到PY3，并且Pevent3尚不支持gevent本身，因此在早期使用wrt并发性/ RQ时效果很好。但这为我提供了可能并发的线索。
Note that rq-gevent-worker package (which worked great earlier w.r.t. concurrency/RQ) is no longer available as I migrated to PY3 and gevent itself is not yet supported on PY3. But this gives my a clue that concurrency is possible.
推荐答案
使用 supervisord 运行多个 rqworker 进程是<$ c中的 $ c> python-rq ，所以不必担心您在强制执行它。您实际上有一个正确的想法。
Using supervisord to run multiple rqworker processes in parallel is an intended pattern in python-rq, so don't be concerned that you're "forcing" it. You actually have the right idea.
另一方面，编写自己的负载均衡算法是一种反模式： python- rq 为您服务。
On the other hand, writing your own load balancing algorithm is an anti-pattern: that's exactly what python-rq does for you.
如果您想将工作分配给三个工人，那么他们都应该听同一个队列。尝试删除两个超级用户配置块，然后在剩余的一个块中，将 numprocs 更改为 3 。如果您将三个作业快速提交到该队列，则应该看到三个作业同时执行。
If you want to split up work between three workers, then they should all listen to the same queue. Try removing two of your supervisor config blocks, and in the one block that remains, change numprocs to 3. If you submit three jobs rapidly to that queue, you should see three workers executing concurrently.
这篇关于与主管进行RQ并发？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！