且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Celery 广播与 RabbitMQ 扇出

更新时间:2023-02-05 12:32:09

这有帮助吗?
http://celery.readthedocs.org/en/latest/userguide/routing.html#exchanges-queues-and-routing-keys

似乎 Celery 中的 'queue' 定义包含交换,因此您可以在 Exchange('fanout') 交换类型之上定义一个 Celery 队列,它将具有多个的底层实现RabbitMQ 队列.

在这种情况下,我猜您不希望 Celery 配置中有广播"队列,除非您真的希望多个工作人员处理同一任务.

I've been working with Celery lately and I don't like it. It's configuration is messy, overcomplicated and poorly documented.

I want to send broadcast messages with Celery from a single producer to multiple consumers. What confuses me is discrepancy between Celery terms and terms of underlying transport RabbitMQ.

In RabbitMQ you can have a single fanout Exchange and multiple Queues to broadcast messages:

But in Celery the terms are all messed up: here you can have a broadcast Queue, which sends messages to multiple consumers:

I don't even understand, how Celery broadcast queue is supposed to work at all, cause RabbitMQ queues with multiple consumers are meant for load balancing. So in RabbitMQ if multiple consumers (i.e. a pool of consumers) are connected to the same queue, only one consumer will receive and process message, which is called round robin in RabbitMQ docs.

Also, Celery documentation on broadcast is really insufficient. What type of RabbitMQ exchange should I specify for Broadcast queue, fanout or not? Could you supply a full example?

So, what I'm asking for is (1) clarification of concept and implementation of Broadcast queues in Celery and (2) a complete example of Broadcast queues configuration. Thank you.

Does this help?
http://celery.readthedocs.org/en/latest/userguide/routing.html#exchanges-queues-and-routing-keys

It appears the 'queue' definition in Celery includes the exchange, so you can define a Celery queue on top of Exchange('fanout') exchange type, which will have an underlying implementation of multiple RabbitMQ queues.

In this case I would guess you don't want a 'broadcast' queue in the Celery config, unless you really want multiple workers processing the same task.