且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Kafka 流过滤:经纪人还是消费者?

更新时间:2023-11-26 20:04:58

Kafka 不支持代理端过滤.如果您使用 Streams API,过滤将在您的应用程序中完成(谓词不会由 KafkaConsumer 评估,而是在您的拓扑的处理器节点"内——即在 Streams API 运行时代码内).

Kafka does not support broker side filtering. If you use Streams API, filtering will be done in your application (the predicate will not be evaluated by KafkaConsumer but within a "processor node" of your topology -- ie, within Streams API runtime code).

这可能有帮助:https://docs.confluent.io/current/streams/架构.html

不支持代理端过滤的原因是,代理仅使用 (1) 字节数组作为键和值数据类型,并使用 (2) 零复制机制来实现高吞吐量.需要代理端过滤,以便在代理端反序列化数据,这会对性能造成重大影响(反序列化成本和无零拷贝优化).

The reason for not supporting broker side filtering is, that brokers only use (1) byte arrays as key and value data types and use (2) zero-copy mechanism to achieve high throughput. Broker side filtering would required, to deserialize the data at the broker side what would be a major performance hit (deserialization cost and no zero-copy optimization).