且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

用于 Azure Blob 存储的 Kafka 连接器

更新时间:2022-11-05 20:19:01

自定义接收器连接器绝对有效.Kafka Connect 是绝对设计的,因此您可以插入连接器.事实上,连接器的开发是完全联合的.Confluent 的 JDBC 和 HDFS 连接器最初的实现仅仅是因为这两个用例的流行,但还有更多(我们保留了一个我们知道的连接器列表 这里.

A custom sink connector definitely works. Kafka Connect was absolutely designed so you could plugin connectors. In fact, connector development is entirely federated. Confluent's JDBC and HDFS connectors were implemented first simply due to the popularity of those two use cases, but there are many more (we keep a list of connectors we're aware of here.

就 Azure blob 存储是否合适而言,您提到了 JSON 对象.我认为您唯一需要考虑的是对象的大小以及 Azure 存储是否可以处理大小和大小.对象的数量很好.我不确定 Azure 存储的特性,但在许多其他对象存储系统中,您可能需要将许多对象聚合到一个 blob 中才能为大量对象获得良好的性能(即您可能需要一种支持许多 JSON 对象的文件格式).

In terms of whether Azure blob storage is appropriate, you mention JSON objects. I think the only thing you'll want to consider is the size of the objects and whether Azure storage will handle the size & number of objects well. I am not sure about Azure storage's characteristics, but in many other object storage systems you might need to aggregate many objects into a single blob to get good performance for a large number of objects (i.e. you might need a file format that supports many JSON objects).