且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

如何将JavaPairDStream的结果写入Spark Streaming的输出kafka主题中?

更新时间:2023-10-15 11:23:34

如果dStream包含要发送到Kafka的数据:

if dStream contains data that you want to send to Kafka:

dStream.foreachRDD(rdd -> {
    rdd.foreachPartition(iter ->{
        Producer producer = createKafkaProducer();  
        while (iter.hasNext()){
               sendToKafka(producer, iter.next())
        }
    }

});

因此,您为每个RDD分区创建一个生产者.

So, you create one producer per each RDD partition.