且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Flume HDFS Sink在HDFS上生成大量小文件

更新时间:2023-11-10 11:08:46

我t是您在conf中的错字。

It is your typo in conf.

#sink config
a1.sinks.i1.type=hdfs
a1.sinks.i1.hdfs.path=hdfs://localhost:8020/user/myName/flume/events
#never roll-based on time
a1.sinks.i1.hdfs.rollInterval=0
#10MB=10485760
a1.sinks.il.hdfs.rollSize=10485760
#never roll base on number of events
a1.sinks.il.hdfs.rollCount=0

在'rollSize'和'rollCount'行中,您将 il 作为 i1
请尝试使用DEBUG,然后您会看到:

where in the line 'rollSize' and 'rollCount', you put il as i1. Please try to use DEBUG, then you will find like:

[SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.shouldRotate:465)  - rolling: rollSize: 1024, bytes: 1024

由于 il ,正在使用rollSize 1024的默认值。

Due to il, default value of rollSize 1024 is being used .