下面是我的水槽配置文件。即使更改了rollInterval和rollSize之后，仅写入了10个事件，控制台仍显示rollCount = 10和events = 10。我也尝试将rollCount增加到1000，但输出没有变化。任何人都可以建议增加在hdfs中写入的文件大小。下面的conf文件怎么了？
#naming components
NetAgent.sources = NetCat_1 NetCat_2
NetAgent.sinks = HDFS
NetAgent.channels = MemChannel
NetAgent.sources.NetCat_1.type = netcat
NetAgent.sources.NetCat_1.bind = localhost
NetAgent.sources.NetCat_1.port = 8671
NetAgent.sources.NetCat_2.type = netcat
NetAgent.sources.NetCat_2.bind = localhost
NetAgent.sources.NetCat_2.port = 8672
NetAgent.sinks.HDFS.type = hdfs
NetAgent.sinks.HDFS.hdfs.path = file path here
NetAgent.sinks.HDFS.hdfs.filePrefix = test
NetAgent.sinks.HDFS.hdfs.rollSize = 67108864
NetAgent.sinks.HDFS.hdfs.rollInterval = 3600
NetAgent.sinks.HDFS.rollCount = 0
NetAgent.sinks.HDFS.hdfs.batchSize = 10000
NetAgent.sinks.HDFS.hdfs.writeFormat = Text
NetAgent.sinks.HDFS.hdfs.fileType = DataStream
NetAgent.channels.MemChannel.type = memory
NetAgent.channels.MemChannel.capacity = 20000
NetAgent.channels.MemChannel.transactionCapacity = 20000
NetAgent.sources.NetCat_1.channels = MemChannel
NetAgent.sources.NetCat_2.channels = MemChannel
NetAgent.sinks.HDFS.channel = MemChannel
控制台记录为
(SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUg-org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java)]
rolling: rollCount: 10, events: 10
the image shows the files written in HDFS
最佳答案
您忘记将hdfs添加到rollCount配置中。它使用默认值10，因为它看不到您的配置。请注意，HDFS的配置为:
NetAgent.sinks.HDFS.type = hdfs
NetAgent.sinks.HDFS.hdfs.rollSize = 67108864
NetAgent.sinks.HDFS.hdfs.rollInterval = 3600
NetAgent.sinks.HDFS.rollCount = 0
NetAgent.sinks.HDFS.hdfs.batchSize = 10000
NetAgent.sinks.HDFS.hdfs.writeFormat = Text
NetAgent.sinks.HDFS.hdfs.fileType = DataStream
在rollCount行中，它必须为:
NetAgent.sinks.HDFS.hdfs.rollCount = 0
这将覆盖默认的rollCount，并且Flume代理将按照您希望的方式运行。