本文介绍了Flume HDFS Sink在HDFS上生成大量小文件的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有一个玩具设置,使用水槽向hdfs发送log4j消息。我无法配置hdfs接收器以避免许多小文件。我想我可以在每次文件大小达到10MB时配置hdfs接收器来创建一个新文件,但它仍然创建约1.5KB的文件。

I have a toy setup sending log4j messages to hdfs using flume. I'm not able to configure the hdfs sink to avoid many small files. I thought I could configure the hdfs sink to create a new file every-time the file size reaches 10mb, but it is still creating files around 1.5KB.

这是我的当前水槽配置:

Here is my current flume config:

a1.sources=o1
a1.sinks=i1
a1.channels=c1

#source configuration
a1.sources.o1.type=avro
a1.sources.o1.bind=0.0.0.0
a1.sources.o1.port=41414

#sink config
a1.sinks.i1.type=hdfs
a1.sinks.i1.hdfs.path=hdfs://localhost:8020/user/myName/flume/events
#never roll-based on time
a1.sinks.i1.hdfs.rollInterval=0
#10MB=10485760
a1.sinks.il.hdfs.rollSize=10485760
#never roll base on number of events
a1.sinks.il.hdfs.rollCount=0

#channle config
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100

a1.sources.o1.channels=c1
a1.sinks.i1.channel=c1


推荐答案

我t是您在conf中的错字。

It is your typo in conf.

#sink config
a1.sinks.i1.type=hdfs
a1.sinks.i1.hdfs.path=hdfs://localhost:8020/user/myName/flume/events
#never roll-based on time
a1.sinks.i1.hdfs.rollInterval=0
#10MB=10485760
a1.sinks.il.hdfs.rollSize=10485760
#never roll base on number of events
a1.sinks.il.hdfs.rollCount=0

在'rollSize'和'rollCount'行中,您将 il 作为 i1
请尝试使用DEBUG,然后您会看到:

where in the line 'rollSize' and 'rollCount', you put il as i1.Please try to use DEBUG, then you will find like:

[SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.shouldRotate:465)  - rolling: rollSize: 1024, bytes: 1024

由于 il ,正在使用rollSize 1024的默认值。

Due to il, default value of rollSize 1024 is being used .

这篇关于Flume HDFS Sink在HDFS上生成大量小文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-24 03:12