将数据写入槽，然后写入HDFS

问题描述

我使用flume 1.5.0.1和hadoop 2.4.1试图将一个字符串放入水槽并保存到HDFS。 Flume配置文件如下：

  agentMe.channels =内存通道
 agentMe.sources = my-source AvroSource 
 agentMe.sinks = log-sink hdfs-sink 
 
 agentMe.sources.AvroSource.channels =记忆频道
 agentMe.sources.AvroSource.type = avro 
 agentMe.sources.AvroSource.bind = 0.0.0.0＃我试过客户端ip以及
 agentMe.sources.AvroSource.port = 41414 
 
 agentMe.channels.memory-channel.type = memory 
 agentMe.channels.memory-channel.capacity = 1000 
 agentMe.channels.memory-channel.transactionCapacity = 100 
 
 agentMe.sources.my-source.type = netcat 
 agentMe.sources.my-source.bind = 127.0.0.1＃如果我使用任何其他IP，如字符串将从哪里来的客户端，那么我无法绑定异常。 
 agentMe.sources.my-source.port = 9876 
 agentMe.sources.my-source.channels =内存通道
 
 
＃定义一个接收器到hdfs。 
 agentMe.sinks.hdfs-sink.channel =内存频道
 agentMe.sinks.hdfs-sink.type = hdfs 
 agentMe.sinks.hdfs-sink.hdfs.path = hdfs： //localhost:54310/user/netlog/flume.txt 
 agentMe.sinks.hdfs-sink.hdfs.fileType = DataStream 
 agentMe.sinks.hdfs-sink.hdfs.batchSize = 2 
 agentMe.sinks.hdfs-sink.hdfs.rollCount = 0 
 agentMe.sinks.hdfs-sink.hdfs.inUsePrefix = tcptest- 
 agentMe.sinks.hdfs-sink.hdfs.inUseSuffix = .txt 
 agentMe.sinks.hdfs-sink.hdfs.rollSize = 0 
 agentMe.sinks.hdfs-sink.hdfs.rollInterval = 3 
 agentMe.sinks.hdfs-sink.hdfs.writeFormat = Text 
 agentMe.sinks.hdfs-sink.hdfs.path = / user / name /％y-％m-％d /％H％M /％S 
  

我已经提出了相同的问题

  client.sendDataToFlume（hello world）
  code> 

我看NettyAvroRpcClien无法连接到正在运行水槽的服务器。但我只是发送一个简单的字符串，我错过了任何东西。

专家善意地建议 div>

配置必须正确，否则事情可能无法解决。因此，这里的配置是将数据读入flume，然后读入HDFS。

  a1.sources = r1 
 a1。 sinks = k2 
1.channels = c1 
 
 a1.channels.c1.type =内存
 
 a1.sources.r1.channels = c1 
 a1.sources.r1.type = avro 
 a1.sources.r1.bind = 0.0.0.0 
 a1.sources.r1.port = 41414 
 a1.sources.r1.interceptors = a 
 a1.sources.r1.interceptors.a.type = org.apache.flume.interceptor.TimestampInterceptor $ Builder 
 
 a1.sinks.k2.type = hdfs 
 a1。 sinks.k2.channel = c1 
 a1.sinks.k2.hdfs.fileType = DataStream 
 a1.sinks.k2.hdfs.batchSize = 10 
1.ashs.k2.hdfs.rollCount = 10 
1.a1.sinks.k2.hdfs.rollSize = 10 
 a1.sinks.k2.hdfs.rollInterval = 10 
 a1.sinks.k2.hdfs.writeFormat =文本
 a1.sinks.k2.hdfs.path = / user / flume /％y-％m-％d /％H％M / 
 
＃使用缓冲存储器中事件的通道
 a1.channels.c1.type =内存
 a1.channels.c1.capacity = 1000 
 a1.channels.c1.transactionCapacity = 100 
 
＃将源和汇绑定到频道
 a1.sources.r1.channels = c1 
 a1.sinks.k2.channel = c1 
  

这应该有助于：）

I am using flume 1.5.0.1 and hadoop 2.4.1 trying to put a string into flume and save to HDFS. Flume configuration file is as follows:

    agentMe.channels = memory-channel
agentMe.sources = my-source AvroSource
agentMe.sinks = log-sink hdfs-sink

agentMe.sources.AvroSource.channels = memory-channel
agentMe.sources.AvroSource.type = avro
agentMe.sources.AvroSource.bind = 0.0.0.0 # i tried client ip as well
agentMe.sources.AvroSource.port = 41414

agentMe.channels.memory-channel.type = memory
agentMe.channels.memory-channel.capacity = 1000
agentMe.channels.memory-channel.transactionCapacity = 100

agentMe.sources.my-source.type = netcat
agentMe.sources.my-source.bind = 127.0.0.1 #If i use any other IP like the client from where the string is going to come from then i get unable to bind exception.
agentMe.sources.my-source.port = 9876
agentMe.sources.my-source.channels = memory-channel


# Define a sink that outputs to hdfs.
agentMe.sinks.hdfs-sink.channel = memory-channel
agentMe.sinks.hdfs-sink.type = hdfs
agentMe.sinks.hdfs-sink.hdfs.path = hdfs://localhost:54310/user/netlog/flume.txt
agentMe.sinks.hdfs-sink.hdfs.fileType = DataStream
agentMe.sinks.hdfs-sink.hdfs.batchSize = 2
agentMe.sinks.hdfs-sink.hdfs.rollCount = 0
agentMe.sinks.hdfs-sink.hdfs.inUsePrefix = tcptest-
agentMe.sinks.hdfs-sink.hdfs.inUseSuffix = .txt
agentMe.sinks.hdfs-sink.hdfs.rollSize = 0
agentMe.sinks.hdfs-sink.hdfs.rollInterval = 3
agentMe.sinks.hdfs-sink.hdfs.writeFormat = Text
agentMe.sinks.hdfs-sink.hdfs.path = /user/name/%y-%m-%d/%H%M/%S

I have already put the same question here

client.sendDataToFlume("hello world")

I see NettyAvroRpcClient not able to connect to the server where flume is running. But am just sending a simple string am i missing anything.

Experts kindly suggest

解决方案

Configuration have to be correct otherwise things may not work out. So here is the config to read data into flume and then into HDFS.

a1.sources = r1
a1.sinks =  k2
a1.channels = c1

a1.channels.c1.type = memory

a1.sources.r1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 41414
a1.sources.r1.interceptors = a
a1.sources.r1.interceptors.a.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

a1.sinks.k2.type = hdfs
a1.sinks.k2.channel = c1
a1.sinks.k2.hdfs.fileType = DataStream
a1.sinks.k2.hdfs.batchSize = 10
a1.sinks.k2.hdfs.rollCount = 10
a1.sinks.k2.hdfs.rollSize = 10
a1.sinks.k2.hdfs.rollInterval = 10
a1.sinks.k2.hdfs.writeFormat = Text
a1.sinks.k2.hdfs.path = /user/flume/%y-%m-%d/%H%M/

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k2.channel = c1

this should help :)

这篇关于将数据写入槽，然后写入HDFS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！