Channel组件

Flume的常用Channel

Memory Channel

File Channel

agent-aisi.sources = source-aisi
agent-aisi.channels = channel-aisi
agent-aisi.sinks =  sink-aisi

agent-aisi.sources.source-aisi.type = netcat
agent-aisi.sources.source-aisi.bind = localhost
agent-aisi.sources.source-aisi.port = 44444

agent-aisi.channels.channel-aisi.type = file
agent-aisi.channels.channel-aisi.dataDir = /root/filedata

agent-aisi.sinks.sink-aisi.type = logger

agent-aisi.sources.source-aisi.channels = channel-aisi
agent-aisi.sinks.sink-aisi.channel = channel-aisi

运行命令

flume-ng agent -n agent-aisi -c /usr/local/flume/conf/ -f ./flume-file-channel.txt -Dflume.root.logger=INFO,console

 telnet localhost 44444

数据保存在 /root/.flume/file-channel/data这个目录

Kafka Channel

kafka-topics.sh --zookeeper 11.112.227.47:2181 --create --replication-factor 1 --partitions 1 --topic demo\

kafka-topics.sh --list --zookeeper 11.112.227.47:2181

消费数据

kafka-console-consumer.sh --bootstrap-server 11.112.227.47:9092 --topic demo

生产数据

kafka-console-producer.sh --broker-list 11.112.227.47:9092 --topic demo 

agent-aisi.sources = source-aisi
agent-aisi.channels = channel-aisi
agent-aisi.sinks =  sink-aisi

agent-aisi.sources.source-aisi.type = netcat
agent-aisi.sources.source-aisi.bind = localhost
agent-aisi.sources.source-aisi.port = 44444

agent-aisi.channels.channel-aisi.type = org.apache.flume.channel.kafka.KafkaChannel
agent-aisi.channels.channel-aisi.kafka.bootstrap.servers = 11.112.227.47:9092
agent-aisi.channels.channel-aisi.kafka.topic = aisi

agent-aisi.sinks.sink-aisi.type = logger

agent-aisi.sources.source-aisi.channels = channel-aisi
agent-aisi.sinks.sink-aisi.channel = channel-aisi

agent启动

flume-ng agent -n agent-aisi -c /usr/local/flume/conf/ -f ./flume-kafka-channel.txt -Dflume.root.logger=INFO,console

装有flume的linux监听

telnet 11.112.227.32 44444

kafkachannel启动

kafka-console-consumer.sh --bootstrap-server 11.112.227.47:9092 --topic aisi