Flink-on-yarn

介绍

官网下载

• https://www.apache.org/dyn/closer.lua/flink/flink-1.6.1/flink-1.6.1-bin-hadoop28-scala_2.11.tgz

flink集群地址

• ddcve: master
• dddve: slave
• ddeve: slave

解压

tar -xzvf flink-1.6.1-bin-hadoop28-scala_2.11.tgz

mv flink-1.6.1 flink

cd flink

修改配置

• 指定master
  • vim conf/flink-conf.yaml

jobmanager.rpc.address: ddcve   # 指定master

• 指定slaves

echo -e "dddve\nddeve" > conf/slaves

添加yarn配置项

• hadoop/etc/hadoop/yarn-site.xml

<property>

    <name>yarn.nodemanager.resource.cpu-vcores</name>

    <value>8</value>

</property>

启动flink

./bin/start-cluster.sh

• 输出结果如下

Starting HA cluster with 2 masters.

Starting standalonesession daemon on host ddcvc.

Starting standalonesession daemon on host ddcve.

Starting taskexecutor daemon on host dddve.

Starting taskexecutor daemon on host ddeve.

flink on yarn

• 启动yarn-session

./bin/yarn-session.sh -d -n 4 -tm 2048 -s 4

• 提交job

# 先在hdfs创建输入文件

hadoop fs -copyFromLocal /tmp/passwd /tmp/passwd

# 提交

./bin/flink run ./examples/batch/WordCount.jar --input hdfs:///tmp/passwd

flink with zookeeper

• 修改conf/flink-conf.yaml

high-availability: zookeeper

high-availability.storageDir: hdfs:///flink/ha/

high-availability.zookeeper.quorum: ddcva:2181,dddva:2181,ddeva:2181

recovery.zookeeper.path.root: /flink

high-availability.zookeeper.client.acl: open

state.backend: filesystem

• 修改conf/masters

ddcvc:8081

ddcve:8081