咘雷扎克

咘雷扎克

关注
发信
关注(28)粉丝(399)

环境篇：Atlas2.1.0兼容CDH6.3.2部署

环境篇：Atlas2.1.0兼容CDH6.3.2部署

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

  • 字段与字段之间的血缘依赖

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

1 Atlas 架构原理

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

2 Atlas 安装及使用

2.1 下载源码包2.0.0，IDEA打开

  • 因与CDH集成，修改pom文件
  • 在repositories标签中增加CDH仓库
<repository>
     <id>cloudera</id>
     <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
     <releases>
         <enabled>true</enabled>
     </releases>
     <snapshots>
         <enabled>false</enabled>
     </snapshots>
 </repository>

2.2 修改相关版本与CDH版本对应

<lucene-solr.version>7.4.0</lucene-solr.version>
<hadoop.version>3.0.0-cdh6.3.2</hadoop.version>
<hbase.version>2.1.0-cdh6.3.2</hbase.version>
<solr.version>7.4.0-cdh6.3.2</solr.version>
<hive.version>2.1.1-cdh6.3.2</hive.version>
<kafka.version>2.2.1-cdh6.3.2</kafka.version>
<kafka.scala.binary.version>2.11</kafka.scala.binary.version>
<zookeeper.version>3.4.5-cdh6.3.2</zookeeper.version>
<sqoop.version>1.4.7-cdh6.3.2</sqoop.version>

2.3 兼容Hive2.1.1

  • 所需修改的项目位置：apache-atlas-sources-2.1.0\addons\hive-bridge

①.org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 577行

String catalogName = hiveDB.getCatalogName() != null ? hiveDB.getCatalogName().toLowerCase() : null;

改为：

String catalogName = null;

②.org/apache/atlas/hive/hook/AtlasHiveHookContext.java 81行

this.metastoreHandler = (listenerEvent != null) ? metastoreEvent.getIHMSHandler() : null;

改为：C:\Users\Heaton\Desktop\apache-atlas-2.1.0-sources\apache-atlas-sources-2.1.0\addons

this.metastoreHandler = null;

2.3 编译

mvn clean -DskipTests package -Pdist -X -T 8
  • 编译完成的文件在此目录apache-atlas-sources-2.1.0\distro\target

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP
环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

2.5 安装

mkdir /usr/local/src/atlas
cd /usr/local/src/atlas
#复制apache-atlas-2.1.0-bin.tar.gz到安装目录
tar -zxvf apache-atlas-2.1.0-bin.tar.gz
cd apache-atlas-2.1.0/

2.6 修改配置文件

#集成修改hbase配置
atlas.graph.storage.hostname=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181
#集成修改solr配置
atlas.graph.index.search.solr.zookeeper-url=cdh01.cm:2181/solr,cdh02.cm:2181/solr,cdh03.cm:2181/solr
#集成修改kafka配置
atlas.notification.embedded=false		    #false外置的kafka
atlas.kafka.zookeeper.connect=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181
atlas.kafka.bootstrap.servers=cdh01.cm:9092,cdh02.cm:9092,cdh03.cm:9092
atlas.kafka.zookeeper.session.timeout.ms=60000
atlas.kafka.zookeeper.connection.timeout.ms=30000
atlas.kafka.enable.auto.commit=true
#集成修改其他配置
atlas.rest.address=http://cdh01.cm:21000	#访问地址端口，此值修改不生效，默认本地21000端口，此端口和impala冲突
atlas.server.run.setup.on.start=false		   #如果启用并设置为true，则在服务器启动时将运行安装步骤
atlas.audit.hbase.zookeeper.quorum=cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181
#集成添加hive钩子配置(文件最下面即可)
#在hive中做任何操作，都会被钩子所感应到，并生成相应的事件发往atlas所订阅的kafka-topic，再由atlas进行元数据生成和存储管理
######### Hive Hook Configs #######
atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.queueSize=10000
atlas.cluster.name=primary

#配置用户名密码(选做)
    #开启或关闭三种验证方法
atlas.authentication.method.kerberos=true|false
atlas.authentication.method.ldap=true|false
atlas.authentication.method.file=true
   #vim users-credentials.properties(修改该文件)
   #>>>源文件
   #username=group::sha256-password
   admin=ADMIN::8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
   rangertagsync=RANGER_TAG_SYNC::e3f67240f5117d1753c940dae9eea772d36ed5fe9bd9c94a300e40413f1afb9d
   #<<<
   #>>>修改成用户名bigdata123，密码bigdata123
   #username=group::sha256-password
   bigdata123=ADMIN::aa0336d976ba6db36f33f75a20f68dd9035b1e0e2315c331c95c2dc19b2aac13
   rangertagsync=RANGER_TAG_SYNC::e3f67240f5117d1753c940dae9eea772d36ed5fe9bd9c94a300e40413f1afb9d
   #<<<
   #计算sha256：echo -n "bigdata123"|sha256sum

#集成添加hbase配置->下面的目录为atlas下的hbase配置目录，需要后面加入集群hbase配置
export HBASE_CONF_DIR=/usr/local/src/atlas/apache-atlas-2.1.0/conf/hbase/conf
#export HBASE_CONF_DIR=/etc/hbase/conf------------------------------
export MANAGE_LOCAL_HBASE=false  (false外置的zk和hbase)
export MANAGE_LOCAL_SOLR=false  （false外置的solr）

#修改内存指标（根据线上机器配置）
export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0
-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=dumps/atlas_server.hprof
-Xloggc:logs/gc-worker.log -verbose:gc
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC
-XX:+PrintGCTimeStamps"

#优化 JDK1.8(以下需要16G内存)
export ATLAS_SERVER_HEAP="-Xms15360m -Xmx15360m
-XX:MaxNewSize=5120m -XX:MetaspaceSize=100M
-XX:MaxMetaspaceSize=512m"

#去掉如下代码的注释(开启如下代码)
    <appender name="perf_appender" class="org.apache.log4j.DailyRollingFileAppender">
        <param name="file" value="${atlas.log.dir}/atlas_perf.log" />
        <param name="datePattern" value="'.'yyyy-MM-dd" />
        <param name="append" value="true" />
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d|%t|%m%n" />
        </layout>
    </appender>

    <logger name="org.apache.atlas.perf" additivity="false">
        <level value="debug" />
        <appender-ref ref="perf_appender" />
    </logger>

2.7 集成Hbase

  • 添加hbase集群配置文件到apache-atlas-2.0.0/conf/hbase下(这里连接的路径需要和上面atlas-env.sh配置中一样)
ln -s /etc/hbase/conf/ /usr/local/src/atlas/apache-atlas-2.1.0/conf/hbase/

2.8 集成Solr

  • 将apache-atlas-2.1.0/conf/solr文件拷贝到solr所有节点的安装目录下，更名为atlas-solr
scp -r /usr/local/src/atlas/apache-atlas-2.1.0/conf/solr [email protected]:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/
scp -r /usr/local/src/atlas/apache-atlas-2.1.0/conf/solr [email protected]:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/
scp -r /usr/local/src/atlas/apache-atlas-2.1.0/conf/solr [email protected]:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/

#在solr节点
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/
mv solr/ atlas-solr
#在任意solr节点修改solr对应的bash
vi /etc/passwd
/sbin/nologin 修改为 /bin/bash
#切换solr用户执行
su solr
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/bin/solr create -c  vertex_index -d /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/atlas-solr -shards 3 -replicationFactor 2

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/bin/solr create -c  edge_index -d /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/atlas-solr -shards 3 -replicationFactor 2

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/bin/solr create -c  fulltext_index -d /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/atlas-solr -shards 3 -replicationFactor 2

#如果创建错误，可使用 /opt/cloudera/parcels/CDH/lib/solr/bin/solr delete -c ${collection_name} 删除
#切换root用户继续配置其他
su root

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

2.9 集成kafka

  • 创建kafka-topic
kafka-topics --zookeeper cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181 --create --replication-factor 3 --partitions 3 --topic _HOATLASOK
kafka-topics --zookeeper cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181 --create --replication-factor 3 --partitions 3 --topic ATLAS_ENTITIES
kafka-topics --zookeeper cdh01.cm:2181,cdh02.cm:2181,cdh03.cm:2181 --create --replication-factor 3 --partitions 3 --topic ATLAS_HOOK

2.10 启动测试

cd /usr/local/src/atlas/apache-atlas-2.1.0/

./bin/atlas_start.py

#停止：./bin/atlas_stop.py

2.11 集成Hive

  • 将 atlas-application.properties 配置文件，压缩加入到 atlas-plugin-classloader-2.0.0.jar 中
#必须在此路径打包，才能打到第一级目录下
cd /usr/local/src/atlas/apache-atlas-2.1.0/conf

zip -u /usr/local/src/atlas/apache-atlas-2.1.0/hook/hive/atlas-plugin-classloader-2.1.0.jar atlas-application.properties
  • 修改 hive-site.xml

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

<property>
    <name>hive.exec.post.hooks</name>
    <value>org.apache.atlas.hive.hook.HiveHook</value>
</property>
  • 修改 hive-env.sh 的 Gateway 客户端环境高级配置代码段（安全阀）

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

HIVE_AUX_JARS_PATH=/usr/local/src/atlas/apache-atlas-2.1.0/hook/hive
  • 修改 HIVE_AUX_JARS_PATH

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

  • 修改 hive-site.xml 的 HiveServer2 高级配置代码段（安全阀）

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

<property>
    <name>hive.exec.post.hooks</name>
    <value>org.apache.atlas.hive.hook.HiveHook</value>
</property>
<property>
    <name>hive.reloadable.aux.jars.path</name>
    <value>/usr/local/src/atlas/apache-atlas-2.1.0/hook/hive</value>
</property>
  • 修改 HiveServer2 环境高级配置代码段

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

HIVE_AUX_JARS_PATH=/usr/local/src/atlas/apache-atlas-2.1.0/hook/hive
  • 将配置好的Atlas包发往各个hive节点后重启集群
scp -r /usr/local/src/atlas/apache-atlas-2.1.0 [email protected]:/usr/local/src/atlas/

scp -r /usr/local/src/atlas/apache-atlas-2.1.0 [email protected]:/usr/local/src/atlas/
  • 将atlas配置文件copy到/etc/hive/conf下(集群各个节点)
scp /usr/local/src/atlas/apache-atlas-2.1.0/conf/atlas-application.properties [email protected]:/etc/hive/conf

scp /usr/local/src/atlas/apache-atlas-2.1.0/conf/atlas-application.properties [email protected]:/etc/hive/conf

scp /usr/local/src/atlas/apache-atlas-2.1.0/conf/atlas-application.properties [email protected]:/etc/hive/conf

2.12 再次启动 Atlas

#启动
./bin/atlas_start.py
#停止：./bin/atlas_stop.py

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

2.13 将 Hive 元数据导入 Atlas

  • atlas节点添加hive环境变量
vim /etc/profile
#>>>
#hive
export HIVE_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive
export HIVE_CONF_DIR=/etc/hive/conf
export PATH=$HIVE_HOME/bin:$PATH
#<<<

source /etc/profile
  • 执行atlas脚本
./bin/import-hive.sh
#输入用户名：admin；输入密码：admin(如修改请使用修改的)

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

环境篇：Atlas2.1.0兼容CDH6.3.2部署-LMLPHP

12-16 21:49
Powered by LMLPHP ©2024 咘雷扎克 0.079449