cd /opt/cm-5.5.1/具体参考http://www.aboutyun.com/thread-10852-1-1.html
离线安装:下载所需文件
http://archive.cloudera.com/cm5/cm/5/中的cloudera-manager-trusty-cm5.5.1_amd64.tar.gz
http://archive.cloudera.com/cdh5/parcels/5.5.1/ 中的CDH-5.5.1-1.cdh5.5.1.p0.11-trusty.parcel和 CDH-5.5.1-1.cdh5.5.1.p0.11-trusty.parcel.sha1,还有manifest.json
mysql配置
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.html#cmig_topic_5_5
所有机器:
apt-get install ntp
修改主机名/etc/hostname
修改/etc/hosts只保留127.0.0.1 localhost
dpkg -i oracle-j2sdk1.7_1.7.0+update67-1_amd64.deb
CM Server节点配置mysql数据库
apt-get install mysql-server
cm server:
创建数据库:
#hive
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#activity monitor
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
tar zxvf /cloudera-manager-trusty-cm5.5.1_amd64.tar.gz
然后mv cm-5.5.1/ /opt
将mysql-connector添加到/opt/cm-5.5.1/share/cmf/lib/
初始化mysql的库
cd /opt/cm-5.5.1/
share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -proot --scm-host localhost scm scm scm
修改/opt/cm-5.5.1/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名
将/opt/cm-5.5.1同步到所有节点
所有节点创建用户
useradd --system --home-dir=/opt/cm-5.5.1/run/cloudera-scm-server --no-create-home --shell=/bin/false cloudera-scm
将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/(自己创建)目录中
CDH-5.5.1-1.cdh5.5.1.p0.11-trusty.parcel
CDH-5.5.1-1.cdh5.5.1.p0.11-trusty.parcel.sha
启动server
/opt/cm-5.5.1/etc/init.d/cloudera-scm-server start
启动所有agent
/opt/cm-5.5.1/etc/init.d/cloudera-scm-agent start
cp /mysql-connector-java-5.1.38-bin.jar /usr/share/java/mysql-connector-java.jar
cp /mysql-connector-java-5.1.38-bin.jar /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive/lib/
关闭虚拟内存:echo 0 > /proc/sys/vm/swappiness
create database oozie;
错误:
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/bin/hue", line 12, in <module>
load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
execute_from_command_line(sys.argv)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 284, in execute
self.validate()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 310, in validate
num_errors = get_validation_errors(s, app)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/validation.py", line 34, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/loading.py", line 196, in get_app_errors
self._populate()
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/loading.py", line 78, in _populate
self.load_app(app_name)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/loading.py", line 99, in load_app
models = import_module('%s.models' % app_name)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/apps/jobbrowser/src/jobbrowser/models.py", line 20, in <module>
import lxml.html
File "/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/lxml-2.2.2-py2.7-linux-x86_64.egg/lxml/html/__init__.py", line 12, in <module>
from lxml import etree
ImportError: libxslt.so.1: cannot open shared object file: No such file or directory
解决方法:apt-get install python-libxslt1
一些简单操作:
sudo -u hdfs hadoop fs -put /test.txt /hh/
运行wordcount
sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep /hh /out 'd[a-z.]+'
查看结果:
sudo -u hdfs hadoop fs -cat /out/*
sudo -u hdfs hadoop fs -rmr /out
关闭节点服务:
/opt/cm-5.5.1/etc/init.d/cloudera-scm-agent hard_stop_confirmed