下载安装包 | sqoop-1.99.3-bin-hadoop200.tar.gz |
解压 | tar zxvf sqoop-1.99.3-bin-hadoop200.tar.gz |
建立sqoop链接 | ln -s sqoop-1.99.3-bin-hadoop200 sqoop |
修改sqoop配置 | cd sqoop vi server/conf/catalina.properties 修改内容如下：
找到common.loader行，把/usr/lib/hadoop/lib/*.jar改成你的hadoop jar 包目录
例如:/home/hadoop/hadoop/share/hadoop/yarn/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/yarn/*.jar,
/home/hadoop/hadoop/share/hadoop/hdfs/*.jar,
/home/hadoop/hadoop/share/hadoop/hdfs/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/mapreduce/*.jar,
/home/hadoop/hadoop/share/hadoop/mapreduce/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/common/lib/*.jar,
/home/hadoop/hadoop/share/hadoop/common/*.jar vi server/conf/sqoop.properties
找到：mapreduce.configuration.directory行，修改值为你的hadoop配置文件目录
如：/home/hadoop/hadoop/etc/hadoop/
并且替换@LOGDIR@ 和@BASEDIR@ ：
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g 然后找到你的数据库jdbc驱动复制到sqoop/lib目录下，如果不存在则创建 |
修改环境参数 | vi /etc/profile 增加以下内容： export SQOOP_HOME=/home/hadoop/sqoop export PATH=$PATH:$SQOOP_HOME/bin export CATALINA_BASE=$SQOOP_HOME/server export LOGDIR=$SQOOP_HOME/logs/ |
执行环境参数 | source /etc/profile |
启动 | ./bin/sqoop.sh server start |
测试 | bin/sqoop.sh client
默认sqoop开启ports 12000 and 12001 |
停止 | ./bin/sqoop.sh server stop |
Configure client to use your Sqoop server:
sqoop:000> set server --host your.host.com --port 12000 --webapp sqoop
显示版本：show version --all
显示连接器：show connector --all
创建连接：create connection --cid 1
Creating connection for connector with id 1
Please fill following values to create new connection object
Name: First connection
Configuration configuration
JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://mysql.server/database
Username: sqoop
Password: *****
JDBC Connection Properties:
There are currently 0 values in the map:
entry#
Security related configuration options
Max connections: 0
New connection was successfully created with validation status FINE and persistent id 1
显示连接：show connection
创建任务：create job --xid 1 --type import
sqoop:000> create job --xid 1 --type import
Creating job for connection with id 1
Please fill following values to create new job object
Name: First job
Database configuration
Table name: users
Table SQL statement:
Table column names:
Partition column name:
Boundary query:
Output configuration
Storage type:
0 : HDFS
Choose: 0
Output directory: /user/jarcec/users
New job was successfully created with validation status FINE and persistent id 1
Throttling resources
Extractors: 20
Loaders: 10
注意创建job过程中会出现Extractors跟Loaders分别对应map 跟reduce个数
启动任务：start job --jid 1
启动任务同步执行:start job --jid 1 -s
显示任务：status job --jid 1
显示所有任务：show job -a
停止任务：stop job --jid 1
克隆连接：clone connection --xid 1
克隆任务：clone job --jid 1
运行wordcount出现:Application application_1396260476774_0001 failed 2 times due to AM Container for appattempt_1396260476774_0001_000002 exited with exitCode: 1 due to: Exception from container-launch
查看
hadoop/logs/userlogs/application_1386683368281_0001/container_1386683368281_0001_01_000001/stderr
yarn配置修改完后，可以正常跑wordcount,sqoop还是提示Exception from container-launch: 这个时候把sqoop server 重启就行
导出数据出现异常
is running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical memory used; 1.6 GB of 6 GB virtual memory used. Killing container.
修改mapred-site.xml
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx8000m</value>
</property>
yarn-site.xml
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>8</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>2046</value>
</property>
使用sqoop导入数据时，当数据量变大时，在map/reduce的过程中就会提示 java heap space error。经过总结，解决方法有两个：
1、 修改每个运行子进程的jvm大小
修改mapred-site.xml文件，添加以下属性：
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx8000m</value>
</property>
<property>
<name>mapred.reduce.child.java.opts</name>
<value>-Xmx8000m</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx8000m</value>
</property>
2、 增加map数量，
sqoop job里设置Extractors与Loaders数量