hadoop - Sqoop与SQL Anywhere 17连接

我试图让sqoop与sqlanywhere17的sajdbc4.jar一起使用。我不断收到库错误，因为该命令无法找到jar需要的与.so相关的文件。

现在，我在sajdbc4.jar -> /opt/sqlanywhere17/java/sajdbc4.jar内创建了一个指向/usr/hdp/current/sqoop-client/lib/的符号链接(symbolic link)。

ln -s  /opt/sqlanywhere17/java/sajdbc4.jar /usr/hdp/current/sqoop-client/lib/sajdbc4.jar

我遇到此错误:



在this guide和this documentation之后:

cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bashrc
cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bash_profile

并检查echo $LD_LIBRARY_PATH:

/opt/sqlanywhere17/lib64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/client:/opt/sqlanywhere17/bin64/jre180/lib/amd64/server:/opt/sqlanywhere17/bin64/jre180/lib/amd64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads:

并检查java.library.path中的:/opt/sqlanywhere17/lib64

     java -XshowSettings:properties
Property settings:
    awt.toolkit = sun.awt.X11.XToolkit
    com.sap.vm.codeline = 81_REL
    com.sap.vm.compressedoops = true
    com.sap.vm.profilingserver = true
    com.sap.vm.type = opt
    com.sap.vm.version = 10
    file.encoding = UTF-8
    file.encoding.pkg = sun.io
    file.separator = /
    java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
    java.awt.printerjob = sun.print.PSPrinterJob
    java.class.path = .
    java.class.version = 52.0
    java.endorsed.dirs = /opt/sqlanywhere17/bin64/jre180/lib/endorsed
    java.ext.dirs = /opt/sqlanywhere17/bin64/jre180/lib/ext-sap
        /opt/sqlanywhere17/bin64/jre180/lib/ext
        /usr/java/packages/lib/ext
    java.home = /opt/sqlanywhere17/bin64/jre180
    java.io.tmpdir = /tmp
    java.library.path = /opt/sqlanywhere17/lib64
        /opt/sqlanywhere17/bin64/jre180/lib/amd64/client
        /opt/sqlanywhere17/bin64/jre180/lib/amd64/server
        /opt/sqlanywhere17/bin64/jre180/lib/amd64
        /opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads

        /usr/java/packages/lib/amd64
        /usr/lib64
        /lib64
        /lib
        /usr/lib

完整错误:

    19/10/22 08:30:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.3.0-37
19/10/22 08:30:07 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/10/22 08:30:07 INFO manager.SqlManager: Using default fetchSize of 1000
Exception in thread "main" java.lang.UnsatisfiedLinkError: no dbjdbc17 in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at sap.jdbc4.sqlanywhere.IDriver.try_load(IDriver.java:451)
        at sap.jdbc4.sqlanywhere.IDriver.<clinit>(IDriver.java:413)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:873)
        at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
        at org.apache.sqoop.tool.EvalSqlTool.run(EvalSqlTool.java:64)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
[root@server1 dag_scripts]# echo $LD_LIBRARY_PATH
/opt/sqlanywhere17/lib64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/client:/opt/sqlanywhere17/bin64/jre180/lib/amd64/server:/opt/sqlanywhere17/bin64/jre180/lib/amd64:/opt/sqlanywhere17/bin64/jre180/lib/amd64/native_threads:

我发现的唯一解决方法是在运行命令之前更改目录，这可行:

 cd /opt/sqlanywhere17/lib64
 sqoop eval --connect 'jdbc:sqlanywhere:Host=xxx.xx.xx.xxx:2638' --driver sap.jdbc4.sqlanywhere.IDriver --username user --password-file /tmp/pwfile --query "SELECT * from db.table"

最佳答案

该解决方案似乎可以立即解决该问题:

根据32位/ 64位要求选择/opt/sqlanywhere17/lib32或/opt/sqlanywhere17/lib64/

创建到当前JDK路径和JVM路径的符号链接(symbolic link)

如果以root

身份安装，则可以访问/opt/sqlanywhere17/java/的打开权限

确保用户属于“hadoop”组-获得sqoop文件夹权限(.env)

向运行sqoop命令的任何用户添加默认导出

ln -s /opt/sqlanywhere17/lib64/* /usr/jdk64/jdk1.8.0_112/jre/lib/amd64/

ln -s /opt/sqlanywhere17/lib64/* /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64/jre/lib/amd64/

chmod 755 -R /opt/sqlanywhere17/*
usermod -a -G hadoop user

#For user executing sqoop:
cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bashrc
cat /opt/sqlanywhere17/bin64/sa_config.sh >> ~/.bash_profile