Giraph

Giraph

关注
发信
关注(28)粉丝(399)

用Hadoop构建Giraph

本文介绍了用Hadoop构建Giraph的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述



我尝试了使用Hadoop 2.7.1
来设置Giraph。尝试尽我所能，似乎并不奏效。接下来的链接，









当我尝试使用 mvn -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests包 

  hadoopuser： / usr / local / giraph $ mvn -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests包
 [INFO]扫描项目... 
 [信息] --------- ------------------------------------------------ --------------- 
 [INFO] Reactor Build Order：
 [INFO] 
 [INFO] Apache Giraph Parent 
 [信息] Apache Giraph Core 
 [INFO] Apache Giraph示例
 [INFO] Apache Giraph分发
 [信息] 
 [信息] ------------- -------------------------------------------------- --------- 
 [INFO]构建Apache Giraph Parent 1.1.0 
 [INFO] -------------------- -------------------------------------------------- -  
 [INFO] 
 [INFO] --- mavanagaiata：0.5.0：分支（git-commit）@ giraph-parent --- 
 [信息] 
 [ INFO] --- mavanagaiata：0.5.0：commit（git-commit）@ giraph-parent --- 
 [INFO] 
 [INFO] --- maven-enforcer-plugin：1.2：enforce （默认）@ giraph-parent --- 
 [信息]跳过规则执行。 
 [INFO] 
 [INFO] --- maven-dependency-versions-check-plugin：2.0.2：check（default）@ giraph-parent --- 
 [INFO] 
 [INFO] --- maven-remote-resources-plugin：1.4：进程（默认）@ giraph-parent --- 
 [信息] 
 [信息] --- maven-dependency -plugin：2.8：仅用于分析（默认）@ giraph-parent --- 
 [INFO]跳过插件执行
 [INFO] 
 [INFO] --- maven-dependency-plugin ：2.8：analyze-duplicate（默认）@ giraph-parent --- 
 [INFO]跳过插件执行
 [INFO] 
 [INFO] --- maven-dependency-plugin：2.8 ：analyze-dep-mgt（默认）@ giraph-parent --- 
 [信息]跳过插件执行
 [信息] 
 [信息] --- maven-duplicate-finder-plugin ：1.0.4：check（默认）@ giraph-parent --- 
 [信息] 
 [信息] --- maven-site-plugin：3.3：附加描述符（附加描述符）@ giraph-parent --- 
 [信息] 
 [信息] ------------------------------ ------------------------------ ------------ 
 [INFO]构建Apache Giraph Core 1.1.0 
 [INFO] ----------------- -------------------------------------------------- ----- 
 [信息] 
 [信息] --- mavanagaiata：0.5.0：分支（git-commit）@ giraph-core --- 
 [信息] 
 [INFO] --- mavanagaiata：0.5.0：commit（git-commit）@ giraph-core --- 
 [信息] 
 [信息] --- maven-enforcer-plugin： 1.2：执行（默认）@ giraph-core --- 
 [信息]跳过规则执行。 
 [INFO] 
 [INFO] --- maven-dependency-versions-check-plugin：2.0.2：check（default）@ giraph-core --- 
 [INFO] 
 [INFO] --- munge-maven-plugin：1.0：munge（munge）@ giraph-core --- 
 [信息] 
 [信息] --- maven-remote-resources -plugin：1.4：process（默认）@ giraph-core --- 
 [信息] 
 [信息] --- maven-resources-plugin：2.6：资源（默认资源）@ giraph-核心--- 
 [信息]使用'UTF-8'编码来复制过滤的资源。 
 [INFO]复制2资源
 [INFO]复制3资源
 [INFO] 
 [信息] --- maven-compiler-plugin：3.0：compile（default-compile ）@ giraph-core --- 
 [INFO]检测到更改 - 重新编译模块！ 
 [INFO]将711个源文件编译到/ usr / local / giraph / giraph-core / target / munged / classes 
 [INFO] --------------- ---------------------------------------------- 
 [警告]编译警告：
 [信息] ------------------------------------- ------------------------ 
 [警告] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeByteArrayOutputStream.java：[42,29] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeByteArrayOutputStream.java：[44,25] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeByteArrayOutputStream.java：[57,32] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [警告] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / ut ils / UnsafeArrayReads.java：[39,32] sun.misc.Unsafe是内部专有API，可能会在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeArrayReads.java：[42,29] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeArrayReads.java：[44,25] sun.misc.Unsafe是内部专有API，可能会在未来版本中删除
 [WARNING ] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java：某些输入文件使用或覆盖弃用的API。 
 [警告] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java：使用-Xlint：deprecation重新编译以获取详细信息。 
 [警告] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java：某些输入文件使用未经检查或不安全的操作。 
 [警告] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java：使用-Xlint重新编译：取消选中以查看详细信息。 
 [信息] 10个警告
 [信息] ---------------------------------- --------------------------- 
 [INFO] ---------------- --------------------------------------------- 
 [错误]编译错误：
 [信息] -------------------------------------- ----------------------- 
 [错误] / usr / local / giraph / giraph-core / target / munged / main / org / apache /giraph/comm/netty/SaslNettyClient.java:[84,68]无法找到符号
符号：变量SASL_PROPS 
位置：类org.apache.hadoop.security.SaslRpcServer 
 [错误] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]找不到符号
符号：变量SASL_PROPS 
 location：class org.apache.hadoop.security.SaslRpcServer 
 [INFO] 2错误
 [INFO] --------------------- ---------------------------------------- 
 [INFO] --- -------------------------------------------------- ------------------- 
 [信息]反应堆摘要：
 [信息] 
 [信息]阿帕che Giraph Parent ............................... SUCCESS [4.374 s] 
 [INFO] Apache Giraph Core。 ................................ FAILURE [8.469 s] 
 [信息] Apache Giraph示例... .......................... SKIPPED 
 [INFO] Apache Giraph Distribution ............. ............ SKIPPED 
 [INFO] ------------------------------ ------------------------------------------ 
 [INFO] BUILD FAILURE 
 [INFO] ------------------------------------------ ------------------------------ 
 [信息]总时间：13.136 s 
 [信息]完成at：2015-11-16T10：19：07 + 05：30 
 [INFO] Final Memory：45M / 335M 
 [INFO] --------------- -------------------------------------------------- ------- 
 [错误]无法执行目标org.apache.maven.plugins：maven-compiler-plugin：3.0：编译（默认编译）项目giraph-core：编译失败：编译失败：
 [错误] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]无法找到符号
 [错误]符号：变量SASL_PROPS 
 [错误]位置：类org.apache.hadoop.security.SaslRpcServer 
 [错误] / usr / local / giraph / giraph-core /target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]找不到符号
 [错误]符号：变量SASL_PROPS 
 [错误] location：class org.apache.hadoop.security.SaslRpcServer 
 [错误]  - > [帮助1] 
 [错误] 
 [错误]要查看错误的完整堆栈跟踪，请使用-e开关重新运行Maven。 
 [错误]使用-X开关重新运行Maven以启用完整的调试日志记录。 
 [错误] 
 [错误]有关错误和可能解决方案的更多信息，请阅读以下文章：
 [错误] [帮助1] http://cwiki.apache.org / confluence / display / MAVEN / MojoFailureException 
 [错误] 
 [错误]纠正问题后，您可以使用命令
恢复编译[错误] mvn< goals> -rf：giraph-cor





当我尝试使用> mvn -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests install 

  hadoopuser：/ usr / local / giraph $ mvn -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests安装
 [信息]扫描项目... 
 [信息]  - -------------------------------------------------- --------------------- 
 [INFO] Reactor Build Order：
 [INFO] 
 [INFO] Apache Giraph Parent 
 [INFO] Apache Giraph Core 
 [INFO] Apache Giraph示例
 [INFO] Apache Giraph分发
 [信息] 
 [信息] ------- -------------------------------------------------- --------------- 
 [INFO]构建Apache Giraph Parent 1.1.0 
 [INFO] -------------- -------------------------------------------------- -------- 
 [INFO] 
 [INFO] --- mavanagaiata：0.5.0：分支（git-commit）@ giraph-parent --- 
 [INFO] 
 [INFO] --- mavanagaiata：0.5.0：commit（git-commit）@ giraph-parent --- 
 [信息] 
 [信息] --- maven -enforcer-plugin：1.2：执行（默认）@ giraph-parent --- 
 [INFO]跳过规则执行。 
 [INFO] 
 [INFO] --- maven-dependency-versions-check-plugin：2.0.2：check（default）@ giraph-parent --- 
 [INFO] 
 [INFO] --- maven-remote-resources-plugin：1.4：进程（默认）@ giraph-parent --- 
 [信息] 
 [信息] --- maven-dependency -plugin：2.8：仅用于分析（默认）@ giraph-parent --- 
 [INFO]跳过插件执行
 [INFO] 
 [INFO] --- maven-dependency-plugin ：2.8：analyze-duplicate（默认）@ giraph-parent --- 
 [INFO]跳过插件执行
 [INFO] 
 [INFO] --- maven-dependency-plugin：2.8 ：analyze-dep-mgt（默认）@ giraph-parent --- 
 [信息]跳过插件执行
 [信息] 
 [信息] --- maven-duplicate-finder-plugin ：1.0.4：check（默认）@ giraph-parent --- 
 [信息] 
 [信息] --- maven-site-plugin：3.3：附加描述符（附加描述符）@ giraph-parent --- 
 [INFO] 
 [INFO] --- maven-install-plugin：2.3.1：install（default-install）@ giraph-parent --- 
 [INFO]将/usr/local/giraph/pom.xml安装到/ ho me / hadoopuser / .m2 / repository / org / apache / giraph / giraph-parent / 1.1.0 / giraph-parent-1.1.0.pom 
 [INFO]安装/ usr / local / giraph / target / giraph -parent-1.1.0-site.xml添加到/home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0-site.xml 
 [信息] 
 [信息] ---------------------------------------- -------------------------------- 
 [INFO]构建Apache Giraph Core 1.1.0 
 [INFO] ----------------------------------------------- ------------------------- 
 [信息] 
 [信息] --- mavanagaiata：0.5.0：分支（ git-commit）@ giraph-core --- 
 [INFO] 
 [INFO] --- mavanagaiata：0.5.0：commit（git-commit）@ giraph-core --- 
 [INFO] 
 [信息] --- maven-enforcer-plugin：1.2：执行（默认）@ giraph-core --- 
 [信息]跳过规则执行。 
 [INFO] 
 [INFO] --- maven-dependency-versions-check-plugin：2.0.2：check（default）@ giraph-core --- 
 [INFO] 
 [INFO] --- munge-maven-plugin：1.0：munge（munge）@ giraph-core --- 
 [信息] 
 [信息] --- maven-remote-resources -plugin：1.4：process（默认）@ giraph-core --- 
 [信息] 
 [信息] --- maven-resources-plugin：2.6：资源（默认资源）@ giraph-核心--- 
 [信息]使用'UTF-8'编码来复制过滤的资源。 
 [INFO]复制2资源
 [INFO]复制3资源
 [INFO] 
 [信息] --- maven-compiler-plugin：3.0：compile（default-compile ）@ giraph-core --- 
 [INFO]检测到更改 - 重新编译模块！ 
 [INFO]将711个源文件编译到/ usr / local / giraph / giraph-core / target / munged / classes 
 [INFO] --------------- ---------------------------------------------- 
 [警告]编译警告：
 [信息] ------------------------------------- ------------------------ 
 [警告] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeByteArrayOutputStream.java：[42,29] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeByteArrayOutputStream.java：[44,25] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeByteArrayOutputStream.java：[57,32] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [警告] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / ut ils / UnsafeArrayReads.java：[39,32] sun.misc.Unsafe是内部专有API，可能会在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeArrayReads.java：[42,29] sun.misc.Unsafe是内部专有API，可能在未来版本中删除
 [WARNING] / usr / local / giraph / giraph-core / target / munged / main / org / apache / giraph / utils / UnsafeArrayReads.java：[44,25] sun.misc.Unsafe是内部专有API，可能会在未来版本中删除
 [WARNING ] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java：某些输入文件使用或覆盖弃用的API。 
 [警告] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java：使用-Xlint：deprecation重新编译以获取详细信息。 
 [警告] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java：某些输入文件使用未经检查或不安全的操作。 
 [警告] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java：使用-Xlint重新编译：取消选中以查看详细信息。 
 [信息] 10个警告
 [信息] ---------------------------------- --------------------------- 
 [INFO] ---------------- --------------------------------------------- 
 [错误]编译错误：
 [信息] -------------------------------------- ----------------------- 
 [错误] / usr / local / giraph / giraph-core / target / munged / main / org / apache /giraph/comm/netty/SaslNettyClient.java:[84,68]无法找到符号
符号：变量SASL_PROPS 
位置：类org.apache.hadoop.security.SaslRpcServer 
 [错误] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]找不到符号
符号：变量SASL_PROPS 
 location：class org.apache.hadoop.security.SaslRpcServer 
 [INFO] 2错误
 [INFO] --------------------- ---------------------------------------- 
 [INFO] --- -------------------------------------------------- ------------------- 
 [信息]反应堆摘要：
 [信息] 
 [信息]阿帕che Giraph Parent ............................... SUCCESS [4.360 s] 
 [INFO] Apache Giraph Core。 ................................失败[7.473 s] 
 [信息] Apache Giraph示例... .......................... SKIPPED 
 [INFO] Apache Giraph Distribution ............. ............ SKIPPED 
 [INFO] ------------------------------ ------------------------------------------ 
 [INFO] BUILD FAILURE 
 [INFO] ------------------------------------------ ------------------------------ 
 [信息]总时间：12.122 s 
 [信息]完成at：2015-11-16T10：26：04 + 05：30 
 [INFO] Final Memory：46M / 374M 
 [INFO] --------------- -------------------------------------------------- ------- 
 [错误]无法执行目标org.apache.maven.plugins：maven-compiler-plugin：3.0：编译（默认编译）项目giraph-core：编译失败：编译失败：
 [错误] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]无法找到符号
 [错误]符号：变量SASL_PROPS 
 [错误]位置：类org.apache.hadoop.security.SaslRpcServer 
 [错误] / usr / local / giraph / giraph-core /target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]找不到符号
 [错误]符号：变量SASL_PROPS 
 [错误] location：class org.apache.hadoop.security.SaslRpcServer 
 [错误]  - > [帮助1] 
 [错误] 
 [错误]要查看错误的完整堆栈跟踪，请使用-e开关重新运行Maven。 
 [错误]使用-X开关重新运行Maven以启用完整的调试日志记录。 
 [错误] 
 [错误]有关错误和可能解决方案的更多信息，请阅读以下文章：
 [错误] [帮助1] http://cwiki.apache.org / confluence / display / MAVEN / MojoFailureException 
 [错误] 
 [错误]纠正问题后，您可以使用命令
恢复编译[错误] mvn< goals> -rf：giraph-core 
 
 
 
 
 
 
 
 --- 
版本信息如下：
 
 Hadoop：
 
 hadoopuser：〜$ hadoop版本
 Hadoop 2.7.1 
 Subversion https://git-wip-us.apache.org/ repos / asf / hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a 
编译jenkins于2015-06-29T06：04Z 
用protoc编译2.5.0 
从校验和源代码fc0a1a23fc1868e4d5ee7fa2b28a58a 
此命令使用/usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar 
 
运行Giraph是从这里下载的版本1.1.0：http：//archive.apache .org / dist / giraph / giraph-1.1.0 / giraph-dist-1.1.0-src.tar.bz2 
 
 Java 
 
 hadoopuser：〜$ java -version 
 Java版本1.8.0_05
 Java™SE运行时环境（build 1.8.0_05-b13）
 Java HotSpot™64位服务器虚拟机（版本25.5-b02，混合模式）
 
 Maven 
 
 hadoopuser：〜$ mvn -version 
 Apache Mave 3.3.3（7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T17：27：37 + 05：30）
 Maven home：/opt/apache-maven-3.3.3 
 Java版本：1.8.0_05，供应商：Oracle Corporation 
 Java主页：/usr/lib/jvm/jdk1.8.0/jre 
默认语言环境：en_IN，平台编码：UTF-8 
操作系统名称：linux，版本：3.13.0-68-通用，arch：amd64，family：unix





编辑：



正如Steve Kaeser在他的我更改了我的pom.xlm，并更改了部分如下所示。

 < properties> 
< hadoop.version> SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION< /hadoop.version> 
< munge.symbols> PURE_YARN< /munge.symbols> 
 < project.enforcer.skip> true< /project.enforcer.skip> 
< giraph.maven.dependency.plugin.skip> true< /giraph.maven.dependency.plugin.skip> 
< giraph.maven.duplicate.finder.skip> true< /giraph.maven.duplicate.finder.skip> 
< / properties>

完成此操作后，当我运行 mvn -Phadoop_yarn -Dhadoop.version = 2.7 .1 -DskipTests install 我得到很多输出，并且失败。部分输出如下所示，

  [信息] 
 [信息] --- apache-rat-plugin ：0.8：check（默认）@ giraph-core --- 
排除：CODE_CONVENTIONS 
排除：** / target / ** 
排除：/排除：.checkstyle 
排除：.reviewboardrc 
排除：.git / ** 
 [INFO]排除：.bsb / ** 
 [信息]排除： gitignore 
排除：.arcconfig 
排除：* .patch 
 [信息]排除：.idea / ** 
 [信息]排除：** / * .iml 
 [INFO]排除：** / * .ipr 
 [信息]排除：** / test / resources / ** 
 [信息]排除：giraph-gora / conf / * 
 [信息]排除：giraph-rexster / giraph-kibble / src / main / resources / META-INF / services / com.tinkerpop.rexster.extension.RexsterExtension 
排除：src /site/resources/images/*.svg 
 [信息] 
 [信息]>>> findbugs-maven-plugin：2.5.1：check（default）> ：findbugs @ giraph-core>>> 
 [INFO] 
 [INFO] --- findbugs-maven-plugin：2.5.1：findbugs（findbugs）@ giraph-core --- 
 [INFO] Fork Value is true 
 [java]在分析过程中出现以下错误：
 [java] Nov 17，2015 3:16:36 PM edu.umd.cs.findbugs.TextUIBugReporter reportAnalysisError 
 [java] SEVERE：Error扫描java / util / Comparator的引用类
 [java] java.lang.ArrayIndexOutOfBoundsException：51966 
 [java] at org.objectweb.asm.ClassReader.readUTF8（Unknown Source）
 [java ] at org.objectweb.asm.ClassReader.readClass（Unknown Source）
 
。 
。 
。 
。 
。 
。 
。 
。 
 [INFO] Reactor摘要：
 [信息] 
 [信息] Apache Giraph Parent ....................... ........ SUCCESS [4.140 s] 
 [INFO] Apache Giraph核心........................... ...... FAILURE [37.541 s] 
 [INFO] Apache Giraph示例............................. SKIPPED 
 [INFO] Apache Giraph Distribution ......................... SKIPPED 
 [信息] ------ -------------------------------------------------- ---------------- 
 [INFO] BUILD FAILURE 
 [INFO] ------------------ -------------------------------------------------- ---- 
 [信息]总时间：41.972 s 
 [信息]完成于：2015-11-17T15：16：37 + 05：30 
 [INFO] Final Memory：68M / 537M 
 [信息] ----------------------------------------- ------------------------------- 
 [错误]无法执行目标org.codehaus.mojo：findbugs- maven-plugin：项目giraph-core上的2.5.1：findbugs（findbugs）：目标执行findbugs org.codehaus.mojo：findbugs-maven-plugin：2.5.1：findbugs失败：Java返回：1  - > [Help 1] 
 [错误]


解决方案

我今天遇到这个问题。最后，我需要两个补丁来修复它：


  1. 在基本目录中打开pom.xml并替换< munge.symbols> PURE_YARN，STATIC_SASL_SYMBOL< /munge.symbols> 带有：< munge.symbols> PURE_YARN< /munge.symbols>


  2. 在这里找到补丁：

    3. 使用 clean maven标志： mvn -Phadoop_yarn -Dhadoop.version = 2.7.3 clean package -DskipTests



I am trying to set up Giraph with Hadoop 2.7.1Try as I might, it doesn't seem to work.

I have tried following the below links,

What happens when I try to build using mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.374 s]
[INFO] Apache Giraph Core ................................. FAILURE [  8.469 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.136 s
[INFO] Finished at: 2015-11-16T10:19:07+05:30
[INFO] Final Memory: 45M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-cor

What happens when I try to build using, mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ giraph-parent ---
[INFO] Installing /usr/local/giraph/pom.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0.pom
[INFO] Installing /usr/local/giraph/target/giraph-parent-1.1.0-site.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0-site.xml
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.360 s]
[INFO] Apache Giraph Core ................................. FAILURE [  7.473 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.122 s
[INFO] Finished at: 2015-11-16T10:26:04+05:30
[INFO] Final Memory: 46M/374M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-core







---
Version information follows,

Hadoop:  

    hadoopuser:~$ hadoop version
    Hadoop 2.7.1
    Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
    Compiled by jenkins on 2015-06-29T06:04Z
    Compiled with protoc 2.5.0
    From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

Giraph is version 1.1.0 downloaded from here : http://archive.apache.org/dist/giraph/giraph-1.1.0/giraph-dist-1.1.0-src.tar.bz2

Java

    hadoopuser:~$ java -version
    java version "1.8.0_05"
    Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

Maven 

    hadoopuser:~$ mvn -version
    Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T17:27:37+05:30)
    Maven home: /opt/apache-maven-3.3.3
    Java version: 1.8.0_05, vendor: Oracle Corporation
    Java home: /usr/lib/jvm/jdk1.8.0/jre
    Default locale: en_IN, platform encoding: UTF-8
    OS name: "linux", version: "3.13.0-68-generic", arch: "amd64", family: "unix"

Edit:

As was suggested by Steve Kaeser in his answer I changed my pom.xlm and changed part is shown below.

  <properties>
<hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
    <munge.symbols>PURE_YARN</munge.symbols>
    <!-- TODO: add these checks eventually -->
    <project.enforcer.skip>true</project.enforcer.skip>
    <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
    <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
  </properties>

After doing this when I run mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install I get a lot of output and it fails. Parts of the output are shown below,

[INFO] 
[INFO] --- apache-rat-plugin:0.8:check (default) @ giraph-core ---
[INFO] Exclude: CODE_CONVENTIONS
[INFO] Exclude: **/target/**
[INFO] Exclude: /_bsp/**
[INFO] Exclude: .checkstyle
[INFO] Exclude: .reviewboardrc
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: .arcconfig
[INFO] Exclude: *.patch
[INFO] Exclude: .idea/**
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.ipr
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: giraph-gora/conf/*
[INFO] Exclude: giraph-rexster/giraph-kibble/src/main/resources/META-INF/services/com.tinkerpop.rexster.extension.RexsterExtension
[INFO] Exclude: src/site/resources/images/*.svg
[INFO] 
[INFO] >>> findbugs-maven-plugin:2.5.1:check (default) > :findbugs @ giraph-core >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
[INFO] Fork Value is true
     [java] The following errors occurred during analysis:
     [java] Nov 17, 2015 3:16:36 PM edu.umd.cs.findbugs.TextUIBugReporter reportAnalysisError
     [java] SEVERE: Error scanning java/util/Comparator for referenced classes
     [java] java.lang.ArrayIndexOutOfBoundsException: 51966
     [java]     at org.objectweb.asm.ClassReader.readUTF8(Unknown Source)
     [java]     at org.objectweb.asm.ClassReader.readClass(Unknown Source)

.
.
.
.
.
.
.
.
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.140 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 37.541 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.972 s
[INFO] Finished at: 2015-11-17T15:16:37+05:30
[INFO] Final Memory: 68M/537M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs (findbugs) on project giraph-core: Execution findbugs of goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs failed: Java returned: 1 -> [Help 1]
[ERROR]
解决方案

I ran into this issue today. Finally I needed two patches to fix it:

  1. Open pom.xml in base directory and replace
    <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols> with:
    <munge.symbols>PURE_YARN</munge.symbols>

  2. Apply patch found here: GIRAPH-1110.02.patch

  3. Build using the clean flag of maven:
    mvn -Phadoop_yarn -Dhadoop.version=2.7.3 clean package -DskipTests

这篇关于用Hadoop构建Giraph的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！

09-15 22:00
Powered by LMLPHP ©2024 Giraph 0.002030