Tez作业在由不同用户提交时失败

问题描述

配置了具有Kerberos安全性的Hadoop-2.6.0 HA群集。当使用tez-example-0.6.0.jar在不同用户的 yarn-tez 框架中提交示例作业时，获取下面的异常

异常

  java.io.IOException：临时目录hdfs上的所有权：// clustername / tmp / staging不符合预期。它由Kumar拥有。该目录必须由提交者TestUser或TestUser拥有
  

目录具有完全权限，但仍然获得以上例外。但是，当使用 yarn-tez 框架中的mapreduce-examples-2.6.0.jar提交作业时，作业成功完成。

命令

  yarn jar C ：\Tez\tez-examples-0.6.0.jar orderedwordcount -Dmapreduce.framework.name = yarn-tez / user / Kumar / names / user / Kumar / names1 
  

帮助将不胜感激。

解决方案

在您的tez-site.xml中设置以下内容：

 <性> 
< name> tez.staging-dir< / name> 
< value> / tmp / $ {user.name} / tez / staging /< / value> 
< / property> 
  

当前登台目录似乎已配置为使用/ clustername / tmp / staging /如果您使用/ clustername / tmp作为前缀，请务必预先创建tmp目录并拥有777个权限就可以了。

Configured Hadoop-2.6.0 HA cluster with Kerberos security. When submitting example job using tez-example-0.6.0.jar in yarn-tez framework from different user, getting the below exception

Exception

java.io.IOException: The ownership on the staging directory hdfs://clustername/tmp/staging is not as expected. It is owned by Kumar. The directory must be owned by the submitter TestUser or by TestUser

The directory has full permission but still getting the above exception. But when submitting a job using mapreduce-examples-2.6.0.jar in yarn-tez framework then the job completed successfully.

Command

yarn jar C:\Tez\tez-examples-0.6.0.jar orderedwordcount -Dmapreduce.framework.name=yarn-tez  /user/Kumar/names /user/Kumar/names1

Help would be greatly appreciated.

解决方案

Set the following in your tez-site.xml:

<property>
   <name>tez.staging-dir</name>
   <value>/tmp/${user.name}/tez/staging/</value>
</property>

The current staging directory seems to be configured to use /clustername/tmp/staging/ which ends up using the same path for multiple users causing permission conflicts.

If you use /clustername/tmp as the prefix, please be sure to pre-create the tmp dir and have 777 permissions on it.

这篇关于Tez作业在由不同用户提交时失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！