Hadoop集成BTrace

1.BTrace

1.1BTrace部署

1.下载BTrace工具包,官网地址:https://kenai.com/projects/btrace/downloads/directory/releases

2.解压btrace-bin.tar.gz,命令如下:

tar -zxvf btrace-bin.tar.gz

3.配置环境变量

BTRACE_HOME=/usr/btrace-bin

PATH=$PATH:$BTRACE_HOME/bin

4.执行btrace(注意:$BTRACE_HOME/bin下btrace、btracec、btracer文件权限)命令如下,表示部署成功:

BTrace追踪Hadoop部署-LMLPHP

1.2BTrace脚本

BTrace测试脚本代码:

 importcom.sun.btrace.annotations.*;

 import static com.sun.btrace.BTraceUtils.*;

 @BTrace
public class TraceMR
{
@TLS privatestaticlongmapperSetupStartTime = 0l; @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper",
method="run",
location=@Location(where=Where.BEFORE, value=Kind.CALL,clazz="/.*/", method="setup"))
public static void onMapper_run_Before_Call_setup() {
mapperSetupStartTime =timeNanos();
println(strcat("MAP\tSTARTUP_MEM\t",str(used(heapUsage()))));
} @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper",
method="run",
location=@Location(where=Where.AFTER, value=Kind.CALL,clazz="/.*/", method="setup"))
public static void onMapper_run_After_Call_setup() {
println(strcat("MAP\tSETUP\t",str(timeNanos() -mapperSetupStartTime)));
println(strcat("MAP\tSETUP_MEM\t",str(used(heapUsage()))));
}
}

1.3编译BTrace脚本

编译命令:btracec TraceMR.java

2.Hadoop配置

2.1分发文件

将BTrace文件:$BTRACE_HOME/build目录下的btrace-agent.jar和btrace-boot.jar,以及编译Btrace脚本后生成的class文件分发到各个子节点上。建议使用脚本分发,代码如下:

 #!/usr/bin/envbash
#Usage:
# ./install_btrace.sh <slaves_file>
#
# where:
# slaves_file = File containing a list of slavemachines
#
#Example:
# ./install_btrace.sh /root/SLAVE_NAMES.txt
#Make sure we have all the arguments
if [$# -ne ]; then
echo "Usage: $0<slaves_file>"
echo " slaves_file = File containing a list of slave machines"
echo ""
exit -
fi #Get the slaves file
declareSLAVES_FILE=$;
iftest ! -e $SLAVES_FILE; then
echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting"
exit -
fi
#Get the slaves file
declareSLAVES_FILE=$;
iftest ! -e $SLAVES_FILE; then
echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting"
exit -
fi MASTER_BTRACE_DIR=/home/wangpeng/zhiming/usr/btrace/build
SLAVES_BTRACE_DIR=/home/wangpeng/zhiming/usr forslave in `cat "$SLAVES_FILE"`; do
{
printf "Installing on host:$slave\n"
ssh $slave "mkdir -p$SLAVES_BTRACE_DIR"
scp ${MASTER_BTRACE_DIR}/btrace-agent.jar$slave:$SLAVES_BTRACE_DIR/.
scp ${MASTER_BTRACE_DIR}/btrace-boot.jar$slave:$SLAVES_BTRACE_DIR/.
scp ${MASTER_BTRACE_DIR}/TraceMR.class$slave:$SLAVES_BTRACE_DIR/.
}
done

2.2配置Hadoop文件

在所有子节点的$HADOOP_HOME安装目录下找到mapred-site.xml文件,进行如下设置:

BTrace追踪Hadoop部署-LMLPHP

代码:

 <property>
<name>mapred.map.child.java.opts</name>
<value>
-Xmx1024m -javaagent:/home/wangpeng/zhiming/usr/btrace/build/btrace-agent.jar =script=/home/wangpeng/zhiming/usr//TraceMR.class,stdout=true,noServer=true
</value>
</property>

2.3运行MR程序

1.准备一个写好的MR程序,如WordCount.java程序;

2.执行hadoop jar WordCount.jar WordCount /user/wzm/input /user/wzm/output。

2.4查看输出文件

在所有子节点的$HADOOP_HOME安装目录下找到logs目录中的stdout文件,执行命令:cat stdout,且有如下输出表示配置成功:

BTrace追踪Hadoop部署-LMLPHP

版权声明:本文为博主原创文章,未经博主允许不得转载。

05-04 04:27