Hadoop集成BTrace
1.BTrace
1.1BTrace部署
1.下载BTrace工具包,官网地址:https://kenai.com/projects/btrace/downloads/directory/releases
2.解压btrace-bin.tar.gz,命令如下:
tar -zxvf btrace-bin.tar.gz
3.配置环境变量
BTRACE_HOME=/usr/btrace-bin
PATH=$PATH:$BTRACE_HOME/bin
4.执行btrace(注意:$BTRACE_HOME/bin下btrace、btracec、btracer文件权限)命令如下,表示部署成功:
1.2BTrace脚本
BTrace测试脚本代码:
importcom.sun.btrace.annotations.*; import static com.sun.btrace.BTraceUtils.*; @BTrace
public class TraceMR
{
@TLS privatestaticlongmapperSetupStartTime = 0l; @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper",
method="run",
location=@Location(where=Where.BEFORE, value=Kind.CALL,clazz="/.*/", method="setup"))
public static void onMapper_run_Before_Call_setup() {
mapperSetupStartTime =timeNanos();
println(strcat("MAP\tSTARTUP_MEM\t",str(used(heapUsage()))));
} @OnMethod(clazz="org.apache.hadoop.mapreduce.Mapper",
method="run",
location=@Location(where=Where.AFTER, value=Kind.CALL,clazz="/.*/", method="setup"))
public static void onMapper_run_After_Call_setup() {
println(strcat("MAP\tSETUP\t",str(timeNanos() -mapperSetupStartTime)));
println(strcat("MAP\tSETUP_MEM\t",str(used(heapUsage()))));
}
}
1.3编译BTrace脚本
编译命令:btracec TraceMR.java
2.Hadoop配置
2.1分发文件
将BTrace文件:$BTRACE_HOME/build目录下的btrace-agent.jar和btrace-boot.jar,以及编译Btrace脚本后生成的class文件分发到各个子节点上。建议使用脚本分发,代码如下:
#!/usr/bin/envbash
#Usage:
# ./install_btrace.sh <slaves_file>
#
# where:
# slaves_file = File containing a list of slavemachines
#
#Example:
# ./install_btrace.sh /root/SLAVE_NAMES.txt
#Make sure we have all the arguments
if [$# -ne ]; then
echo "Usage: $0<slaves_file>"
echo " slaves_file = File containing a list of slave machines"
echo ""
exit -
fi #Get the slaves file
declareSLAVES_FILE=$;
iftest ! -e $SLAVES_FILE; then
echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting"
exit -
fi
#Get the slaves file
declareSLAVES_FILE=$;
iftest ! -e $SLAVES_FILE; then
echo "ERROR: The file '$SLAVES_FILE'does not exist. Exiting"
exit -
fi MASTER_BTRACE_DIR=/home/wangpeng/zhiming/usr/btrace/build
SLAVES_BTRACE_DIR=/home/wangpeng/zhiming/usr forslave in `cat "$SLAVES_FILE"`; do
{
printf "Installing on host:$slave\n"
ssh $slave "mkdir -p$SLAVES_BTRACE_DIR"
scp ${MASTER_BTRACE_DIR}/btrace-agent.jar$slave:$SLAVES_BTRACE_DIR/.
scp ${MASTER_BTRACE_DIR}/btrace-boot.jar$slave:$SLAVES_BTRACE_DIR/.
scp ${MASTER_BTRACE_DIR}/TraceMR.class$slave:$SLAVES_BTRACE_DIR/.
}
done
2.2配置Hadoop文件
在所有子节点的$HADOOP_HOME安装目录下找到mapred-site.xml文件,进行如下设置:
代码:
<property>
<name>mapred.map.child.java.opts</name>
<value>
-Xmx1024m -javaagent:/home/wangpeng/zhiming/usr/btrace/build/btrace-agent.jar =script=/home/wangpeng/zhiming/usr//TraceMR.class,stdout=true,noServer=true
</value>
</property>
2.3运行MR程序
1.准备一个写好的MR程序,如WordCount.java程序;
2.执行hadoop jar WordCount.jar WordCount /user/wzm/input /user/wzm/output。
2.4查看输出文件
在所有子节点的$HADOOP_HOME安装目录下找到logs目录中的stdout文件,执行命令:cat stdout,且有如下输出表示配置成功:
版权声明:本文为博主原创文章,未经博主允许不得转载。