正常情况下,什么时候上报blocks,是由NameNode通过回复心跳响应的方式触发的。

一次机房搬迁中,原机房hadoop版本为2.7.2,新机房版本为2.8.0,采用先扩容再缩容的方式搬迁。由于新旧机房机型不同和磁盘数不同,操作过程搞混过hdfs-site.xml,因为两种不同的机型,hdfs-site.xml不便做到相同,导致了NameNode报大量“missing block”。

然而依据NameNode所报信息,在DataNode能找到那些被标记为“missing”的blocks。修复配置问题后,“missing block”并没有消失。结合DataNode源代码,估计是因为DataNode没有向NameNode上报blocks。

结合DataNode的源代码,发现了HDFS自带的工具triggerBlockReport,它可以强制指定的DataNode向NameNode上报块,使用方法为:

hdfs dfsadmin -triggerBlockReport datanode_host:ipc_port

如:hdfs dfsadmin -triggerBlockReport 192.168.31.35:50020

正常情况下NameNode启动时,会要求DataNode上报一次blocks(通过fullBlockReportLeaseId值来控制),相关源代码如下:

DataNode相关代码(BPServiceActor.java):

private void offerService() throws Exception {

    HeartbeatResponse resp = sendHeartBeat(requestBlockReportLease); // 向NameNode发向心跳

    long fullBlockReportLeaseId = resp.getFullBlockReportLeaseId(); // 心跳响应

    boolean forceFullBr = scheduler.forceFullBlockReport.getAndSet(false); // triggerBlockReport强制上报仅一次有效

    if (forceFullBr) {

        LOG.info("Forcing a full block report to " + nnAddr);

    }

    if ((fullBlockReportLeaseId != 0) || forceFullBr) {

        cmds = blockReport(fullBlockReportLeaseId);

        fullBlockReportLeaseId = 0;

    }

}

// NameNode相关代码(FSNamesystem.java):

/**

* The given node has reported in.  This method should:

* 1) Record the heartbeat, so the datanode isn't timed out

* 2) Adjust usage stats for future block allocation



* If a substantial amount of time passed since the last datanode 

* heartbeat then request an immediate block report.  



* @return an array of datanode commands 

* @throws IOException

*/

HeartbeatResponse handleHeartbeat(DatanodeRegistration nodeReg,

  StorageReport[] reports, long cacheCapacity, long cacheUsed,

  int xceiverCount, int xmitsInProgress, int failedVolumes,

  VolumeFailureSummary volumeFailureSummary,

  boolean requestFullBlockReportLease) throws IOException {

    readLock();

    try {

        //get datanode commands

        final int maxTransfer = blockManager.getMaxReplicationStreams() - xmitsInProgress;

        DatanodeCommand[] cmds = blockManager.getDatanodeManager().handleHeartbeat(

            nodeReg, reports, blockPoolId, cacheCapacity, cacheUsed,

            xceiverCount, maxTransfer, failedVolumes, volumeFailureSummary);

        long fullBlockReportLeaseId = 0;

        if (requestFullBlockReportLease) {

            fullBlockReportLeaseId =  blockManager.requestBlockReportLeaseId(nodeReg);

        }

        //create ha status

        final NNHAStatusHeartbeat haState = new NNHAStatusHeartbeat(

            haContext.getState().getServiceState(),

            getFSImage().getCorrectLastAppliedOrWrittenTxId());

        return new HeartbeatResponse(cmds, haState, rollingUpgradeInfo, fullBlockReportLeaseId);

    } finally {

        readUnlock("handleHeartbeat");

    }

}

05-12 17:15