本文介绍了在过滤结果HBase时出现OutOfOrderScannerNextException的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

  List< Filter> andFilterList = new ArrayList<>(); 
SingleColumnValueFilter sourceLowerFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(source),CompareFilter.CompareOp.GREATER,Bytes.toBytes(lowerLimit));
sourceLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter sourceUpperFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(source),CompareFilter.CompareOp.LESS_OR_EQUAL,Bytes.toBytes(upperLimit));
sourceUpperFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetLowerFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(target),CompareFilter.CompareOp.GREATER,Bytes.toBytes(lowerLimit));
targetLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetUpperFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(target),CompareFilter.CompareOp.LESS_OR_EQUAL,Bytes.toBytes(upperLimit));
targetUpperFilter.setFilterIfMissing(true);

andFilterList.add(sourceUpperFilter);
andFilterList.add(targetUpperFilter);

FilterList andFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL,andFilterList);

列表<过滤器> orFilterList = new ArrayList<>();
orFilterList.add(sourceLowerFilter);
orFilterList.add(targetLowerFilter);
FilterList orFilter = new FilterList(FilterList.Operator.MUST_PASS_ONE,orFilterList);

FilterList fl = new FilterList(FilterList.Operator.MUST_PASS_ALL);
fl.addFilter(andFilter);
fl.addFilter(orFilter);

扫描edgeScan = new Scan();
edgeScan.setFilter(fl);
ResultScanner edgeScanner = table.getScanner(edgeScan);
结果edgeResult;
logger.info(Writing edges ...); ((edgeResult = edgeScanner.next())!= null){
//某些代码
}

这段代码启动了这个错误:

  org.apache.hadoop.hbase。 DoNotRetryIOException:在重试OutOfOrderScannerNextException之后失败:是否存在rpc超时? 
,位于org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.writeFile(RDF2Subdue.java:
,位于org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:402) 150)
at org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.run(RDF2Subdue.java:39)
at org.deustotech.internet.phd.Main.main(Main.java: 32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo $ 1.run(ExecJavaMojo.java:297 )
在java.lang.Thread.run(Thread.java:745)
引起:org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException :预计nextCallSeq:1但下一个CallSeq从客户端:0; request = scanner_id:178 number_of_rows:100 close_scanner:false next_call_seq:0
在org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
在org.apache.hadoop。 hbase.protobuf.generated.ClientProtos $ ClientService $ 2.callBlockingMethod(ClientProtos.java:29497)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org .apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at org .apache.hadoop.hbase.ipc.SimpleRpcScheduler.access $ 000(SimpleRpcScheduler.java:39)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler $ 1.run(SimpleRpcScheduler.java:111)
在Sun.reflect.NativeConstructorAccessorImpl.newInstance0(本地方法)
在sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl。 java:57)
a (java.lang.reflect.Constructor.newInstance)($ Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException .instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException (ProtobufUtil.java:285)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
at org.apache.hadoop.hbase.client.ScannerCallable.call (ScannerCallable.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries (RpcRetryingCaller.java:90)
在org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:354)
... 9 more
引起:org。 apache.hadoop .hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:期望nextCallSeq:1但是nextCallSeq从客户端获得:0; request = scanner_id:178 number_of_rows:100 close_scanner:false next_call_seq:0
在org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
在org.apache.hadoop。 hbase.protobuf.generated.ClientProtos $ ClientService $ 2.callBlockingMethod(ClientProtos.java:29497)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org .apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at org .apache.hadoop.hbase.ipc.SimpleRpcScheduler.access $ 000(SimpleRpcScheduler.java:39)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler $ 1.run(SimpleRpcScheduler.java:111)
在java.lang.Thread.run(Thread.java:745)

在org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1453)
在org .apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1657)
在org.apache.hadoop.h base.ipc.RpcClient $ BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1715)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos $ ClientService $ BlockingStub.scan(ClientProtos.java:29900)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174)
... 13 more

RPC超时设置为600000.我已经尝试删除一些过滤器给出这些结果:




  • sourceUpperFilter&& (sourceLowerFilter || targetLowerFilter) - >成功

  • targetUpperFilter&& (sourceLowerFilter || targetLowerFilter) - >成功

  • (sourceUpperFilter&& targetUpperFilter)&& (sourceLowerFilter) - >失败

  • (sourceUpperFilter&& targetUpperFilter)&& (targetLowerFilter) - >失败



任何帮助,将不胜感激。谢谢。

解决方案

我通过设置 hbase.client.scanner.caching



另请参阅




I am trying to filter results in HBase this way:

List<Filter> andFilterList = new ArrayList<>();
SingleColumnValueFilter sourceLowerFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("source"), CompareFilter.CompareOp.GREATER, Bytes.toBytes(lowerLimit));
sourceLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter sourceUpperFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("source"), CompareFilter.CompareOp.LESS_OR_EQUAL, Bytes.toBytes(upperLimit));
sourceUpperFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetLowerFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("target"), CompareFilter.CompareOp.GREATER, Bytes.toBytes(lowerLimit));
targetLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetUpperFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("target"), CompareFilter.CompareOp.LESS_OR_EQUAL, Bytes.toBytes(upperLimit));
targetUpperFilter.setFilterIfMissing(true);

andFilterList.add(sourceUpperFilter);
andFilterList.add(targetUpperFilter);

FilterList andFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL, andFilterList);

List<Filter> orFilterList = new ArrayList<>();
orFilterList.add(sourceLowerFilter);
orFilterList.add(targetLowerFilter);
FilterList orFilter = new FilterList(FilterList.Operator.MUST_PASS_ONE, orFilterList);

FilterList fl = new FilterList(FilterList.Operator.MUST_PASS_ALL);
fl.addFilter(andFilter);
fl.addFilter(orFilter);

Scan edgeScan = new Scan();
edgeScan.setFilter(fl);
ResultScanner edgeScanner = table.getScanner(edgeScan);
Result edgeResult;
logger.info("Writing edges...");
while ((edgeResult = edgeScanner.next()) != null) {
    // Some code
}

This code launchs this error:

org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:402)
    at org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.writeFile(RDF2Subdue.java:150)
    at org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.run(RDF2Subdue.java:39)
    at org.deustotech.internet.phd.Main.main(Main.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 178 number_of_rows: 100 close_scanner: false next_call_seq: 0
    at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29497)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
    at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
    at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
    at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
    at java.lang.Thread.run(Thread.java:745)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:354)
    ... 9 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException): org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 178 number_of_rows: 100 close_scanner: false next_call_seq: 0
    at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29497)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
    at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
    at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
    at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
    at java.lang.Thread.run(Thread.java:745)

    at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1453)
    at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1657)
    at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1715)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29900)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174)
    ... 13 more

The RPC timeout is set to 600000. I have tried to remove some filters given these results:

Any help would be appreciated. Thank you.

I solve this problem by setting hbase.client.scanner.caching

see also

Hope this answer helps.

这篇关于在过滤结果HBase时出现OutOfOrderScannerNextException的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

05-26 17:30
查看更多