本文介绍了在过滤结果HBase时出现OutOfOrderScannerNextException的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
List< Filter> andFilterList = new ArrayList<>();
SingleColumnValueFilter sourceLowerFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(source),CompareFilter.CompareOp.GREATER,Bytes.toBytes(lowerLimit));
sourceLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter sourceUpperFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(source),CompareFilter.CompareOp.LESS_OR_EQUAL,Bytes.toBytes(upperLimit));
sourceUpperFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetLowerFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(target),CompareFilter.CompareOp.GREATER,Bytes.toBytes(lowerLimit));
targetLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetUpperFilter = new SingleColumnValueFilter(Bytes.toBytes(cf),Bytes.toBytes(target),CompareFilter.CompareOp.LESS_OR_EQUAL,Bytes.toBytes(upperLimit));
targetUpperFilter.setFilterIfMissing(true);
andFilterList.add(sourceUpperFilter);
andFilterList.add(targetUpperFilter);
FilterList andFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL,andFilterList);
列表<过滤器> orFilterList = new ArrayList<>();
orFilterList.add(sourceLowerFilter);
orFilterList.add(targetLowerFilter);
FilterList orFilter = new FilterList(FilterList.Operator.MUST_PASS_ONE,orFilterList);
FilterList fl = new FilterList(FilterList.Operator.MUST_PASS_ALL);
fl.addFilter(andFilter);
fl.addFilter(orFilter);
扫描edgeScan = new Scan();
edgeScan.setFilter(fl);
ResultScanner edgeScanner = table.getScanner(edgeScan);
结果edgeResult;
logger.info(Writing edges ...); ((edgeResult = edgeScanner.next())!= null){
//某些代码
}
这段代码启动了这个错误:
org.apache.hadoop.hbase。 DoNotRetryIOException:在重试OutOfOrderScannerNextException之后失败:是否存在rpc超时?
,位于org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.writeFile(RDF2Subdue.java:
,位于org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:402) 150)
at org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.run(RDF2Subdue.java:39)
at org.deustotech.internet.phd.Main.main(Main.java: 32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo $ 1.run(ExecJavaMojo.java:297 )
在java.lang.Thread.run(Thread.java:745)
引起:org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException :预计nextCallSeq:1但下一个CallSeq从客户端:0; request = scanner_id:178 number_of_rows:100 close_scanner:false next_call_seq:0
在org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
在org.apache.hadoop。 hbase.protobuf.generated.ClientProtos $ ClientService $ 2.callBlockingMethod(ClientProtos.java:29497)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org .apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at org .apache.hadoop.hbase.ipc.SimpleRpcScheduler.access $ 000(SimpleRpcScheduler.java:39)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler $ 1.run(SimpleRpcScheduler.java:111)
在Sun.reflect.NativeConstructorAccessorImpl.newInstance0(本地方法)
在sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl。 java:57)
a (java.lang.reflect.Constructor.newInstance)($ Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException .instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException (ProtobufUtil.java:285)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
at org.apache.hadoop.hbase.client.ScannerCallable.call (ScannerCallable.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries (RpcRetryingCaller.java:90)
在org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:354)
... 9 more
引起:org。 apache.hadoop .hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException):org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:期望nextCallSeq:1但是nextCallSeq从客户端获得:0; request = scanner_id:178 number_of_rows:100 close_scanner:false next_call_seq:0
在org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
在org.apache.hadoop。 hbase.protobuf.generated.ClientProtos $ ClientService $ 2.callBlockingMethod(ClientProtos.java:29497)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org .apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at org .apache.hadoop.hbase.ipc.SimpleRpcScheduler.access $ 000(SimpleRpcScheduler.java:39)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler $ 1.run(SimpleRpcScheduler.java:111)
在java.lang.Thread.run(Thread.java:745)
在org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1453)
在org .apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1657)
在org.apache.hadoop.h base.ipc.RpcClient $ BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1715)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos $ ClientService $ BlockingStub.scan(ClientProtos.java:29900)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174)
... 13 more
RPC超时设置为600000.我已经尝试删除一些过滤器给出这些结果:
- sourceUpperFilter&& (sourceLowerFilter || targetLowerFilter) - >成功
- targetUpperFilter&& (sourceLowerFilter || targetLowerFilter) - >成功
- (sourceUpperFilter&& targetUpperFilter)&& (sourceLowerFilter) - >失败
- (sourceUpperFilter&& targetUpperFilter)&& (targetLowerFilter) - >失败
任何帮助,将不胜感激。谢谢。
解决方案
我通过设置 hbase.client.scanner.caching
I am trying to filter results in HBase this way:
List<Filter> andFilterList = new ArrayList<>();
SingleColumnValueFilter sourceLowerFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("source"), CompareFilter.CompareOp.GREATER, Bytes.toBytes(lowerLimit));
sourceLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter sourceUpperFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("source"), CompareFilter.CompareOp.LESS_OR_EQUAL, Bytes.toBytes(upperLimit));
sourceUpperFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetLowerFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("target"), CompareFilter.CompareOp.GREATER, Bytes.toBytes(lowerLimit));
targetLowerFilter.setFilterIfMissing(true);
SingleColumnValueFilter targetUpperFilter = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("target"), CompareFilter.CompareOp.LESS_OR_EQUAL, Bytes.toBytes(upperLimit));
targetUpperFilter.setFilterIfMissing(true);
andFilterList.add(sourceUpperFilter);
andFilterList.add(targetUpperFilter);
FilterList andFilter = new FilterList(FilterList.Operator.MUST_PASS_ALL, andFilterList);
List<Filter> orFilterList = new ArrayList<>();
orFilterList.add(sourceLowerFilter);
orFilterList.add(targetLowerFilter);
FilterList orFilter = new FilterList(FilterList.Operator.MUST_PASS_ONE, orFilterList);
FilterList fl = new FilterList(FilterList.Operator.MUST_PASS_ALL);
fl.addFilter(andFilter);
fl.addFilter(orFilter);
Scan edgeScan = new Scan();
edgeScan.setFilter(fl);
ResultScanner edgeScanner = table.getScanner(edgeScan);
Result edgeResult;
logger.info("Writing edges...");
while ((edgeResult = edgeScanner.next()) != null) {
// Some code
}
org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:402)
at org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.writeFile(RDF2Subdue.java:150)
at org.deustotech.internet.phd.framework.rdf2subdue.RDF2Subdue.run(RDF2Subdue.java:39)
at org.deustotech.internet.phd.Main.main(Main.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 178 number_of_rows: 100 close_scanner: false next_call_seq: 0
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29497)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:354)
... 9 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException): org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq got from client: 0; request=scanner_id: 178 number_of_rows: 100 close_scanner: false next_call_seq: 0
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3098)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29497)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:168)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:39)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:111)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1453)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1657)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1715)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29900)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:174)
... 13 more
The RPC timeout is set to 600000. I have tried to remove some filters given these results:
- sourceUpperFilter && (sourceLowerFilter || targetLowerFilter) --> Success
- targetUpperFilter && (sourceLowerFilter || targetLowerFilter) --> Success
- (sourceUpperFilter && targetUpperFilter) && (sourceLowerFilter) --> Fail
- (sourceUpperFilter && targetUpperFilter) && (targetLowerFilter) --> Fail
Any help would be appreciated. Thank you.
I solve this problem by setting hbase.client.scanner.caching
Hope this answer helps.
这篇关于在过滤结果HBase时出现OutOfOrderScannerNextException的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!