Camel JDBC StreamList查询似乎在拆分之前加载了整个结果集

问题描述

我正在运行一个SQL使用者，以从表中读取更改，这一切都很好.但是，在某些情况下，质量会发生大量变化，然后我的查询会因您可能期望的错误而中断，出现内存不足错误.

I'm running a SQL consumer to read changes from a table, which is all well and good. However, there are occasions where changes happen on mass, and then my query breaks with out of memory error, as you might expect.

不幸的是，我坚持使用Camel 2.17.6，因此SQL组件的StreamList选项不可用. (尽管根据 Camel-SQL为什么使用StreamList似乎可以加载所有ResultSet吗?由于Spring JDBC的限制，这不能用作流列表.)

Unfortunately, I'm stuck on Camel 2.17.6, so the StreamList option for the SQL component isn't available. (Although according to Camel-SQL Why using StreamList seems to load all ResultSet? this doesn't work as a stream list due to Spring JDBC limitations.)

因此，我已经使用支持流列表的JDBC组件重新编写了路由，并且一旦增加要提取的记录数量，我仍然会遇到内存异常.出于某种原因，看来JDBC组件试图在将所有记录传递到拆分器之前提取所有记录.

So I've re-written my route using the JDBC component, which supports a stream list, and I'm still getting out of memory exceptions as soon as I increase the number of records to extract. It would appear that for some reason, the JDBC component is trying to extract all the records before passing to the splitter.

我现在所拥有的是以下形式:

What I have now is of the form:

from("timer:timer...")
  .to( "language:constant:resource:classpath:pathToSqlStatement/sqlStatement.sql" )
  .to( "jdbc:msSqlServerDataSource?outputType=StreamList" )
  .split( body() ).streaming()
  .setBody().simple("$body[XMLDOC]")
  .setHeader("HeaderName").xpath("xpath/to/data")
  .to("jms:topic:name");

我本来确实有一个聚合策略UseLatestAggregationStrategy，并且在split()之后又做了一个额外的步骤，但是我将其删除，试图删除所有可能导致整个查询保留在内存中的内容，但是我无法看到我现在还能做些什么.

I did originally have an aggregation strategy UseLatestAggregationStrategy and an extra step after the split() but I've stripped that out in an attempt to remove everything that could possibly result in the whole query being held in memory, but I can't see what else I can do now.

我注意到问题 camel jdbc内存不足异常提出了类似的问题问题，并且似乎没有解决方法.

I note the question camel jdbc out of memory exception raises a similar problem, and didn't appear to have a resolution.

(我应该注意，我曾经遇到的内存不足错误出现在不同的位置，并且在WinNTFileSystem中包含了我不理解的GC overhead limit exceeded，以及与ZippedInputStream有关的其他内容，我不明白.)

(I should note that the out of memory errors I've had do appear in different places, and included GC overhead limit exceeded at WinNTFileSystem which I don't understand, and something else to do with a ZippedInputStream, which again I don't understand.)

这是否意味着StreamList在JDBC组件上也不起作用，还是我必须做一些特定的事情以确保JDBC组件不会尝试缓存整个结果?

Does that mean that StreamList doesn't work on the JDBC component either, or do I have to do something specific to ensure that the JDBC component doesn't try to cache the whole results?

推荐答案

StreamList输出类型.在早期版本中，camel-sql组件加载在内存中设置为列表的结果.我认为在camel-sql 2.17.x版中无法避免.

StreamList output type is supported in camel-sql since version v.18.x. In earlier version, camel-sql component load the result set in memory as list. I don't think it can be avoided in camel-sql version 2.17.x.

通过骆驼SQL组件将结果加载到内存中后，聚合/拆分确实适用.

Aggregation /splitting does apply after loading the result in memory by camel-sql component.

另一个相关的答案.

这篇关于Camel JDBC StreamList查询似乎在拆分之前加载了整个结果集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！