配置以加速分布式模式下的拓扑

问题描述

I have a topology running with parallelism as (1,8,1)(spout,logic bolt, write bolt) with number of ackers set as 12( 12 are available slots in my cluster). The max spout pending is 200 and timeout.secs is 200. I have to process 14 lac inputs.

My cluster consist of 1 nimbus & 3 supervisors ( dual core , 4 GB each) SO currently it takes 44 hours to give the output for 14 lac inputs.

My application is running separately as an application on another server. I did this to separate the application from the storm jar because I suspected that application was reserving some memory. I ran 50 instances of the application so it can process 50 tuples at a time But even this didnt help.

with this configuration, the topology processes around 6000/6500 tuples in 10 mins Messages are not failing anywhere but overall latency 16000 ms. If i try increasing the paralelism the rate of topology reduces(4000/ 3000 in 10 mins).

The behaviour is not constanst so kindly help me to do the math.

我的尝试：



我已经在我的问题中提到了一切

What I have tried:

I have mentioned everything in my question

推荐答案

这篇关于配置以加速分布式模式下的拓扑的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！