Saturday, September 6, 2014

Optimizing Apache Storm deployment

Recently we happen to run some pretty large Storm topologies in a Storm cluster which runs on Linux. When we running it there were two main issues occurred due to system limitations. First one was logged in Storm logs as,

“java.lang.OutOfMemoryError : unable to create new native Thread”.


We fixed this problem by increasing the Ulimit for Storm. Usually storm spawns processes with an user named Storm. So we have to increase the Ulimit for storm user. You can see the ulimits using command "ulimit -u". To increase the ulimits you can follow an approach like this.

The second problem was communication link failures between Storm nodes as well as communication link failures between other services (e.g. external APIs, databases, etc). To resolve this problem we had to enable tcp time wait reuse and also we increased the port range for tcp. hat can be done in following manner.

 Put these to /etc/sysctl.conf file and issue 'sysctl -f'
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.ip_local_port_range = 18000    65000

Note that we have to apply this setting for every node in Storm cluster. In addition to there were some errors like below due to Netty timeouts (Storm uses Netty as underlying messaging layer).

java.lang.RuntimeException: java.lang.RuntimeException: Client is being closed, and does not take requests any more
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107) ~[storm-core-0.9.1.2.1.4.0-632.jar:0.9.1.2.1.4.0-632]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:78)

This can be corrected by increasing netty timeout values in configurations.

2 comments:

  1. This is really one of the best blog on Optimizing Apache Storm deployment. I got many information form your blog thanks for sharing such a nice post.

    ReplyDelete
  2. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Storm , kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Sangita Mohanty
    MaxMunus
    E-mail: sangita@maxmunus.com
    Skype id: training_maxmunus
    Ph:(0) 9738075708 / 080 - 41103383
    http://www.maxmunus.com/

    ReplyDelete