Optimizing Apache Storm deployment

By -
Recently we happen to run some pretty large Storm topologies in a Storm cluster which runs on Linux. When we running it there were two main issues occurred due to system limitations. First one was logged in Storm logs as,

“java.lang.OutOfMemoryError : unable to create new native Thread”.
We fixed this problem by increasing the Ulimit for Storm. Usually storm spawns processes with an user named Storm. So we have to increase the Ulimit for storm user. You can see the ulimits using command "ulimit -u". To increase the ulimits you can follow an approach like this.

The second problem was communication link failures between Storm nodes as well as communication link failures between other services (e.g. external APIs, databases, etc). To resolve this problem we had to enable tcp time wait reuse and also we increased the port range for tcp. hat can be done in following manner.

 Put these to /etc/sysctl.conf file and issue 'sysctl -f'
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.ip_local_port_range = 18000    65000

Note that we have to apply this setting for every node in Storm cluster. In addition to there were some errors like below due to Netty timeouts (Storm uses Netty as underlying messaging layer).

java.lang.RuntimeException: java.lang.RuntimeException: Client is being closed, and does not take requests any more
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107) ~[storm-core-0.9.1.2.1.4.0-632.jar:0.9.1.2.1.4.0-632]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:78)

 This can be corrected by increasing netty timeout values in configurations.

Comments

  1. streamanalytixSeptember 10, 2016 at 5:18 PM

    This is really one of the best blog on Optimizing Apache Storm deployment. I got many information form your blog thanks for sharing such a nice post.

    ReplyDelete
  2. UnknownMay 18, 2017 at 1:42 PM

    I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Storm , kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Sangita Mohanty
    MaxMunus
    E-mail: sangita@maxmunus.com
    Skype id: training_maxmunus
    Ph:(0) 9738075708 / 080 - 41103383
    http://www.maxmunus.com/

    ReplyDelete

Post a Comment

Popular posts from this blog

Fixing Error "Failed to load VMMR0.r0 (VERR_SUPLIB_WORLD_WRITABLE)" in Virtualbox

By -
Recently I  to installed Oracle Virtualbox on my Ubuntu 12.04 computer to run Hortanworks sandbox. When I am trying to start the Sanbox it gave me an error like this. Failed to load VMMR0.r0 (VERR_SUPLIB_WORLD_WRITABLE). Unknown error creating VM (VERR_SUPLIB_WORLD_WRITABLE). What it says is /usr/bin forlder is world writable. Usually this is not world writable. But somehow I have unintentionally changed the permissions. I fixed this using the following command. chmod o-w /usr/bin
Read more

Create new Java Class files at runtime

By -
When we write programs usual method is to write all the code compile it and run. But when we try to complex things some times we may want to generate a new class and use it at runtime. As an example I wanted to create a class which parse a xml message and create a new Java object from that data. This is used in web server. So that result object varies according to the web service which is deployed. The solution was to inspect the service at deployment time and create a new parser class according to that in the service deployment time. It has to be noted that this parser had to be generated as a .class file and dynamically added to the classpath. So the result was to use a bytecode manupulation library. There are few bytecode manuplulation libraries available as ASM and JavaAssist . I used JavaAssist for this. The reason for using JavaAssist is it is simple to create a new class from the scratch using JavaAssit. When we create new class files using JavaAssit we have to create all t...
Read more

Using iReport plugin for Jasper Reports in Netbeans

By -
Image
Jasper reports is a very useful frame work for genarating reports, saving them as PDF,etc. iReport make it more easy because we can design reports using graphical methods from it. Also it is available as a plugin for NetBeans. You can download iReport Netbeans plugin from here . there are for files in the Zip folder. You can install them by at one by selecting all of them when you select the plugin to install from Netbeans (If you don't know how to install a plugin for Netbeans search for a tutorial). To use Jasper Reports in your application you have to add some libraries. But the problem is these libraries are heavily coupled with each other. So downloading each one is a very difficult task because there are problems like versions don't match, some libraries are outdated, etc.etc. There is a easy method. After installing the plugin Netbeans will automatically add the Jasper Reports library to its library collection. But it is missing one important...
Read more