Whirlpool: Microservices Using Netty And Kafka

John Boardman Java, Technology Snapshot, Tutorial 2 Comments


Update! Whirlpool now uses the “just released” Netty version 4.1.3. The great news about this is the upgrade required zero code changes, just update the pom and rebuild!

In my last blog, I introduced Netty being used as a web server. That example worked well… as long as a broadcast server is what was needed.

Most of the time that is not very useful. It’s more likely that the need is for each client to receive only the data intended for them, with broadcasts reserved for special circumstances like “The server is going down in 15 minutes!” The other thing about that particular server example was that everything was self-contained. Monolithic applications are fine for examples, but in today’s environments, distributed microservices are much better. Scalability and reliability are paramount.

Netty and Kafka are fantastic together. Netty is great at handling a boatload of clients, and Kafka is great at enabling a boatload of services to work together. Combined, they are a sweet spot in development. However, there are some “gotchas” that can make it cumbersome. This blog, along with the example microservice/Netty architecture and fully working code, will hopefully help to alleviate the irritations and enable the sweetness.

First Things First

The code for the example is located here.

There is a detailed README that describes what is needed to set up the environment. I tried to keep the requirements at a minimum, just Java 8 and Maven. SLF4J and Logback are used for logging. I set up scripts for Mac OSX and Ubuntu (14.04 running in a Parallels container is what I tested with), so apologies if you develop on Windows. The code is all Java, and I’ve seen Kafka tutorials out there for Windows, so everything should run there. The Maven build should also produce targets that can be started, so with a little elbow grease installing Zookeeper/Kafka (you can follow the script to see what settings are needed), it shouldn’t be a huge deal to get it running manually on Windows.

NOTE: As explained in the, the script will remove any existing Zookeeper/Kafka installation and data. If you have an existing setup, don’t use the script!

After installing and configuring the pre-requisites, either run mvn package if you aren’t using the script, or (or if you are. The script downloads (if it hasn’t yet) Zk/Kafka, installs them, configures them, starts them, runs mvn package, starts the services, and finally starts the server. Once it starts, resist the urge to navigate away from the shell, because it automatically pops new tabs for each part of the architecture. After the Whirlpool server starts, then you are ready to go.

I highly recommend creating a script that installs, configures, builds, and starts up your microservice environment locally. Creating each individual service is a big pain. Docker could also be used if necessary, but I find it requires far less downloading to just run everything natively.

As a teaser, here is the UI (you can also see this from the on GitHub).


  • To add a stock symbol, type it in (i.e. “GOOG”) and click the A button under “Stock”. To remove it, click the X.
  • To add a website to test whether it is up or down, type in the fully-qualified URL (i.e. and click the A button under “UpDown”. To remove it, click the X.
  • To add a weather check, type the city,state in (i.e. “chicago,il”) and click the A button under “City,State”. To remove it, click the X.
  • Subscriptions survive page refresh and even login/logout (with the same userid) because they are stored with each service in memory. A “real” system would of course use a database.
  • Subscriptions are updated every 10 seconds so I don’t overwhelm the Yahoo APIs, so be patient after adding data.


With this example I was trying to think of good generic services that might be useful. I ended up choosing a stock quote service, an “is this website up or down” service, and a weather service. Each of these runs independently of the others with their own Kafka topics.

The way I chose to configure Kafka was with a command topic per service and a data topic per service. Everything could also just use a single global topic with the readers deciding what to process, but separating things out makes it more clear and clean.

Here’s a diagram of how data flows through Kafka. It was done with a free Keyhole web-based utility called Mockola. Notice that the server knows about all topics, but the services only know about their own topics. The cmd topics are used to send commands to the services, whereas the data topics (those without -cmd on them) are used to send data from the services. Again, all of this could be handled on a single bus topic, but it is much easier to see what is going on by separating them out.



Now let’s talk about the services. All three are very similar, so there is a base service that does most of the work. Each service has three threads, handled by Java ExecutorService. One nice thing about the Executor service is that it automatically restarts the thread if something goes wrong. This helps resiliency.

Each service starts itself by telling the base class what topic and command topic to use. The base class then starts the three threads: one for reading commands off of the cmd topic, one for periodically collecting data for clients, and one for sending data on the data topic. These threads communicate using the non-blocking Java concurrency classes ConcurrentLinkedQueue and ConcurrentHashMap. The hash map stores per-user sets of subscriptions, and the queue stores responses ready to be sent to the data topic.

The flow for each service is the three threads working concurrently. The Reader uses a Kafka Consumer to read commands from its command topic. Based on the command, the subscription is added or removed. This thread is fairly dumb in that it doesn’t ask the service to do any validation on the request, it just blindly adds whatever is sent to the subscription. Production code would obviously add a call to ask the service to validate the command before allowing the subscription to succeed. A response is created to put on the topic and then it waits for the next command.

NOTE: A few words about data being placed on topics. I’m using JSON as a transport format, but XML or anything else you want will work too. The important thing is that everyone agrees on the data format and sticks to it. The common module has POJO classes that define the contracts that the data will conform to. Things that are generally useful for all messages are a timestamp, the message type, and the id of the client.

Another useful thing would be an expiration timestamp. These example messages just live forever. The Message class only looks at the type and the id of a message. This is used by the server to determine what kind of message needs to be processed, and who is interested in the message. Without these it is very difficult if not impossible to process data. Now, message formats can get quite involved, with some using headers and sections to describe complex data. This example attempts to keep everything as simple as possible.

Netty Server

Let’s go through the server one class at a time.


This class is mostly unchanged from the prior blog. The reusable pieces have been moved to the WebSocketHelper class. The main use of this file is to serve up files that are asked for by the browser.


The first item that might be confusing is the class variable clientAttr. Storing data in a Netty Channel requires it to be attached to an AttributeKey. This is similar to an Atomic instance from the Java concurrent classes – it provides a container for data. We will store the client id (in our case the username, but it could just as easily be a session id) so we can figure out which Channel needs to receive messages.

The realWriteAndFlush() method sets the appropriate headers, the content length, and the cookie. It then writes and flushes the HTTP response. the line


tells Netty that this is the end of the data that needs to be written to the client, so Netty will send it out.

SPECIAL NOTE: Regarding cookie creation, make sure the HTTP Only flag is NOT set. If it is, JavaScript cannot see the cookie, nor will it be sent with the WebSocket Upgrade Request. This makes it so that you have to create your own method of page refresh management and session management.

The other thing about cookies is to use the STRICT version of the Netty cookie encoder so it will not allow multiple cookies with the same name. I’m not sure when it would be useful to allow this situation to occur.


This class just defines an interface that WhirlpoolServerHandler uses to talk to the WhirlpoolMessageHandler.


This is where the connection exists between Netty and Kafka. Two Executors handle a reader thread and a writer thread.

The writer thread looks for messages in the request queue (more on where those messages come from in a minute), and places the messages on the appropriate Kafka command topic.

The reader thread looks for incoming messages on the Kafka data topics, looks up the correct Channel for each topic, and writes the messages to those topics.

When the client sends a message over WebSockets, WhirlpoolServerHandler will make sure a complete message has arrived, and then call handleMessage(). This method figures out if it is a valid message, then adds the request to the request queue so the reader thread can pick it up and give it to Kafka.


There are several interesting things in this class. First, it can tell the difference between a HTTP, REST, and WebSocket message. The Netty overridden method that does this is channelRead0. This is the method Netty uses to tell us when a message arrives, and what type of message it is. For HTTP and REST calls, handleHttpRequest is called, and for websockets, handleWebSocketFrame is called.

The method handleHttpRequest reads the cookie if one is present. On POSTs it looks for login and logout. For login, it figures out the username/password, creates the cookie, and prevents multiple logins with the same name. All that code would be split out with additional security added in a production version of the application. For logout, it looks up the Channel, cleans it up, closes it, and expires the cookie.

For a WebSocketUpgrade, it asks Netty to handle the complex handshake required to get a websocket going. When that is complete, it adds the user to the Channel that was created during the handshake. This is where the user is connected to the Channel, and wouldn’t be very easy if the cookie did not come across in the request.

The only other thing to note here is that this class is set up to handle clients coded for SPA (single-page application) as it will redirect any unrecognized call to index.html.

The other methods in the class are more for informational purposes and would be used in advanced situations.


This class starts up the Netty server and creates the channel pipeline. It is a standard class for Netty that follows the Netty examples.

Final Thoughts

Obviously a whole lot more could go into this code. Multiple instances of each service and the server could be running at the same time, and Zk/Kafka could be clustered to help with resiliency. A great utility that tests the resiliency of microservice applications is another free open source Keyhole utility called TroubleMaker. I haven’t had the chance to test this example yet, but I am looking forward to the opportunity.

We didn’t touch on security, and although I previously hoped to show integration of Netty with Shiro, that is a very complex topic. All I can say about it is that it is possible, but I haven’t wrapped my head around all the parts enough to formulate a coherent blog yet.

I hope you have enjoyed the blog and find the code useful. Contact me through the blog or Twitter (@johnwboardman where I always appreciate new follows).

About the Author
John Boardman

John Boardman


John is a Sr. Keyhole Consultant with 20+ years of experience in C, C++, Java, and IoT enterprise software design and development. Also currently writing a multi-platform, multi-user game in Unity3d (and server in Java) and have written custom graphical game engine clients in C and C++ on several platforms.

Share this Post

Comments 2

  1. Philippe Derome

    This example is an ideal tradeoff between too little detail and too much. It has enough to show the way as a possible microarchitecture with specific details on how to use Kafka , Netty, JS, proper shell commands, and WebSockets for server-browser communication.

    What is nice is also the adoption of modern revisions/releases for the various components: for instance Kafka is at version 9 and takes advantage if a subtle call to flush (committed by Jay Kreps only a year ago). Also, there are good code samples of JDK 7-8 features such as FP Lambdas (8), switch on strings and try blocks with resources that avoid ugly catch/finally-s.

    There are also fine and simple usage of concurrency methods, some from Google Guava, some standard Java.

    Usage of Apache Commons interceptors for gzip compression may warrant further discussion. The solution enables transparent usage of compressed data within the infrastructure, but as a Kafka senior writer Neha Narkhede pointed out somewhere, tradeoffs are between disk/network latencies and cpu latencies warrant close examination for the implementation at hand (she mentions the need to evaluate offsets correctly as a challenge).

    In the real world, coordination of starting up components and shutting them down is critical. This has been done with skill and care even though it’d appear not to be overly important, but it sure does help a great deal to get a complex configuration running on first attempt easily. The script automation of tabs for OSX terminals is really sweet for instance.

    The C/C++/Linux developers will appreciate the use of daemons and clean asynchronous shut down of services using a while loop an an atomic boolean variable.

    What is nicest is the that the code follows the README file rather well and straightforwardly with a very clean architecture that is easy to understand (it takes me several hours but I had a lot of material to learn first time and the real world is not trivial, so that’s fair price to pay). Specifically, the demonstration of Netty technology allowing highly scalable solution without the bloat of servlets is nice to see in a small enough example, so it should encourage many developers to question the need of big J2EE frameworks with no big scare. There’s no route, no MVC, no Dependency Injection, no religion if you get my drift.

    There’s a nice maven plugin used here so that the jar files for services appear to be normal executable programs (really-executable-jars), that’s sweet too.

    I didn’t mind the low level use of DOM functionality in JS but I had expected some higher level library there such as JQuery. Anyhow, that’s not the focus of the example.

    Keep the good work.

    1. John Boardman Post
      John Boardman

      Thank you so much for your well thought out, thorough comment and review. You nailed every aspect of the goals I had set for myself when creating this example. The fact that it has translated so well makes me exceedingly happy, because if even one person finds what I have provided to be useful, then all the hard work put into it has totally paid off.

      I appreciate your kind words. Thanks again,

Leave a Reply