RabbitMQ2

Messaging with RabbitMQ

by on May 13, 2013 10:00 am

We previously published a brief tutorial showing how to set up a Spring RabbitMQ Template. Now, let’s take a step back for a broader view of what RabbitMQ is and how you might use it.

This article introduces the relevant concepts, hopefully helps you to navigate around one or two common “gotchas,” and provides links to more in-depth resources. It doesn’t get into the “nuts and bolts” of each usage pattern or of coding with any particular client library; see the tutorials linked at the end of the article for information on those topics.

What Is RabbitMQ

RabbitMQ is messaging software built on the AMQP protocol. Applications and services can exchange messages via queues managed by a RabbitMQ broker.

There are many potential use cases for RabbitMQ. It can be used to dispatch tasks to be handled by a specialized service synchronously (as an RPC mechanism) or asynchronously (work queues). It can be used to broadcast information about events, such as task status updates that might be monitored by logging services and dashboard applications (publish/subscribe).

Message Queues and Exchanges

To users of other message queuing systems, a few things about RabbitMQ may be surprising. Queues can be created dynamically by message producers and/or message consumers; in fact, patterns like publish/subscribe rely on each consumer to create its own queue from which to read messages, which at first may seem counterintuitive.

Message producers don’t interact directly with queues; instead they send messages to exchanges. Each exchange may be bound to any number of queues. There are several types of exchange, each with its own set of routing rules that determine which of its bound queues, if any, will receive a given message.

Rabbit-diagram

Any number of consumers may listen for messages on a queue (unless the queue was set up by a consumer for its own exclusive use). This is not a publish/subscribe mechanism, however, because any given message placed on the queue will be delivered to only one of the consumers listening on that queue. By default a round-robin approach is used to decide which consumer gets each message; you can configure the queue such that a given consumer is ineligible to receive more messages until it is done processing any messages it’s already received. This can be useful for work queues to ensure a “fair” distribution of work.

Messages

Per the AMQP protocol, a message consists of a set of properties (basically key-value pairs) and a payload. (The protocol actually refers to this as the bare message; additional packaging is added as the message moves through the RabbitMQ infrastructure.) The payload can contain basically anything. One flexible approach is to use JSON-encoded maps containing whatever data your applications/services need to share; the Spring packages provide good support for this approach, or for using JSON-encoded objects in general. This is not required, though; whatever your consumers and producers agree on is acceptable.

There are a handful of standard properties, and applications can define their own properties (sometimes called “headers”) as well. Headers can mean whatever the producer and consumer want them to mean. (This may seem redundant; any application-defined information can be embedded in the payload, so why use application-defined properties? Headers can make sense, though, for information that is logically metadata describing the message.) The conventional wisdom seems to be that even most of the standard properties are open to interpretation by the application, but a certain amount of caution is advisable as some of them have special meaning to the broker.

The best advice is to read up on the standard properties in the AMQP specification so that you’ll recognize when one of them might serve your intended purpose. Referring to a protocol specification may seem daunting, but this one is reasonably accessible, and other sites – even those that otherwise contain pretty good tutorials – tend to gloss over the standard message properties. Misuse of a standard property, if the broker happens to use that particular value, can cause strange and unexpected behaviors.

You’re rarely required to directly set the standard properties, as a good client library will manage them adequately. When in doubt, the safest course is to use application-defined properties or the message payload for the information your message consumers will need.

Administration

RabbitMQ provides a web-based administration interface. Through this interface you can manager users and their permissions; monitor connections, exchanges, queues, and so on; send test messages or examine messages that are waiting in queues; etc. Debugging the interactions of message producers and consumers becomes much easier if you familiarize yourself with this interface.

While the administration interface is great for monitoring the overall behavior of the broker over time, it doesn’t generally provide good visibility of instantaneous events like errors that might occur when accepting a message. You should also be familiar with RabbitMQ’s log output.

Users have three types of permission – read, write, and configure – each consisting of a regular expression. Each of these permissions governs certain corresponding operations; for example, the configure operations governs the ability to create and delete named resources (queues and exchanges). A user is permitted to perform an operation on a given resource if the resource name matches the pattern set for the corresponding permission.

In the most permissive use case, a user’s read, write, and configure permissions can all be set to “.*”. This means that the user can perform any operation (create, delete, read to, write from, etc.) on any resource. Fully dynamic queue creation is possible; a client simply declares that it wants to use a queue or exchange, and if no queue or exchange exists with that name then one is created.

At the other extreme, you could set the configure permission to “^$”, which won’t match any valid resource name, for all users except specially designated queue administrators. This would simulate a more traditional approach where all the queues you’re going to use are created in advance. This may suit some purposes, but will not work well with patterns like publish/subscribe the way they’re typically done in RabbitMQ.

A middle-ground approach might give each user a prefix to be used in naming resources, to avoid naming collisions amongst applications. For example appA may log in as app1User, and app1User’s “configure” permission can be set to “appA_.*”.

More information about RabbitMQ permissions can be found here.

An obvious concern with dynamic queue creation is the possibility that a bunch of stale queues might be left idling, especially if some unforeseen event causes a client to lose connectivity. A queue can be configured as “exclusive” – meaning that only the client that created it may read from it – in which case it will be deleted automatically when that client disconnects. Or, even a shared queue can be marked for auto-deletion when no one is listening to it any more.

Further Reading

The RabbitMQ “Getting Started” page provides tutorials explaining the typical usage patterns and how to implement them using the standard Python or Java client libraries.

More generally, the RabbitMQ web site has a lot of useful information. You may want to look over the resources linked from the Developer Tools section of the Community page, as well as the various documents on the Documentation page.

The Spring AMQP project provides an alternative way to work with RabbitMQ, which can be simpler to work with (especially in Spring-based project environments). Per the Spring philosophy, it can take care of (or at least simplify) many of the rote tasks involved in using RabbitMQ.

— Mark Adelsberger, asktheteam@keyholesoftware.com

  • Share:

Leave a Reply

Things Twitter is Talking About
  • Happy 4th of July from the Keyhole team! We hope that you have a happy and safe holiday with your family and friends.
    July 4, 2015 at 9:55 AM
  • Let's talk testing. Here are common challenges #Agile teams face when writing automated tests & how to overcome them: http://t.co/DrKbNZJcE0
    July 3, 2015 at 11:06 AM
  • #GrokOla users get free educational tutorials. But lucky you, we've released some to the public. #JavaScript primer - http://t.co/nIR9XiWY6O
    July 3, 2015 at 10:55 AM
  • Being able to isolate debugging techniques can help make you a better debugger. Here's Time-Oriented #Debugging http://t.co/UplJgP4VzC
    July 2, 2015 at 10:50 AM
  • RT @zachagardner: @zachagardner has declared it is @ChipotleTweets day at @KeyholeSoftware . You have been warned 🐓🐖🐄
    July 2, 2015 at 10:09 AM
  • Current state of random number generation & the differences in how #Java & #JavaScript approach it - http://t.co/5tBKNXnu8T #security
    July 1, 2015 at 2:45 PM
  • Woohoo - 600 followers! Thanks, everyone. We'd love to ask you - what type of tweets / dev content would you like to see more of from us?
    July 1, 2015 at 10:38 AM
  • We would like to welcome Dallas Monson to the team today! Dallas is a Senior Architect focused on UI/UX and #JavaScript. Welcome, Dallas!
    July 1, 2015 at 8:35 AM
  • Good introduction to TypeScript - http://t.co/0N22fVpAHt Plus, how to approach modularization in #TypeScript - http://t.co/wxRWGBj3Uh
    June 30, 2015 at 3:25 PM
  • .@mrbristopher just delivered a new S911 Night Drone to James Hayes, winner of our #kcdc15 giveaway! Congrats, James! http://t.co/RriJIxubH2
    June 30, 2015 at 11:35 AM
  • It feels like primitives could have been left out of the initial implementation of #Java. See why - http://t.co/A8ChCBHXJO
    June 29, 2015 at 4:05 PM
  • Developers in a bounce house! I repeat, developers in a bounce house! We had a blast at our 1st company picnic. Pics: http://t.co/XIqs7ECUst
    June 29, 2015 at 1:40 PM
  • New #SpringBatch tutorial from @jhackett01: Spring Batch – Replacing XML Job Configuration With JavaConfig http://t.co/PmdXnriKQu #java
    June 29, 2015 at 11:46 AM
  • We had such a fun time at the Keyhole company picnic! Pictures to come, including some of our developers in the bounce house. #loveourteam
    June 29, 2015 at 8:41 AM
  • In #JavaScript, how do we harness the power of callbacks without the confusing mess of nested functions? Promises - http://t.co/j1gAJ9hi3D
    June 29, 2015 at 8:40 AM
  • .@zachagardner We are so happy that your family attended! This will definitely need to be repeated every year!
    June 28, 2015 at 8:14 PM
  • Thank you to all on the Keyhole team who came to our first inaugural company picnic! Wonderful food, family and bounce house fun!
    June 28, 2015 at 7:50 PM
  • Debugging is a challenging part of being a programmer. We have a tutorial series to help, with a #JavaScript focus - http://t.co/rfhjJo64P7
    June 27, 2015 at 1:45 PM
  • We love #KCDC15! @PinsightMedia's James Hayes just was drawn as the winner of the @KeyholeSoftware drone giveaway. Stop on by the KHS booth.
    June 26, 2015 at 2:24 PM
  • Congratulations to James Hayes! You have won the Keyhole drone giveaway, come up and get your prize! #KCDC15
    June 26, 2015 at 2:09 PM