RabbitMQ2

Messaging with RabbitMQ

by on May 13, 2013 10:00 am

We previously published a brief tutorial showing how to set up a Spring RabbitMQ Template. Now, let’s take a step back for a broader view of what RabbitMQ is and how you might use it.

This article introduces the relevant concepts, hopefully helps you to navigate around one or two common “gotchas,” and provides links to more in-depth resources. It doesn’t get into the “nuts and bolts” of each usage pattern or of coding with any particular client library; see the tutorials linked at the end of the article for information on those topics.

What Is RabbitMQ

RabbitMQ is messaging software built on the AMQP protocol. Applications and services can exchange messages via queues managed by a RabbitMQ broker.

There are many potential use cases for RabbitMQ. It can be used to dispatch tasks to be handled by a specialized service synchronously (as an RPC mechanism) or asynchronously (work queues). It can be used to broadcast information about events, such as task status updates that might be monitored by logging services and dashboard applications (publish/subscribe).

Message Queues and Exchanges

To users of other message queuing systems, a few things about RabbitMQ may be surprising. Queues can be created dynamically by message producers and/or message consumers; in fact, patterns like publish/subscribe rely on each consumer to create its own queue from which to read messages, which at first may seem counterintuitive.

Message producers don’t interact directly with queues; instead they send messages to exchanges. Each exchange may be bound to any number of queues. There are several types of exchange, each with its own set of routing rules that determine which of its bound queues, if any, will receive a given message.

Rabbit-diagram

Any number of consumers may listen for messages on a queue (unless the queue was set up by a consumer for its own exclusive use). This is not a publish/subscribe mechanism, however, because any given message placed on the queue will be delivered to only one of the consumers listening on that queue. By default a round-robin approach is used to decide which consumer gets each message; you can configure the queue such that a given consumer is ineligible to receive more messages until it is done processing any messages it’s already received. This can be useful for work queues to ensure a “fair” distribution of work.

Messages

Per the AMQP protocol, a message consists of a set of properties (basically key-value pairs) and a payload. (The protocol actually refers to this as the bare message; additional packaging is added as the message moves through the RabbitMQ infrastructure.) The payload can contain basically anything. One flexible approach is to use JSON-encoded maps containing whatever data your applications/services need to share; the Spring packages provide good support for this approach, or for using JSON-encoded objects in general. This is not required, though; whatever your consumers and producers agree on is acceptable.

There are a handful of standard properties, and applications can define their own properties (sometimes called “headers”) as well. Headers can mean whatever the producer and consumer want them to mean. (This may seem redundant; any application-defined information can be embedded in the payload, so why use application-defined properties? Headers can make sense, though, for information that is logically metadata describing the message.) The conventional wisdom seems to be that even most of the standard properties are open to interpretation by the application, but a certain amount of caution is advisable as some of them have special meaning to the broker.

The best advice is to read up on the standard properties in the AMQP specification so that you’ll recognize when one of them might serve your intended purpose. Referring to a protocol specification may seem daunting, but this one is reasonably accessible, and other sites – even those that otherwise contain pretty good tutorials – tend to gloss over the standard message properties. Misuse of a standard property, if the broker happens to use that particular value, can cause strange and unexpected behaviors.

You’re rarely required to directly set the standard properties, as a good client library will manage them adequately. When in doubt, the safest course is to use application-defined properties or the message payload for the information your message consumers will need.

Administration

RabbitMQ provides a web-based administration interface. Through this interface you can manager users and their permissions; monitor connections, exchanges, queues, and so on; send test messages or examine messages that are waiting in queues; etc. Debugging the interactions of message producers and consumers becomes much easier if you familiarize yourself with this interface.

While the administration interface is great for monitoring the overall behavior of the broker over time, it doesn’t generally provide good visibility of instantaneous events like errors that might occur when accepting a message. You should also be familiar with RabbitMQ’s log output.

Users have three types of permission – read, write, and configure – each consisting of a regular expression. Each of these permissions governs certain corresponding operations; for example, the configure operations governs the ability to create and delete named resources (queues and exchanges). A user is permitted to perform an operation on a given resource if the resource name matches the pattern set for the corresponding permission.

In the most permissive use case, a user’s read, write, and configure permissions can all be set to “.*”. This means that the user can perform any operation (create, delete, read to, write from, etc.) on any resource. Fully dynamic queue creation is possible; a client simply declares that it wants to use a queue or exchange, and if no queue or exchange exists with that name then one is created.

At the other extreme, you could set the configure permission to “^$”, which won’t match any valid resource name, for all users except specially designated queue administrators. This would simulate a more traditional approach where all the queues you’re going to use are created in advance. This may suit some purposes, but will not work well with patterns like publish/subscribe the way they’re typically done in RabbitMQ.

A middle-ground approach might give each user a prefix to be used in naming resources, to avoid naming collisions amongst applications. For example appA may log in as app1User, and app1User’s “configure” permission can be set to “appA_.*”.

More information about RabbitMQ permissions can be found here.

An obvious concern with dynamic queue creation is the possibility that a bunch of stale queues might be left idling, especially if some unforeseen event causes a client to lose connectivity. A queue can be configured as “exclusive” – meaning that only the client that created it may read from it – in which case it will be deleted automatically when that client disconnects. Or, even a shared queue can be marked for auto-deletion when no one is listening to it any more.

Further Reading

The RabbitMQ “Getting Started” page provides tutorials explaining the typical usage patterns and how to implement them using the standard Python or Java client libraries.

More generally, the RabbitMQ web site has a lot of useful information. You may want to look over the resources linked from the Developer Tools section of the Community page, as well as the various documents on the Documentation page.

The Spring AMQP project provides an alternative way to work with RabbitMQ, which can be simpler to work with (especially in Spring-based project environments). Per the Spring philosophy, it can take care of (or at least simplify) many of the rote tasks involved in using RabbitMQ.

— Mark Adelsberger, asktheteam@keyholesoftware.com

  • Share:

Leave a Reply

Things Twitter is Talking About
  • .@NebraskaCC is just 2 weeks away & tickets are still available! #KC developers: Lincoln is just a quick 3-hour drive away. Well worth it!
    March 3, 2015 at 10:59 AM
  • Happy Punday from the Keyhole team! :-) We hope you're having a great day. http://t.co/p0iDzAY9qL
    March 2, 2015 at 2:54 PM
  • There's a new post on the Keyhole dev blog from @joshuarob01 - Agile Team Member Anti-Patterns http://t.co/2nfYQNIkP0 #agile
    March 2, 2015 at 10:57 AM
  • Heard of #GrokOla yet? Q&A with our expert dev team + code-sensitive wiki for tribal knowledge http://t.co/oCITNW9xEf http://t.co/61dIP4Q4jE
    February 27, 2015 at 1:38 PM
  • Rapid appdev has a bad rep, but there are ways to bring development time down the right way. Don't Fear the Rapid - http://t.co/8CKhAzmysb
    February 27, 2015 at 1:05 PM
  • RT @DZoneLinks: Swift 1.2 Arrives: 13 New Features - http://t.co/07q5vavZsT - @DZoneLinks Big Link by mswatcher
    February 26, 2015 at 2:17 PM
  • Our GrokStars are at it again - they've released a free #GrokOla primer. Get to know Java Lambdas: http://t.co/D2iLLj8mph
    February 26, 2015 at 9:13 AM
  • RT @dbgrandi: OH: “Do programmers have any specific superstitions?” “Yeah, but we call them best practices.”
    February 25, 2015 at 6:05 PM
  • New Primer: Introduction to the #Backbonejs MVC Framework - http://t.co/VLRJ4b5fwj Free #GrokOla tutorial available to the public.
    February 25, 2015 at 4:26 PM
  • #RabbitMQ: messaging software built on AMQP protocol. Learn relevant concepts & how to avoid common "gotchas" here: http://t.co/ZwMXlhspJ0
    February 25, 2015 at 3:20 PM
  • #Java is OO but contains non-object primitives. Autoboxing feels more like a band-aid. Do primitives need to go? http://t.co/A8ChCBHXJO
    February 25, 2015 at 1:44 PM
  • We're excited for Tech Night tonight! @bricemciver will present to the team on Leaflet.js in preparation for his @nebraskacc talk. Lucky us!
    February 24, 2015 at 4:15 PM
  • Do Primitives need to go in enterprise apps? - http://t.co/A8ChCBqmle New #Java post on the Keyhole blog.
    February 24, 2015 at 3:31 PM
  • When you pair #JAXB & #JPA, you can expect some "gotchas." Here are techniques to help you overcome the hurdles - http://t.co/J1s5DpcsCp
    February 24, 2015 at 8:15 AM
  • Do Primitives Need To Go? - http://t.co/A8ChCBHXJO New #Java post on the Keyhole blog by @jhoestje
    February 23, 2015 at 10:44 AM
  • New to #JavaScript prototypal inheritance? Here are some notes to help you along the way - http://t.co/NTIDZS6Uhy
    February 20, 2015 at 12:10 PM
  • XML Manipulation With XML Copy Editor: http://t.co/iHHmyAUQqU Good, free tool for when you need to manipulate an #XML document directly.
    February 20, 2015 at 10:35 AM
  • We've been releasing some free #Grokola tutorials. See them all in one spot - http://t.co/WDt5fWa728 #JavaScript #Backbonejs #nodejs #java
    February 19, 2015 at 2:42 PM
  • Code For Maintainability So The Next Developer Doesn't Hate You - http://t.co/iG2wW2rSWj Eight helpful tips to do so.
    February 19, 2015 at 11:20 AM
  • Functional Reactive Programing & #JavaScript offer an elegant way to reduce complexity of time-varying events. Intro: http://t.co/YGSsz5e0xN
    February 18, 2015 at 3:15 PM
Keyhole Software
8900 State Line Road, Suite 455
Leawood, KS 66206
ph: 877-521-7769
© 2015 Keyhole Software, LLC. All rights reserved.