RabbitMQ2

Messaging with RabbitMQ

by on May 13, 2013 10:00 am

We previously published a brief tutorial showing how to set up a Spring RabbitMQ Template. Now, let’s take a step back for a broader view of what RabbitMQ is and how you might use it.

This article introduces the relevant concepts, hopefully helps you to navigate around one or two common “gotchas,” and provides links to more in-depth resources. It doesn’t get into the “nuts and bolts” of each usage pattern or of coding with any particular client library; see the tutorials linked at the end of the article for information on those topics.

What Is RabbitMQ

RabbitMQ is messaging software built on the AMQP protocol. Applications and services can exchange messages via queues managed by a RabbitMQ broker.

There are many potential use cases for RabbitMQ. It can be used to dispatch tasks to be handled by a specialized service synchronously (as an RPC mechanism) or asynchronously (work queues). It can be used to broadcast information about events, such as task status updates that might be monitored by logging services and dashboard applications (publish/subscribe).

Message Queues and Exchanges

To users of other message queuing systems, a few things about RabbitMQ may be surprising. Queues can be created dynamically by message producers and/or message consumers; in fact, patterns like publish/subscribe rely on each consumer to create its own queue from which to read messages, which at first may seem counterintuitive.

Message producers don’t interact directly with queues; instead they send messages to exchanges. Each exchange may be bound to any number of queues. There are several types of exchange, each with its own set of routing rules that determine which of its bound queues, if any, will receive a given message.

Rabbit-diagram

Any number of consumers may listen for messages on a queue (unless the queue was set up by a consumer for its own exclusive use). This is not a publish/subscribe mechanism, however, because any given message placed on the queue will be delivered to only one of the consumers listening on that queue. By default a round-robin approach is used to decide which consumer gets each message; you can configure the queue such that a given consumer is ineligible to receive more messages until it is done processing any messages it’s already received. This can be useful for work queues to ensure a “fair” distribution of work.

Messages

Per the AMQP protocol, a message consists of a set of properties (basically key-value pairs) and a payload. (The protocol actually refers to this as the bare message; additional packaging is added as the message moves through the RabbitMQ infrastructure.) The payload can contain basically anything. One flexible approach is to use JSON-encoded maps containing whatever data your applications/services need to share; the Spring packages provide good support for this approach, or for using JSON-encoded objects in general. This is not required, though; whatever your consumers and producers agree on is acceptable.

There are a handful of standard properties, and applications can define their own properties (sometimes called “headers”) as well. Headers can mean whatever the producer and consumer want them to mean. (This may seem redundant; any application-defined information can be embedded in the payload, so why use application-defined properties? Headers can make sense, though, for information that is logically metadata describing the message.) The conventional wisdom seems to be that even most of the standard properties are open to interpretation by the application, but a certain amount of caution is advisable as some of them have special meaning to the broker.

The best advice is to read up on the standard properties in the AMQP specification so that you’ll recognize when one of them might serve your intended purpose. Referring to a protocol specification may seem daunting, but this one is reasonably accessible, and other sites – even those that otherwise contain pretty good tutorials – tend to gloss over the standard message properties. Misuse of a standard property, if the broker happens to use that particular value, can cause strange and unexpected behaviors.

You’re rarely required to directly set the standard properties, as a good client library will manage them adequately. When in doubt, the safest course is to use application-defined properties or the message payload for the information your message consumers will need.

Administration

RabbitMQ provides a web-based administration interface. Through this interface you can manager users and their permissions; monitor connections, exchanges, queues, and so on; send test messages or examine messages that are waiting in queues; etc. Debugging the interactions of message producers and consumers becomes much easier if you familiarize yourself with this interface.

While the administration interface is great for monitoring the overall behavior of the broker over time, it doesn’t generally provide good visibility of instantaneous events like errors that might occur when accepting a message. You should also be familiar with RabbitMQ’s log output.

Users have three types of permission – read, write, and configure – each consisting of a regular expression. Each of these permissions governs certain corresponding operations; for example, the configure operations governs the ability to create and delete named resources (queues and exchanges). A user is permitted to perform an operation on a given resource if the resource name matches the pattern set for the corresponding permission.

In the most permissive use case, a user’s read, write, and configure permissions can all be set to “.*”. This means that the user can perform any operation (create, delete, read to, write from, etc.) on any resource. Fully dynamic queue creation is possible; a client simply declares that it wants to use a queue or exchange, and if no queue or exchange exists with that name then one is created.

At the other extreme, you could set the configure permission to “^$”, which won’t match any valid resource name, for all users except specially designated queue administrators. This would simulate a more traditional approach where all the queues you’re going to use are created in advance. This may suit some purposes, but will not work well with patterns like publish/subscribe the way they’re typically done in RabbitMQ.

A middle-ground approach might give each user a prefix to be used in naming resources, to avoid naming collisions amongst applications. For example appA may log in as app1User, and app1User’s “configure” permission can be set to “appA_.*”.

More information about RabbitMQ permissions can be found here.

An obvious concern with dynamic queue creation is the possibility that a bunch of stale queues might be left idling, especially if some unforeseen event causes a client to lose connectivity. A queue can be configured as “exclusive” – meaning that only the client that created it may read from it – in which case it will be deleted automatically when that client disconnects. Or, even a shared queue can be marked for auto-deletion when no one is listening to it any more.

Further Reading

The RabbitMQ “Getting Started” page provides tutorials explaining the typical usage patterns and how to implement them using the standard Python or Java client libraries.

More generally, the RabbitMQ web site has a lot of useful information. You may want to look over the resources linked from the Developer Tools section of the Community page, as well as the various documents on the Documentation page.

The Spring AMQP project provides an alternative way to work with RabbitMQ, which can be simpler to work with (especially in Spring-based project environments). Per the Spring philosophy, it can take care of (or at least simplify) many of the rote tasks involved in using RabbitMQ.

– Mark Adelsberger, asktheteam@keyholesoftware.com

  • Share:

Leave a Reply

Things Twitter is Talking About
  • We just posted pictures from our National Hot Dog Day Lunch Cookout. Check them out - http://t.co/To06plaw1C
    July 23, 2014 at 4:14 PM
  • Good free cheat sheet - #Java Performance Optimization Refcard from @DZone: http://t.co/7vBgsmqy08
    July 23, 2014 at 10:48 AM
  • Did you know today is a holiday? It's National Hot Dog Day! We're gearing up for our team lunch hot dog cookout & can't wait to celebrate.
    July 23, 2014 at 9:43 AM
  • Check out our newest blog: #JAXB – A Newcomer’s Perspective, Part 1 http://t.co/Ki9G04HV5e
    July 22, 2014 at 1:22 PM
  • New post on the Keyhole blog by Mark Adelsberger: #JAXB – A Newcomer’s Perspective, Part 1 http://t.co/Ki9G04HV5e
    July 21, 2014 at 2:27 PM
  • If you're a Java dev, you're likely familiar with Annotations. But have you created your own #Java Annotations? Ex - http://t.co/BgCsYjxZKF
    July 18, 2014 at 12:10 PM
  • RT @gamasutra: Don't Miss: Unconventional Tips for Improving your Programming Skills http://t.co/6TFox7CKHU
    July 16, 2014 at 3:20 PM
  • We're about to send out our free monthly tech newsletter. Dev tips/articles via email. Not on the list yet? Sign up - http://t.co/F8h0NSiicZ
    July 15, 2014 at 11:57 AM
  • Have you ever tried creating your own #Java annotations? See a situation where it was beneficial - http://t.co/BgCsYjxZKF
    July 15, 2014 at 8:36 AM
  • There's a new post on the Keyhole blog by @jhackett01: Creating Your Own #Java Annotations - http://t.co/BgCsYjxZKF
    July 14, 2014 at 1:43 PM
  • We love development! Have you seen our weekly team blog? We show how to be successful with the tech we use. See it - http://t.co/nlRtb1XNQH
    July 12, 2014 at 2:35 PM
  • Rapid appdev has a bad rep, but there are ways to bring development time down the right way. Don't Fear the Rapid - http://t.co/aTPcAKOj0r
    July 11, 2014 at 3:10 PM
  • Automated Testing is great for dev, but does bring a set of challenges (especially for #agile teams). Success tips: http://t.co/1acl1ngO7i
    July 11, 2014 at 12:16 PM
  • This is fantastic - One small step for Google, one giant leap for empowering girls to code: http://t.co/R90V5DBkv1
    July 10, 2014 at 2:52 PM
  • #RabbitMQ: messaging software built on AMQP protocol. Learn relevant concepts & how to avoid common "gotchas" here: http://t.co/ZwMXlhKyX8
    July 10, 2014 at 9:31 AM
  • A huge welcome to Brian Wetzel who joins the Keyhole team today!
    July 10, 2014 at 8:50 AM
  • Seen our newest blog? @joshuarob01 talks the challenges #Agile teams face in automated testing & how to solve them: http://t.co/1acl1ngO7i
    July 9, 2014 at 3:51 PM
  • How can we ensure that we write unit tests with sufficient coverage? Here are some tips for success in #unittesting - http://t.co/GGTEGEQFtk
    July 9, 2014 at 3:05 PM
  • Are you a .NET dev who likes working with #JavaScript, #Angular & a solid team? Join Keyhole: http://t.co/kWgVdxgvpG http://t.co/VLoeULthOE
    July 9, 2014 at 8:48 AM
  • Counting down until the Keyhole Team Happy Hour this evening @ the District. Pour House + Kitchen... one more hour!
    July 8, 2014 at 4:15 PM
Keyhole Software
8900 State Line Road, Suite 455
Leawood, KS 66206
ph: 877-521-7769
© 2014 Keyhole Software, LLC. All rights reserved.