Let's start by clarifying what we mean by storing messages in memory, since it is not a precise statement. If interpreted as "RabbitMQ uses memory to process messages", this phrase is certainly true. When client applications send data to RabbitMQ, that data appears in memory buffers first (that's true for all network-based software). It is also true, that RabbitMQ might cache some messages in memory, for example to improve performance (again, true for virtually all software that needs to serve data).

However, I keep hearing this phrase used to express concerns about message durability - if you power off the server, your messages will be lost! In this context, modern RabbitMQ versions almost never store messages in memory.

We'll go through different queue types and discuss how the messages are processed, stored and when they are confirmed to the publishers. Publisher confirmations are critical here - RabbitMQ doesn't offer guarantees for messages that were not confirmed. If you haven't received the confirmation, you can't even know whether the message reached RabbitMQ (the network connection could have failed for example). With that in mind, let's go through the different queue types.

Classic Queues​

Classic queues is the oldest kind of queues in RabbitMQ and the main source of "RabbitMQ stores messages in memory" misconception. RabbitMQ was first released in 2007. Disks were significantly slower back then and therefore classic queues were designed to try to avoid writing messages to disk. There was a whole bunch of settings to configure when RabbitMQ should write messages to disk (a process called "paging") so it would not keep messages in memory indefinitely but indeed, you could say RabbitMQ stored messages in memory back then. However, that was a long time ago.

In RabbitMQ 3.6, released in 2015, the "lazy mode" was introduced. A queue configured to be lazy, always stored messages on disk and didn't keep them in memory at all. This means that "RabbitMQ stores messages in memory" was not true 10 years ago. It'd still do this by default, but it was completely optional.

Lazy mode was removed in RabbitMQ in 3.12, released in 2023, but the default (and the only available) behaviour changed and is similar to the lazy mode, although not exactly the same. Therefore, for over a year now, classic queues don't store messages in memory and can't even be configured to do so. The myth is almost entirely false at this point.

Almost entirely? So here's how classic queues work right now: they accumulate incoming messages in a small in-memory buffer and write them to disk as a batch, as soon as the in-memory buffer is full. Since we don't know if more messages will come, there are additional triggers that flush that buffer, including a flush after a certain number of batched messages (even if they are too small to use the whole buffer) and after a certain number of operations on that batch is performed. Additionally, the queue monitors how quickly messages are consumed and makes decisions based on that (if there are fast consumers, more messages will be cached in memory). Finally, there's the last-resort trigger which will flush the buffer every 200ms. So the absolute maximum of how long a message can be stored in memory is 200ms but in practice, I've never seen this happen. Publishers usually receive confirms within a few milliseconds and they are sent only after the message was written to disk.

But I See No Disk Activity When I Use a Classic Queue!​

Indeed, it's entirely possible to publish messages to classic queues and see virtually no disk writes nor disk reads. How is that possible? It's an optimisation for a very specific, but relatively common case. As explained above, messages can be briefly kept in memory but if there are active consumers that are waiting for messages (their prefetch buffer is not full), messages that reach the queue are dispatched to the consumers immediately, without waiting for their batch to be written to disk. A message that gets acknowledged by the consumer before this message's batch is written to disk, will not be written to disk at all, because it simply doesn't need to be. Queues don't store acknowledged messages, so if the message is acknowledged before it is written, it doesn't get written. If it's acknowledged after it was written, it's deleted from the queue (the actual removal from disk will happen later, asynchronously, but it is considered deleted immediately).

It's worth mentioning that classic queues have two separate storage mechanisms. Messages below 4kb (configurable through queue_index_embed_msgs_below) are stored in a per-queue message store and messages above that threshold are stored in a per-vhost message store. The optimisation mentioned above only works for messages that would be stored in the per-queue message store.

So here it is, in modern RabbitMQ versions, classic queues store messages in memory for a very short period of time (milliseconds) and no more than 200ms for sure. They may not write messages to disk at all, if the messages are small, and consumed quickly enough, but this is just a performance optimisation. I'll leave it up to you to decide if that qualifies as "RabbitMQ stores messages in memory", but I think a more accurate statement would be "when a message is delivered to a classic queue, RabbitMQ writes messages to disk with a short delay". But yes, that means that for a brief moment, they are only in memory.

Surely, Transient Messages Are Stored in Memory?​

No. Again, things were different in the past but as of RabbitMQ 4.0, the only difference between persistent and transient messages is when RabbitMQ sends back the publisher confirm. The messages are stored the same way as described above.

For persistent messages, the confirm is sent when either of these two events takes place:

1. The message is written to disk
2. The message is delivered and acknowledged by a consumer (if that happens before it's written to disk)

For transient messages, the confirm is sent as soon as the message reaches the queue and enters the in-memory buffer. Since the message is transient, the guarantees are lax: the queue received the message, the publisher can move on.

What About fsync?​

fsync is a low-level file system operation that should ensure that messages are really written to disk. There are multiple layers of I/O buffers between a user-space process such as RabbitMQ and actual hardware, including operating system buffers and internal disk buffers. Performing a write without performing fsync doesn't guarantee that the data will survive a sudden power loss. Unfortunately, fsync is a relatively slow operation, so any I/O intensive software has to decide if, and when, to call it. While classic queues call fsync in some cases (for example, when RabbitMQ stops gracefully), fsync is not performed before publisher confirms are sent. Therefore, even durable messages that a publisher received a confirmation for, can technically be lost if the server crashes. If you need stronger guarantees, you can use quorum queues.

Quorum Queues​

From the initial release in RabbitMQ 3.8 (released in 2019), quorum queues always stored messages on disk. While the initial versions had an additional in-memory cache for messages, it was removed in RabbitMQ 3.10.

The situation is therefore simple: if the publisher received a confirmation, this means the message had already been written to disk and fsync-ed on the quorum of nodes (in the most common scenario of a 3-node cluster, that means it was written and fsync-ed on at least 2 nodes).

Since RabbitMQ doesn't offer any guarantees for messages that have not been confirmed to publishers, we could pretty much stop here. However, for the sake of completeness, I'll mention that some messages are technically in memory:

1. The queue process has a mailbox (an Erlang/OTP concept) where requests to the queue process (such as enqueue/dequeue operations) arrive for processing. The quorum queue process receives messages from the mailbox and processes them as a batch. When there's a lot of requests, these operations may accumulate in the mailbox and therefore, assuming there are enqueue operations there, at this point, some messages are only in-memory. However, this generally means RabbitMQ is at least briefly overloaded and either way, these operations are usually processed within a few milliseconds. Plus, these messages are not confirmed yet.
2. Quorum queues rely on the Raft protocol and our Raft implementations stores the most recent Raft operations in memory. For enqueue operations this means that the message is in memory as well. However, at this point the message is already written to disk and fsync-ed or it hasn't been confirmed yet.

Streams​

With Streams, the situation is even simpler than with quorum queues: Streams never supported keeping messages in memory, period. The main difference between queues and stream in general, is that streams can be read multiple times and therefore consuming a message doesn't remove that message from a stream. There's no point in storing messages only in memory, if we need to be able to deliver them to consumers multiple times, potentially long after they were published.

Streams do not perform fsync, since they were optimised for high message throughput.

For completeness, just like quorum queues (and any other Erlang process), the stream process has a mailbox where requests to the stream process arrive. There's therefore a moment where the messages are stored in memory for a short time. Once again though, these are messages that have not been confirmed yet and they rarely stay in memory for more than a few milliseconds.

MQTT QoS 0 Queues​

RabbitMQ 3.12 introduced Native MQTT support, and as part of that work, a new queue type was introduced, specifically for MQTT QoS 0 consumers (you can't explicitly declare a queue of this type, you have to create an MQTT QoS 0 subscription). Since QoS 0 basically means best-effort but no guarantees, QoS 0 messages are not written to disk at all and are delivered directly to the consumers that are present. Effectively, there's no queue at all (beyond the Erlang mailbox). Messages received from the publisher are immediately delivered to the consumers and removed from memory.

Does this qualify as storing messages in memory? I'd say it doesn't - the messages are in memory initially, simply because that's how computers work, and are removed from memory as soon as they are delivered to the consumers. We don't really store them in memory - we just process them and never write them to disk in this case. You can disagree and say that this is exactly what "storing messages in memory means" but even then - this only applies to MQTT QoS 0 usage and the messages will generally stay in memory for no more than a fraction of second.

Message Metadata​

So far I focused on message bodies, since that's what people usually mean when talking about storing messages in memory. However, RabbitMQ also needs to keep track of the messages that are currently present in the queues. For example, when a queues has a defined x-max-length limit, RabbitMQ needs to keep track of the total size of all the messages in the queue, so when it delivers a message, it keeps the message size (but not the message body itself) in memory, to just subtract it quickly from the total size of the queue, once the consumer acknowledges the messages.

This kind of metadata is stored differently by different queue type, but even when stored in memory, it will consume significantly less memory than the message bodies would and doesn't change any guarantees about the message durability.

Here's how we store the metadata for different queue types:

• Classic queues
  • for messages stored in the per-queue message store, no data is stored in memory
  • for messages stored in the per-vhost message store, there's some metadata in memory
• Quorum queues
  • metadata is stored in memory (at least 32 bytes per message, sometimes a bit more, for example when message TTL is used)
• Streams
  • no message metadata is stored in memory

This basically means that for messages under 4KB stored in classic queues, as well as for streams, regardless of how many messages there are in the queue/stream, the memory usage is constant. You will run out disk before you run out of memory (you should configure retention/length limits to avoid running out disk, but that's a different story).

Here's an illustration highlighting the difference between the two classic queue storage mechanisms. In this test, I first published 1 million messages of 4000 bytes each, then deleted the queue and published 1 million messages of 4100 bytes each. As you can see, the memory usage was stable in the first phase (small fluctuations notwithstanding), but when publishing larger messages, we can see the memory usage grows as well. This is because 4100 bytes is above the threshold, so these messages are stored in the per-vhost message store and the per-vhost message store keeps some metadata in memory. A million 4KB messages would have taken up 4GB of memory to store, while the actual usage is still below 400MB.

Summary​

Here's the summary of the key points.

The flexibility provided by RabbitMQ, with support for multiple protocols, queue types and other configurations (eg. single node, vs a cluster with queue replication), combined with 18 years of history and evolution, means that almost any "RabbitMQ does/doesn't do X" statement is incorrect or at least imprecise. They should almost always be quantified with a specific version and configuration.

Going back to the title of this post, I think it's fair to say that "RabbitMQ doesn't store messages in memory" is much closer to the truth, than the opposite claim, which still circulates in discussions involving RabbitMQ. Regardless of the queue type, there is no configuration in which publishing, say, 1GB of messages to RabbitMQ with no connected consumers, would lead to 1GB of memory being used to store these messages. Most importantly, if you want high data safety guarantees, quorum queues are available and store data safely by default. If you publish a message to a quorum queue and receive the confirmation, it'd take a disastrous event for RabbitMQ to lose it (and if you want to protect messages from disastrous events, you might be interested in the commercial Warm Standby Replication plugin).

If you don't need such data safety guarantees, you don't have to pay the intrinsic overhead of data safety. Just use the right tool for the job.

The Erlang Port Mapper Daemon (epmd) is a built-in component that helps Erlang-based applications (including RabbitMQ) discover each other’s distribution ports. Together with DNS for hostname resolution, epmd is a piece of infrastructure RabbitMQ nodes rely on for clustering, inter-node communication and CLI tools connectivity.

While epm is very limited in scope, its exposure to the public Internet often means that Erlang distribution ports are also exposed. This creates a potential security risk: if attackers find these distribution ports, they'd be one secret value away from being able to run CLI commands against the node (or cluster).

Recent scans have revealed over 85,000 instances of publicly accessible epmd, with roughly half associated with RabbitMQ servers.

Fortunately, all it usually takes to mitigate this risk is limiting network access to a range of ports. epmd and inter-node communication can also be limited to local network interfaces, in particular for single node clusters used for running tests.

Read the full article on the Erlang Ecosystem Foundation blog.

RabbitMQ 4.0.5 is a new patch release in the 4.0.x series.

This series is currently covered by community support.

Release Artifacts​

Release artifacts can be obtained on GitHub as well as RPM, Debian package repositories.

Upgrade Guidance​

If upgrading from a version prior to 4.0, please consult the 4.0 release notes.

This feature enables RabbitMQ to support multiple concurrent clients, each consuming only a specific subset of messages while preserving message order. Additionally, it minimizes network traffic between RabbitMQ and its clients by dispatching only the messages that match the clients' interests.

In this blog post, we’ll explore what AMQP filter expressions are and walk through a simple Java example of how to use them.

Specification​

As outlined in the Native AMQP 1.0 blog post, one of AMQP 1.0's strengths is its extensibility, supported by numerous extension specifications. RabbitMQ 4.1 takes advantage of the extension specification AMQP Filter Expressions Version 1.0 Working Draft 09.

This specification defines AMQP type definitions for message filter expressions. Filter expressions are predicates evaluated against a message, returning either true or false. If a predicate evaluates to true, the broker dispatches the message to the consumer.

RabbitMQ 4.1 implements a subset of this specification, including:

• § 4.2.4 properties filter: Applies to the immutable properties section of the message.
• § 4.2.5 application-properties filter: Applies to the immutable application-properties section of the message.

Example​

Imagine each message carries metadata specifying a particular color. Different consumers can subscribe to the same stream, filtering messages to receive only those matching the color they are interested in.

The first consumer receives all green messages. The second consumer receives all purple messages. The third consumer receives all blue messages.

docker run -it --rm --name rabbitmq \
    -p 5672:5672 -p 15672:15672 \
    rabbitmq:4.1-rc-management

mvn clean compile exec:java

publisher sent message 0 with color green
publisher sent message 1 with color blue
publisher sent message 2 with color purple
publisher sent message 3 with color purple
publisher sent message 4 with color green
publisher sent message 5 with color green
consumer (filter green) received message 0
consumer (filter green) received message 4
consumer (filter green) received message 5
consumer (filter purple) received message 2
consumer (filter purple) received message 3
consumer (filter blue) received message 1
consumer (filter &s:e) received message 1
consumer (filter &s:e) received message 2
consumer (filter &s:e) received message 3

AMQP filter expressions enable multiple clients to concurrently consume specific subsets of messages from the same stream while preserving message order. This feature also minimizes network traffic between RabbitMQ and its clients by dispatching only the messages that match each client’s interests.

Stream Filtering Comparison​

The AMQP filter expressions feature described in this blog post should not be confused with the Bloom filter-based stream filtering introduced in RabbitMQ 3.13.

Both features serve the same purpose: filtering messages from a stream. However, their implementations differ, resulting in distinct characteristics:

Both features can be used together when consuming via AMQP 1.0.

Summary​

RabbitMQ 4.1 addresses the challenge of enabling multiple consumers on a single queue/stream while ensuring certain messages (e.g., those with the same subject or ID) are always processed by the same consumer, preserving in-order processing.

Although this feature is not available for classic queues or quorum queues, AMQP filter expressions allow consumers to filter messages when consuming from a stream. Since streams are immutable logs, total message order is maintained.

The modified outcome is a feature exclusive to AMQP 1.0 and not available in AMQP 0.9.1 It is supported in quorum queues, but not in classic queues.

This feature enables consumers to add or update message annotations before requeueing or dead lettering a message.

Requeue​

Including additional metadata when requeuing a message can be valuable for improving traceability and debugging during message processing.

For example, an application using the RabbitMQ AMQP 1.0 Java Client can set specific message annotations before requeuing the message at the head of a quorum queue, as shown below:

Consumer consumer = connection.consumerBuilder()
    .queue(ordersQueue)
    .messageHandler((context, message) -> {
        Map<String, Object> annotations = new HashMap<>();
        annotations.put("x-opt-requeue-reason", "external_service_unavailable");
        annotations.put("x-opt-requeue-time", System.currentTimeMillis());
        annotations.put("x-opt-requeued-by", "consumer_1");
        context.requeue(annotations);
    }).build();

These annotations could use different types including map, list, or array. This flexibility allows not only setting details like the last requeue reason, time, and consumer, but also tracking a history of requeue events. Maintaining such a history can reveal patterns, such as identifying consumers that requeue messages more frequently or discovering common requeue reasons across the system. However, keep in mind that quorum queues retain modified message annotations in memory, which increases the memory overhead per requeued message.

Setting custom headers before requeueing a message at the head of the queue is not supported in AMQP 0.9.1.

Whether requeuing a message to a quorum queue via AMQP 1.0 or AMQP 0.9.1, the x-delivery-count annotation will always be incremented.

Dead Letter​

When dead lettering a message, the consumer can include a custom reason for the dead lettering in the message annotations:

Consumer consumer = connection.consumerBuilder()
    .queue(ordersQueue)
    .messageHandler((context, message) -> {
        Map<String, Object> annotations = new HashMap<>();
        annotations.put("x-opt-dead-letter-reason", "Incompatible Message Format");
        context.discard(annotations);
    }).build();

When dead lettering to a headers exchange, the consumer can even decide which target queue the message will be routed to:

In this example, two dead letter quorum queues are bound to the dead letter headers exchange:

1. transient-failures-dlq
2. business-logic-failures-dlq

Different dead letter queues can be processed by different apps or teams, with varying actions taken depending on the nature of the failure. For instance, all messages in the transient-failures-dlq could be re-published to the original orders queue, while messages in the business-logic-failures-dlq might require human intervention.

More dead letter queues could be added, such as:

• data-integrity-dlq for messages with unknown schema
• resource-limit-dlq for cases where rate limits were exceeded
• critical-errors-dlq for situations that require administrator attention.

It’s crucial that all messages dead lettered from the orders queue are routable. The alternate exchange in the above diagram provides "or else" routing semantics, ensuring messages end up in the uncategorised-dlq if no x-opt-dead-letter-category annotation is set. This might occur, for example, if the publisher sets a ttl header but no consumer grants link credit, causing the message to expire and be dead lettered.

The scenario depicted above is demonstrated in the modified-outcome sample application.

publisher: received ACCEPTED outcome
consumer: setting annotations {x-opt-dead-letter-reason=Customer Not Eligible for Discount, x-opt-dead-letter-category=business-logic} and dead lettering...

Dead Letter vs. Re-publish​

An AMQP 0.9.1 consumer cannot set custom headers before dead lettering a message. However, instead of using basic.nack or basic.reject with requeue=false to dead letter a message, an AMQP 0.9.1 client could follow this approach:

1. Re-publish the message directly to a specific "dead letter" queue with new custom headers.
2. Wait for RabbitMQ to confirm the re-published message.
3. Acknowledge the original message via basic.ack.

An AMQP 1.0 client can choose between dead lettering with custom message annotations or re-publishing the message. Both approaches have their advantages and trade-offs:

Wrapping Up​

AMQP 1.0's modified outcome feature allows consumers to modify message annotations before requeueing or dead lettering.

Rather than relying solely on RabbitMQ's built-in dead lettering tracking via x-opt-deaths, consumers can customise dead lettering event tracking and even choose which dead letter queue a message is sent to.

Flow control is the process of managing the rate of data transmission between two nodes to prevent a fast sender from overwhelming a slow receiver.

The AMQP 1.0 protocol defines flow control at two different levels:

1. Link Flow Control
2. Session Flow Control

Link Flow Control​

In AMQP 1.0, messages are sent over a link. A link connects either a sending client application to an exchange in RabbitMQ or a queue in RabbitMQ to a consuming client application.

Link-Credit​

The central idea behind AMQP 1.0 link flow control is simple: To receive messages, a consumer must grant credits to the sending queue.

One credit corresponds to one message. For example, when the consumer grants 10 credits, RabbitMQ is allowed to send 10 messages. This straightforward principle, where the receiver provides feedback to the sender, ensures that the sender never overwhelms the receiver.

Both the receiver and sender maintain their own "link state". Part of this state is the current link credit. Link credit decreases by 1 each time a message is transferred. Specifically, the sender reduces link credit by 1 when it sends a message, and the receiver reduces link credit by 1 when it receives a message. When the sender's link credit reaches 0, it must stop sending messages.

Messages are sent in transfer frames.

Credits are granted in flow frames:

<field name="link-credit" type="uint"/>

As you might have guessed, they are called "flow" frames because these frames carry flow control information. The type uint stands for unsigned integer, a value between 0 and a large number (2^32 - 1).

Even after the link is successfully set up ("attached" in AMQP 1.0 terms), RabbitMQ is not allowed to start sending messages to the consumer until the consumer sends its first flow frame, granting link credit to the sending queue.

In its simplest form, when a client (receiver) grants a single credit to the queue (sender), the queue will send a single message, as illustrated in Figure 2.43: Synchronous Get of the AMQP 1.0 specification:

Receiver                                      Sender
=================================================================
                                      ...
flow(link-credit=1)               ---------->
                                        +---- transfer(...)
*block until transfer arrives*         /
                                  <---+
                                      ...
-----------------------------------------------------------------

Synchronously getting a single message at a time will result in low throughput. Therefore, a client typically grants multiple credits to a queue, as shown in Figure 2.45: Asynchronous Notification of the AMQP 1.0 specification:

Receiver                                          Sender
=====================================================================
                                      ...
                                  <----------     transfer(...)
                                  <----------     transfer(...)
flow(link-credit=delta)           ---+   +---     transfer(...)
                                      \ /
                                       x
                                      / \
                                  <--+   +-->
                                  <----------     transfer(...)
                                  <----------     transfer(...)
flow(link-credit=delta)           ---+   +---     transfer(...)
                                      \ /
                                       x
                                      / \
                                  <--+   +-->
                                      ...
---------------------------------------------------------------------

If the receiver grants N credits and waits for all N messages to arrive before granting the next N credits, throughput will be higher compared to figure 2.43 where N=1. However, if you look closely at figure 2.45, you will observe that the receiver grants more credits before receiving all previous messages. This approach results in the highest throughput. For example, in figure 2.45, the receiver might have granted 6 credits initially, and then sends another flow frame with link-credit = 6 to RabbitMQ whenever it has received 3 messages.

The receiver knows its current processing capacity and therefore it is always the receiver (and never the sender) that determines the current link credit. The sender only "consumes" link credit granted by the receiver by sending more messages.

The receiver is allowed to dynamically increase or decrease the amount of link credit depending on its current processing capacity.

You might wonder: What are good values for link-credit and how often should the client top up link credit? As is often the case, the answer is that you will need to benchmark your specific workload with different values to find out.

Instead of implementing fancy algorithms, I would recommend starting simple: for example, the client could initially grant 200 link credits and send a flow with link-credit = 200 whenever the remaining link credit falls below 100.

In fact, this is what RabbitMQ does the other way around: The RabbitMQ AMQP 1.0 session process grants initially 170 link credits to the publisher, and grants again 170 link credits when the remaining link credit falls below half (i.e. 85) and the number of unconfirmed messages is less than 170. (Internally on the broker, publisher confirms are always enabled between AMQP 1.0 session process and target queues, even when no confirms are sent to the publishing client. This means that if the target queue does not confirm fast enough, RabbitMQ stops granting link credit to the sending application.) Please note that these RabbitMQ implementation details can change at any time.

The value of 170 is configurable via advanced.config setting rabbit.max_link_credit.

Benefit #3

When one target queue is overloaded, the publisher can continue publishing at high speed to all other target queues.

Applications can send to multiple queues on the same AMQP 1.0 connection or session (by attaching multiple links). Let's assume a simple scenario where a client opens two links:

• Link 1 sends to a classic queue.
• Link 2 sends to a 5-replica quorum queue.

Before confirming a message, the quorum queue must have replicated the message to a majority of replicas, with each replica fsyncing the message to its local disk.

In contrast, classic queues do not replicate messages. Furthermore, when a message is consumed and acknowledged quickly enough, classic queues can (thereafter) confirm the message back to the publisher without ever writing it to disk. Hence, in this scenario, throughput for the classic queue will be far higher than for the quorum queue.

The beauty of AMQP 1.0 link flow control is that RabbitMQ can slow down granting credits on Link 2 while continuing to grant credits on Link 1 at a high frequency. Therefore, even when the 5-replica quorum queue does not process messages as quickly as the (single replica) classic queue, the client can continue to send at full speed to the classic queue.

The following picture is copied from a previous AMQP 0.9.1 blog post:

The word "credit" in this picture refers to RabbitMQ's internal flow control for AMQP 0.9.1 connections and is unrelated to link credit in AMQP 1.0.

The reader in the above picture is the Erlang process that reads AMQP 0.9.1 frames from the socket. The picture shows that for AMQP 0.9.1 connections, RabbitMQ will block the reader, causing TCP backpressure to be applied to the client. Therefore, when a single target queue becomes overloaded, RabbitMQ throttles the AMQP 0.9.1 connection, affecting the publishing to all other target queues.

The following benchmark shows how AMQP 1.0 can provide multiple times higher throughput compared to AMQP 0.9.1 when a connection sends to more than one target queue.

Benchmark: Two senders

To put the theory we just discussed into practice, the two_senders program simulates a similar scenario.

This program opens a single AMQP 0.9.1 connection and channel, as well as a single AMQP 1.0 connection and session.

On both the AMQP 0.9.1 channel and the AMQP 1.0 session, there are two goroutines that publish as quickly as possible into a classic queue and a quorum queue. This results in four target queues in total:

    main.go                                                      RabbitMQ
+-------------+                                     +----------------------------------+
|             |        AMQP 0.9.1 connection        |                                  |
|             |#####################################|                                  |
|    +---+    |-------------------------------------|    +------------------------+    |
|    | P |                                               | classic-queue-amqp-091 |    |
|    +---+                                               +------------------------+    |
|                        AMQP 0.9.1 channel                                            |
|    +---+                                               +------------------------+    |
|    | P |                                               | quorum-queue-amqp-091  |    |
|    +---+    |-------------------------------------|    +------------------------+    |
|             |#####################################|                                  |
|             |                                     |                                  |
|             |                                     |                                  |
|             |                                     |                                  |
|             |#####################################|                                  |
|    +---+    |-------------------------------------|    +-----------------------+     |
|    | P |O============================================>O| classic-queue-amqp-10 |     |
|    +---+                                               +-----------------------+     |
|                        AMQP 1.0 session                                              |
|    +---+                                               +-----------------------+     |
|    | P |O======================================+=====>O| quorum-queue-amqp-10  |     |
|    +-+-+    |----------------------------------|--|    +-----------------------+     |
|      |      |##################################|##|                                  |
|      |      |        AMQP 1.0 connection       |  |                                  |
+------|------+                                  |  +----------------------------------+
       |                                         |
       |                                         |
   Publisher                               AMQP 1.0 link
   goroutine

Run the benchmark as follows:

1. Start RabbitMQ server v4.0.0-beta.6 using make run-broker on an Ubuntu box. (On macOS, fsync does not write physically to the platters).
2. Execute the Go program with go run two_senders/main.go. After 10 seconds, the Go program will complete.
3. List the number of messages in each queue:

./sbin/rabbitmqctl --silent list_queues name type messages --formatter=pretty_table
┌────────────────────────┬─────────┬──────────┐
│ name                   │ type    │ messages │
├────────────────────────┼─────────┼──────────┤
│ classic-queue-amqp-091 │ classic │ 159077   │
├────────────────────────┼─────────┼──────────┤
│ quorum-queue-amqp-091  │ quorum  │ 155782   │
├────────────────────────┼─────────┼──────────┤
│ classic-queue-amqp-10  │ classic │ 1089075  │
├────────────────────────┼─────────┼──────────┤
│ quorum-queue-amqp-10   │ quorum  │ 148580   │
└────────────────────────┴─────────┴──────────┘

As explained in the AMQP 1.0 Benchmarks blog post, quorum queues fsync (fdatasync to be precise), whereas classic queues do not. Therefore, even without replication, a quorum queue is significantly slower than a classic queue because I use a consumer-grade disk where each fsync takes at least 5 milliseconds. For production clusters, it is recommended to use high-end, enterprise-grade disks that fsync faster.

The results show that the single AMQP 0.9.1 connection sends roughly the same number of messages to both the target classic queue and the target quorum queue. This is because quorum-queue-amqp-091 causes the entire AMQP 0.9.1 connection to be blocked (and unblocked) around 80 times per second in my benchmark. As a result, the publishing rate to multiple target queues (classic-queue-amqp-091 and quorum-queue-amqp-091) on a single AMQP 0.9.1 connection is constrained by the slowest target queue (quorum-queue-amqp-091). In total, the AMQP 0.9.1 connection sends 159,077 + 155,782 = 314,859 messages.

In contrast, thanks to link flow control, RabbitMQ throttles only the link to the quorum-queue-amqp-10 target, allowing the AMQP 1.0 client to continue publishing at full speed to the classic-queue-amqp-10 target. In total, the AMQP 1.0 connection sends 1,089,075 + 148,580 = 1,237,655 messages.

Therefore, in our simple benchmark, the total send throughput of AMQP 1.0 is four times (!) higher than that of AMQP 0.9.1.

    main.go                                                      RabbitMQ
+-------------+                                     +----------------------------------+
|             |        AMQP 0.9.1 connection        |                                  |
|             |#####################################|                                  |
|    +---+    |-------------------------------------|    +------------------------+    |
|    | C |                                               | classic-queue-amqp-091 |    |
|    +---+                                               +------------------------+    |
|                        AMQP 0.9.1 channel                                            |
|    +---+                                               +------------------------+    |
|    | P |                                               | quorum-queue-amqp-091  |    |
|    +---+    |-------------------------------------|    +------------------------+    |
|             |#####################################|                                  |
|             |                                     |                                  |
|             |                                     |                                  |
|             |                                     |                                  |
|             |#####################################|                                  |
|    +---+    |-------------------------------------|    +-----------------------+     |
|    | C |O<============================================O| classic-queue-amqp-10 |     |
|    +---+                                               +-----------------------+     |
|                        AMQP 1.0 session                                              |
|    +---+                                               +-----------------------+     |
|    | P |O======================================+=====>O| quorum-queue-amqp-10  |     |
|    +-+-+    |----------------------------------|--|    +-----------------------+     |
|      |      |##################################|##|                                  |
|      |      |        AMQP 1.0 connection       |  |                                  |
+------|------+                                  |  +----------------------------------+
       |                                         |
       |                                         |
Publisher or Consumer                      AMQP 1.0 link
   goroutine

./sbin/rabbitmqctl --silent list_queues name type messages --formatter=pretty_table
┌────────────────────────┬─────────┬──────────┐
│ name                   │ type    │ messages │
├────────────────────────┼─────────┼──────────┤
│ classic-queue-amqp-091 │ classic │ 990932   │
├────────────────────────┼─────────┼──────────┤
│ quorum-queue-amqp-091  │ quorum  │ 172800   │
├────────────────────────┼─────────┼──────────┤
│ classic-queue-amqp-10  │ classic │ 336229   │
├────────────────────────┼─────────┼──────────┤
│ quorum-queue-amqp-10   │ quorum  │ 130000   │
└────────────────────────┴─────────┴──────────┘

Delivery-Count​

So far, we understand only one field of the flow frame: link-credit.

What happens in the following scenario?

Receiver                                  Sender
=======================================================================
                              ...
link state:                               link state:
 link-credit = 3                           link-credit = 3

flow(link-credit = 6)     ---+   +---     transfer(...)
                              \ /
                               x
                              / \
                          <--+   +-->

link state:                               link state:
 link-credit = 5                           link-credit = 6

Initially, link-credit is 3. The receiver decides to send a flow frame, setting the new link-credit to 6. In parallel, the sender sends a transfer frame.

Since the receiver received the transfer frame after it sent the flow frame, it will compute its new link-credit as 6 - 1 = 5. However, because the sender received the flow frame after it sent the transfer frame, it will set the credit to 6. As a result, the state - and therefore the view of how many credits the link has left - becomes misaligned. This is problematic because the sender could potentially overflow the receiver.

To prevent such misalignments, a second field is needed in both the link state and the flow frame:

<field name="delivery-count" type="sequence-no"/>

The delivery-count is increased by 1 each time a message is transferred. Specifically, the sender increments the delivery-count whenever it sends a message, and the receiver increments the delivery-count whenever it receives a message.

When the sender receives a flow frame (which contains both link-credit and delivery-count), the sender sets its link-credit according to this formula:

link-credit(snd) := delivery-count(rcv) + link-credit(rcv) - delivery-count(snd).

(snd) refers to the link state at the sender, while the link state at the receiver (rcv) is sent within the flow frame to the sender.

At the sender, this formula means: "Set the new link-credit to the link-credit I just received in the flow frame minus any in-flight deliveries."

The purpose of the delivery-count is to establish an order in the sequence of events, which are:

• Sender sends message
• Receiver receives message
• Receivers grants link-credit
• Sender computes receiver's link-credit

Using the delivery-count resolves the misalignment issue we discussed earlier.

Let’s assume the delivery-count is initially set to 20:

Receiver                                      Sender
========================================================================================
                                  ...
link state:                                   link state:
 delivery-count = 20                           delivery-count = 20
 link-credit = 3                               link-credit = 3

flow(delivery-count = 20,
     link-credit = 6)         ---+   +---     transfer(...)
                                  \ /
                                   x
                                  / \
                              <--+   +-->

link state:                                   link state:
 delivery-count = 21                           delivery-count = 21
 link-credit = 5                               link-credit = 20+6-21 = 5 (above formula)

info

From what I can tell, the AMQP 1.0 link flow control seems to be based on the paper Credit-Based Flow Control for ATM Networks from 1995.

The formula

link-credit(snd) := delivery-count(rcv) + link-credit(rcv) - delivery-count(snd).

from the AMQP 1.0 specification in section 2.6.7 matches the formula

Crd_Bal = Buf_Alloc - (Tx_Cnt - Fwd_Cnt)(1)

in the paper.

The AMQP 1.0 specification even adopts similar terminology from the paper, such as "link" and "node." Furthermore, the specification likely refers to "delivery-count" because it is called "count" in the paper as well (even though the specification clarifies that in AMQP it's not actually a count, but rather a sequence number).

Figure 2 in the paper illustrates the concept nicely:

In this figure:

• Tx_Cnt corresponds to delivery-count(snd)
• Fwd_Cnt corresponds to delivery-count(rcv)
• Crd_Bal corresponds to link-credit(snd)
• Buf_Alloc corresponds to link-credit(rcv)

The paper explains in detail how frequently and by how much a receiver should top up link credit. If you want to become an expert in AMQP 1.0 flow control, I recommend reading both the AMQP 1.0 specification and the paper.

Drain​

Having understood the two most important link control fields of the flow frame (link-credit and delivery-count), let's move on to the drain field:

<field name="drain" type="boolean" default="false"/>

By default, the drain field is set to false. The receiver decides whether drain mode is enabled, and the sender sets drain to the last known value from the receiver.

Draining means that the sender should use all link credit from the receiver by sending available messages. If there are not enough messages to send, the sender must still exhaust all link credit as follows:

1. Advance the delivery-count by the remaining link-credit.
2. Set link-credit to 0.
3. Send a flow frame to the receiver.

By setting the drain field to true, the consumer requests RabbitMQ to "Either send a transfer or a flow frame." If the source queue is empty, RabbitMQ will promptly reply with only a flow frame.

Therefore, the drain field allows a consumer to set a timeout when synchronously getting messages, as shown in Figure 2.44: Synchronous Get with a Timeout of the AMQP 1.0 specification:

    Receiver                                      Sender
    =================================================================
                                          ...
    flow(link-credit=1)               ---------->
  *wait for link-credit <= 0*
    flow(drain=True)                  ---+   +--- transfer(...)
                                          \ /
                                           x
                                          / \
(1)                                   <--+   +-->
(2)                                   <---------- flow(...)
                                          ...
    -----------------------------------------------------------------
      (1) If a message is available within the timeout, it will
          arrive at this point.
      (2) If a message is not available within the timeout, the
          drain flag will ensure that the sender promptly advances the
          delivery-count until link-credit is consumed.

Because the link-credit is consumed quickly, the consumer can unambiguously determine whether a message was received or if the operation timed out.

Echo​

Similar to the drain field, the echo field is:

• set to false by default
• determined by the consumer, as RabbitMQ does not set this field in its current implementation

<field name="echo" type="boolean" default="false"/>

The consumer can set the echo field to request RabbitMQ to reply with a flow frame. One use case is depicted in Figure 2.46: Stopping Incoming Messages of the AMQP 1.0 specification:

    Receiver                                       Sender
    ================================================================
                                           ...
                                       <---------- transfer(...)
    flow(...,                          ---+   +--- transfer(...)
         link-credit=0,                    \ /
         echo=True)                         x
                                           / \
(1)                                    <--+   +-->
(2)                                    <---------- flow(...)
                                           ...
    ----------------------------------------------------------------
      (1) In-flight transfers can still arrive until the flow state
          is updated at the sender.
      (2) At this point no further transfers will arrive.

Available​

RabbitMQ sets the available field in the flow frame to inform the consumer how many messages are available:

<field name="available" type="uint"/>

The available value is only an approximation because, from the time the queue emits this information until the flow frame arrives at the consumer, this information can already be outdated, for instance when other clients publish messages to or consume messages from this queue.

For classic queues and quorum queues, available indicates the number of messages ready for delivery (i.e. the queue length excluding messages that are checked out to consumers).

For streams, available represents the difference between the committed offset and the last consumed offset. Roughly, the committed offset is what the different stream replicas in the RabbitMQ cluster agree is the end of the stream. The last consumed offset might be the same as the committed offset or lag behind it. The available value is merely an estimate because a stream offset does not necessarily represent a message.

If the Single Active Consumer feature is enabled, quorum queues will return available = 0 for all inactive (waiting) consumers. This makes sense since no messages are available for inactive consumers, regardless of how much credit they have topped up.

In the previous section, we learned that one use case for the consumer setting the echo field is to stop a link. Another use case is when a consumer wants to learn about the number of messages available in the queue it consumes from.

The available field could also be set by a publisher to tell RabbitMQ how many messages the publisher has available to send. Currently, RabbitMQ ignores this information.

Properties​

The properties field of the flow frame can carry application specific link state properties:

<field name="properties" type="fields"/>

Currently, RabbitMQ does not make use of this field.

Summary​

We learned that AMQP 1.0 link flow control protects individual consumers or queues from being overwhelmed with messages. The receiver periodically provides feedback to the sender on how many messages it can currently handle.

Each link endpoint (sender and receiver) maintains full link flow control state and exchanges this state via flow frames with the other side.

The following link flow control fields are independently determined by the receiver:

• link-credit
• drain

The following link flow control fields are independently determined by the sender:

• delivery-count
• available

Session Flow Control​

Before diving into session flow control, let's recap what a session actually is.

Session​

A client library creates a single TCP connection per AMQP connection. An AMQP 1.0 connection is established using the open frame.

Within an AMQP 1.0 connection, the client can then start multiple AMQP 1.0 sessions. This is analogous to how an AMQP 0.9.1 client can open multiple AMQP 0.9.1 channels within an AMQP 0.9.1 connection. An AMQP 1.0 session is started using the begin frame.

Within a session, a client can then create links by sending the attach frame.

  Client App                                       RabbitMQ
+-------------+                                +-------------+
|             |################################|             |
|   +---+     |--------------------------------|    +---+    |
|   | C |O<===============================+========O| Q |    |
|   +-+-+ \   |-------------------+-------|----|   |+-+-+    |
|     |    \  |#######+###########|#######|####|   |  |      |
+-----|-----\-+       |           |       |    +---|--|------+
      |      \        |           |       |        |  |
      |     Target    |           |       |    Source |
      |               |           |       |           |
   Consumer           |           |     Link        Queue
                      |       Session
                 Connection

A client application could create, for example:

• A single connection with a single session, or
• A single connection with multiple sessions, or
• Multiple connections with one or more sessions each

Usually, a single connection with a single session is sufficient.

Opening an AMQP connection comes with some overhead: A TCP connection must be established, allocating operating system resources such as sockets and TCP buffers in both the client and server, and incurring latency for the TCP or TLS handshake. Additionally, on the RabbitMQ node, a supervision tree is created for each incoming AMQP connection.

A session can be thought of as a "lightweight" connection. Each AMQP 1.0 session is currently implemented as its own Erlang process. Hence, creating a session is inexpensive if the connection is already established.

Creating a second session within an AMQP connection might be useful in the following scenarios:

• Quickly and cheaply creating another "virtual connection" without incurring the aforementioned AMQP connection setup overhead.
• Ensuring high-priority transfer frames (requiring low latency) are not blocked by other transfer frames.
• Increasing parallelism when the RabbitMQ session process becomes very busy (for example, routing messages to queues).

Flow Fields​

Sessions provide a flow control scheme based on the number of transfer frames transmitted. Since frames have a maximum size for a given connection, this provides flow control based on the number of bytes transmitted.

Remember that a large message is split into multiple transfer frames.

Session flow control in AMQP 1.0 operates at a higher layer than link flow control. While link flow control protects individual consumers and queues, session flow control is designed to protect the entire client application and RabbitMQ as a whole.

Just as each link endpoint maintains link flow control state and exchanges this state via flow frames, each session endpoint maintains session flow control state and exchanges this state within the same flow frames. The remaining fields in the flow frame are therefore used for session flow control:

<field name="next-incoming-id" type="transfer-number"/>
<field name="incoming-window" type="uint" mandatory="true"/>
<field name="next-outgoing-id" type="transfer-number" mandatory="true"/>
<field name="outgoing-window" type="uint" mandatory="true"/>

The incoming-window is similar to the link-credit field in that the receiving side informs the sending side how many "units" it can tolerate receiving. The difference is that for link-credit, the unit is a potentially large application message, while for incoming-window, the unit is a transfer frame.

To provide an extreme example for better understanding: if the negotiated max-frame-size on the connection is very low and a single message is very large, if the receiver grants one link credit to the sender, the sender might still be blocked by session flow control sending this message in its entirety.

next-incoming-id and next-outgoing-id are sequence numbers and serve the same purpose as the delivery-count in link flow control: they ensure that the windows are computed correctly on the other side when transfer frames are sent in parallel with flow frames. Two sequence numbers are needed for session flow control as opposed to one for link flow control because a session is bidirectional, while a link is unidirectional.

Initially, RabbitMQ allows the publisher to send 400 transfer frames. Whenever the RabbitMQ session process has processed half of that number (200 transfer frames), RabbitMQ expands this window by sending a flow frame to the publisher containing incoming_window = 400.

The value of 400 is configurable via advanced.config setting rabbit.max_incoming_window.

RabbitMQ Alarms​

The only exception to this rule is when a memory or disk alarm is triggered. To protect RabbitMQ from running out of memory or disk space, each session will close its incoming window by sending a flow frame with incoming_window = 0 to publishers, effectively preventing them from sending any further transfer frames.

Incoming-Window​

The disadvantage of AMQP 1.0 flow control is its complexity. While the ideas and motivations behind link flow control and session flow control are sound, implementing one layer (session flow control) on top of another layer (link flow control) can be challenging to get right and efficient in every scenario. Consider that clients can:

• Modify session flow control and link flow control independently (either in the same flow frame or in separate flow frames).
• Send a flow frame at any time and at different frequencies.
• Dynamically increase or decrease either the session window or the link credit.
• Grant a link-credit of 0 (causing the queue to stop sending) or a huge number of link credits (up to 4 billion).
• Close its session incoming-window (causing the server session to stop sending) or open it widely (up to 4 billion).
• Stop reading from its socket, applying TCP backpressure to the server.
• Add other special logic, such as drain=true, into the mix.

In AMQP 1.0 programs, concurrency is desirable. Depending on the programming language, different parts of the client or broker are implemented by different threads or processes.

RabbitMQ achieves concurrency by running the connection reader process (parsing frames), connection writer process (serializing frames), session process (e.g. routing messages), and queue process (storing and forwarding messages) in different Erlang processes.

In a scenario where the client grants a huge number of link-credit to the sending queue but maintains a very small session incoming-window, queue processes are allowed to deliver messages, while the session process is not allowed to forward them to the client. In this case, messages are buffered in the session process until the client's incoming-window allows them to be forwarded.

To guard against this scenario - where a large number of messages could be buffered in the session process - RabbitMQ, in its current implementation, internally grants link credits from the session process to the queue process in batches of at most 256 link credits. Even if the client grants a large number of link credits, the queue will only ever see up to 256 credits for a given consumer. Once these 256 messages have been sent out by the server session process, the server session process will grant the next batch of 256 credits to the queue.

The value of 256 is configurable via advanced.config setting rabbit.max_queue_credit.

Wrapping Up​

This blog post explained how AMQP 1.0 link flow control and session flow control work.

We learned how individual processes within RabbitMQ protect themselves from overload:

• Queue processes are protected by link flow control.
• Session processes and RabbitMQ as a whole are protected by session flow control.
• Connection reader processes are protected by applying TCP backpressure when they cannot read frames quickly enough (which should be rare).

This blog post also highlighted ten benefits of flow control in AMQP 1.0 over flow control in AMQP 0.9.1. The main advantages include:

• Fine-grained control for consumers, allowing them to determine at any point in time the exact number of messages they want to consume from specific queue(s).
• Higher throughput for publishing and consuming on a single AMQP connection when a target queue reaches its limits. We ran two benchmarks:
  1. Total send throughput of AMQP 1.0 was four times higher than AMQP 0.9.1 when sending to two different target queues.
  2. In an extreme case, consumption rate of AMQP 1.0 was 73 times higher than AMQP 0.9.1 when receiving from one queue while sending too fast to another queue.
• Safe and efficient usage of a single connection for both publishing and consuming.
• The ability for consumers to stop and resume at any time.
• Extensibility for future use cases.

• message priorities
• consumer priorities combined with Single Active Consumer
• default delivery limit is now 20 (breaking change!)
• faster recovery of long queues

Message Priorities​

Support for message priorities has been probably the most demanded quorum queue feature, mostly requested by existing classic mirrored queue users who wanted to migrate to quorum queues (remember, support for classic queue mirroring was removed for 4.0).

However, the way priorities are supported is significantly different compared to how classic queues handle them. Classic queues require x-max-priority argument to define the maximum number of priorities for a given queue (if this argument is not provided, the queue will treat all messages equally). Values from 0 to 255 are technically allowed, although no more than a handful of priorities per queue should really be used. Quorum queues don't require any upfront declaration (no need to enable priorities for a given queue), but there are exactly two priorities per queue: normal and high. The behavior matches that of AMQP 1.0 specification (see chapter 3.2.1 of the AMQP 1.0 specification):

• a priority value between 0 and 4 (inclusive) is treated as the normal priority
• any value above 4 is considered a high priority
• if the publisher doesn't specify the priority of a message, the value of 4 is assumed (normal priority)

If a quorum queue contains both normal and high priority messages, consumers will receive a mix of both, with a ratio of 2 high priority messages for every 1 normal priority message. This approach avoids starvation, since regardless of the number of high priority messages, a progress is made on processing the normal priority messages as well. This is in contrast with the classic queue implementation, which will always deliver higher priority messages first, if there are any, and therefore the normal priority messages may never get delivered (or, more likely, their delivery latency will be very high).

Here's a visual representation of how this works. In preparation for this test, we first published 100k normal priority messages and then 100k high priority messages. Since quorum queues were not priority-aware until 4.0, if we did that in an older version and then start a consumer, it would simply receive the normal priority messages first (as they are older) and then all the high priority messages. With 4.0, we can see that the consumer immediately starts receiving a mix of about 1500 normal priority messages per second and twice as many high priority messages, for a total of ~4500 messages per second (the actual delivery rates are not important here, they depend on many factors; the 2:1 high/normal priority ratio is what matters in the context of priorities). Once the queue delivers all the high priority messages, the consumer starts receiving ~4500 normal priority messages per second - as many as it can handle in this test scenario. The blue dotted line (with the axis scale on the right) is the number of ready messages in the queue (total for both priorities) - we can see it starts at 200k and ultimately drops to zero.

Let's consider the opposite scenario - what if we publish all the high priority messages first and only then all the normal priority messages? In this case, the consumer will receive the messages in order of publishing. There's simply no reason for a normal priority message to overtake a higher priority message.

# declare a quorum queue (you can use the Management UI or any other method)
rabbitmqadmin declare queue name=qq queue_type=quorum
# publish normal priority messages (10 publishers, 10k messages each)
omq amqp --publishers 10 --consumers 0 --publish-to /queues/qq --message-priority 1 --pmessages 10000
# publish high priority messages
omq amqp --publishers 10 --consumers 0 --publish-to /queues/qq --message-priority 10 --pmessages 10000
# consume all messages from the queue
omq amqp --publishers 0 --consumers 1 --consume-from /queues/qq --consumer-credits 100

What If I Need More Control?​

If two priorities with a 2:1 delivery ratio doesn't meet your requirements, we can recommend two things:

1. Reconsider your requirements. 😄 Reasoning about the message delivery order with many priorities is really hard. It's probably easier to make sure that all your messages are delivered sufficiently quickly and use the priorities just to make sure that in case of an occasional long backlog, important messages can skip the queue.
2. If you really need more priorities and/or more control over how different priorities are handled, using multiple queues is your best bet. You can develop a consumer that subscribes to multiple queues and then decides which queue to consume from.

Consumer Priorities Combined with Single Active Consumer​

Starting with RabbitMQ 4.0, quorum queues will take the consumer priority into account when selecting the single active consumer. If a higher priority becomes available (subscribes), a quorum queue will switch over to it. This is particularly useful if you have multiple queues that should have a single consumer each, but you don't want a single instance of your application to be the consumer for all of them, which is likely to happen when the first application instance that starts, subscribes to all those single-active-consumer queues. Now you can pick a different priority when subscribing to different queues, to make sure that each instance consumes only from its "favorite" queue and only serves as a backup consumer for other queues.

To better explain this functionality, let's review all the moving parts. A Single Active Consumer is a queue argument which prevents a queue from delivering messages to more than one consumer, regardless of how many subscribed to the queue. One consumer is active, all other consumers are not. If the active consumer disconnects, one of the other consumers is activated. This feature is used if a strict message processing order needs to be maintained.

Consumer Priority allows you to specify that rather than delivering messages to all subscribed consumers in a fair round-robin fashion (which is the default behavior of both classic and quorum queues), a certain consumer should be preferred.

Until version 4.0, these features were effectively mutually exclusive - if Single Active Consumer was enabled, a new consumer would never become active, regardless of its priority, as long as the previous consumer remained active. Starting with 4.0, if the new consumer's priority is higher than the currently active consumer's, the quorum queue will switch over to the higher priority consumer: it will stop delivering messages to the current consumer, wait for all the messages to be acknowledged, and then will deactivate the old consumer, and activate the higher priority consumer instead.

The graph below shows this behavior. There are three metrics on this graph:

• the green line shows the number of messages consumed by the first (default priority) consumer (which happens to be configured to consume 10 msgs/s)
• yellow, shows the same value but for the second, higher priority consumer
• blue, shows the number of unacknowledged messages (axis scale on the right)

Initially, we only have one consumer and as expected, it consumes 9-10 msgs/s (these jumps between 9 and 10 are simply a result of how the metrics are emitted and then displayed). This consumer is configured with the prefetch of 1000 messages and since there were many messages in the queue - the prefetch buffer is used to the maximum. Then the yellow line appears, initially at 0 msgs/s. This is the higher priority consumer, which is already connected, but not yet active. From the moment it connected, we can see the number of unacknowledged messages going down, since the queue no longer delivers messages to the original consumer. Once all of them are acknowledged, the new consumer becomes the single active consumer and receives 1000 messages, since that's its prefetch value. It then happily consumes around 10 msgs/s as configured.

# Publish 5000 messages to have a backlog (perf-test will declare a quorum queue `qq-sac`)
perf-test --quorum-queue --queue qq-sac --pmessages 5000 --confirm 100 -qa x-single-active-consumer=true --consumers 0
# Start a consumer with the default priority and prefetch of 1000; consume ~10 msgs/s
perf-test --producers 0 --predeclared --queue qq-sac --consumer-latency 100000 --qos 1000
# In another window, some time after starting the first consumer, start a higher priority consumer
perf-test --producers 0 --predeclared --queue qq-sac --consumer-latency 100000 --qos 1000 --consumer-args x-priority=10

Delivery Limit is Now 20 by Default​

Quorum queues now have the delivery limit set to 20 by default. In the past, the limit wasn't set and therefore was quorum queues would attempt the delivery forever, until the message is either acknowledged or discarded by the consumer. This could lead to a situation where a message is stuck in the queue and can never be delivered.

The downside of this change is that if dead lettering is not configured, messages will be dropped after 20 attempts. Therfore, it is highly recommended to configure dead lettering for all quorum queues.

Faster Recovery of Long Queues​

This is less of a feature and more an internal change, but certainly worth mentioning. Up until now, if a RabbitMQ node was restarted, all quorum queues on that node had to read through all of their data (the Raft log) since the last snapshot to re-build their in-memory state. For example, if you publish a few million messages to a quorum queue right now and then restart a node, you will see that after the node is up, the queue will report 0 ready messages for quite some time (at least a few seconds) and you won't be able to start consuming these messages. The queue is simply not yet ready to serve traffic - it's still reading the data from disk (note: this doesn't mean that all that data is then kept in memory, a vast majority of it is not, but an index / summary of the queue data is). Starting with RabbitMQ 4.0, quorum queues create checkpoint files which include the state of the queue at a certain point in time. Upon startup, the queue can read the most recent checkpoint and only the part of the Raft log from that point in time. These means that quorum queues take significantly less time to start.

For example, a RabbitMQ node with one quorum queue containing 10 million 12-byte messages, takes about 30 seconds to start on my machine. With RabbitMQ 4.0, it just takes a fraction of a second.

You may wonder what the difference is between a snapshot and a checkpoint. In many ways, they are the same - they actually share the code that writes them to disk. The difference is that a snapshot is only created when the Raft log is truncated. For many common queue use cases, this all that is needed - older messages are consumed, we create a snapshot that no longer contains them and we truncate the log. At this point the queue has no memory of those messages ever being present. Checkpoints on the other hand, are created periodicailly when we can't truncate the log. The test case scenario is a good example - since we didn't consume any messages, the oldest messages are still there, we can't just forget about them. But a checkpoint still allows the queue to start more quickly. A checkpoint can be promoted to a snapshot when the log is truncated (in this example - after some of the older messages are consumed).

# Publish 10 million 12-byte messages (feel free to play with other values)
perf-test --quorum-queue --queue qq --consumers 0 --pmessages 5000000 --confirm 1000 --producers 2
# restart the node
rabbitmqctl stop_app && rabbitmqctl start_app
# list the queues (repeat this command until the number of messages is 10 million instead of 0)
rabbitmqctl list_queues

Summary​

RabbitMQ 4.0 is a significant milestone for RabbitMQ. With the removal of classic queue mirroring, quorum queues become the only option for highly available, replicated queues (note: streams are also highly available and replicated, but technically not queues; nevertheless, they might still be a good choice for some use cases where classic mirrored queues were used in the past). Quorum queues have offered higher data safety guarantees and much better performance than mirrored queues for years and with these latest improvements, they become even more robust and performant in a wider range of scenarios.

You can play with RabbitMQ 4.0 beta now: https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.0-beta.5

Additionally, this blog post suggests that AMQP 1.0 can perform slightly better than AMQP 0.9.1 in RabbitMQ 4.0.

[
 {rabbit, [
  {loopback_users, []}
 ]},

 {rabbitmq_management_agent, [
  {disable_metrics_collector, true}
 ]}
].

make run-broker \
    TEST_TMPDIR="$HOME/scratch/rabbit/test" \
    RABBITMQ_CONFIG_FILE="$HOME/scratch/rabbit/advanced.config" \
    PLUGINS="rabbitmq_prometheus rabbitmq_management rabbitmq_amqp1_0" \
    RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 3"

$ docker run -it --rm --add-host host.docker.internal:host-gateway ssorj/quiver:latest
bash-5.1# quiver --version
quiver 0.4.0-SNAPSHOT

Classic Queues​

This section benchmarks classic queues.

We declare a classic queue called my-classic-queue:

deps/rabbitmq_management/bin/rabbitmqadmin declare queue \
    name=my-classic-queue queue_type=classic durable=true

AMQP 1.0 in 4.0​

The client sends and receives 1 million messages. Each message contains a payload of 12 bytes. The receiver repeatedly tops up 200 link credits at a time.

# quiver //host.docker.internal//queues/my-classic-queue \
    --durable --count 1m --duration 10m --body-size 12 --credit 200

RESULTS

Count ............................................. 1,000,000 messages
Duration ............................................... 10.1 seconds
Sender rate .......................................... 99,413 messages/s
Receiver rate ........................................ 99,423 messages/s
End-to-end rate ...................................... 99,413 messages/s

Latencies by percentile:

          0% ........ 0 ms       90.00% ........ 1 ms
         25% ........ 1 ms       99.00% ........ 2 ms
         50% ........ 1 ms       99.90% ........ 2 ms
        100% ........ 9 ms       99.99% ........ 9 ms

AMQP 1.0 in 3.13​

# quiver //host.docker.internal//amq/queue/my-classic-queue \
    --durable --count 1m --duration 10m --body-size 12 --credit 200

RESULTS

Count ............................................. 1,000,000 messages
Duration ............................................... 45.9 seconds
Sender rate .......................................... 43,264 messages/s
Receiver rate ........................................ 21,822 messages/s
End-to-end rate ...................................... 21,790 messages/s

Latencies by percentile:

          0% ....... 67 ms       90.00% .... 24445 ms
         25% .... 23056 ms       99.00% .... 24780 ms
         50% .... 23433 ms       99.90% .... 24869 ms
        100% .... 24873 ms       99.99% .... 24873 ms

The same benchmark against RabbitMQ 3.13 results in 4.5 times lower throughput.

---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
-----------------------------------------------------  -----------------------------------------------------  --------
     2.1        130,814      65,342        8     79.1       2.1          3,509       1,753        1      7.5       777
     4.1        206,588      37,849        6     79.1       4.1          5,995       1,242        0      7.5     2,458
     6.1        294,650      43,987        6     79.1       6.1          9,505       1,753        1      7.5     5,066
     8.1        360,184      32,734        5     79.4       8.1         13,893       2,194        0      7.5     6,190
    10.1        458,486      49,102        6     79.4      10.1         15,793         950        1      7.5     9,259
    12.1        524,020      32,734        5     79.4      12.1         21,644       2,923        1      7.5    11,163
    14.1        622,322      49,102        5     79.4      14.1         25,154       1,753        1      7.5    13,451
    16.1        687,856      32,734        4     79.4      16.1         27,639       1,241        1      7.5    15,246
    18.1        786,158      49,102        6     81.0      18.1         30,124       1,241        1      7.5    17,649
    20.1        884,460      49,102        6     81.0      20.1         32,610       1,242        1      7.5    19,408
    22.1        949,994      32,734        4     81.0      22.1         35,535       1,462        0      7.5    21,293
    24.1        999,912      24,934        4     81.8      24.1         38,167       1,315        1      7.5    23,321
    26.1        999,974          31        2      0.0      26.1        117,745      39,749       11      7.5    24,475
       -              -           -        -        -      28.1        202,589      42,380       11      7.5    24,364
       -              -           -        -        -      30.1        292,554      44,938       13      7.5    24,244
       -              -           -        -        -      32.1        377,691      42,526       15      7.5    23,955
       -              -           -        -        -      34.1        469,704      45,961       14      7.5    23,660
       -              -           -        -        -      36.1        555,719      42,965       12      7.5    23,463
       -              -           -        -        -      38.1        649,048      46,618       12      7.5    23,264
       -              -           -        -        -      40.1        737,696      44,280       15      7.5    23,140
       -              -           -        -        -      42.1        826,491      44,353       15      7.5    23,100
       -              -           -        -        -      44.1        917,187      45,303       16      7.5    23,066
       -              -           -        -        -      46.1        999,974      41,394       14      0.0    22,781

AMQP 0.9.1 in 4.0​

For our AMQP 0.9.1 benchmarks we use PerfTest. We try to run a somewhat fair comparison of our previous AMQP 1.0 benchmark.

Since an AMQP 1.0 /queues/:queue target address sends to the default exchange, we also send to the default exchange via AMQP 0.9.1. Since we used durable messages with AMQP 1.0, we set the persistent flag in AMQP 0.9.1. Since RabbitMQ settles with the released outcome when a message cannot be routed, we set the mandatory flag in AMQP 0.9.1. Since RabbitMQ v4.0.0-beta.5 uses a default rabbit.max_link_credit of 128 granting 128 more credits to the sending client when remaining credit falls below 0.5 * 128, we configure the AMQP 0.9.1 publisher to have at most 1.5 * 128 = 192 messages unconfirmed at a time. Since we used 200 link credits in the previous run, we configure the AMQP 0.9.1 consumer with a prefetch of 200.

$ java -jar target/perf-test.jar \
    --predeclared --exchange amq.default \
    --routing-key my-classic-queue --queue my-classic-queue \
    --flag persistent --flag mandatory \
    --pmessages 1000000 --size 12 --confirm 192 --qos 200 --multi-ack-every 200

id: test-151706-485, sending rate avg: 88534 msg/s
id: test-151706-485, receiving rate avg: 88534 msg/s
id: test-151706-485, consumer latency min/median/75th/95th/99th 99/975/1320/1900/2799 µs
id: test-151706-485, confirm latency min/median/75th/95th/99th 193/1691/2113/2887/3358 µs

Summary​

Quorum Queues​

This section benchmarks quorum queues.

We declare a quorum queue called my-quorum-queue:

deps/rabbitmq_management/bin/rabbitmqadmin declare queue \
    name=my-quorum-queue queue_type=quorum durable=true

Flow Control Configuration​

For highest data safety, quorum queues fsync all Ra commands including:

• enqueue: sender enqueues a message
• settle: receiver accepts a message
• credit: receiver tops up link credit

Before a quorum queue confirms receipt of a message to the publisher, it ensures that any file modifications are flushed to disk, making the data safe even if the RabbitMQ node crashes shortly after.

The SSD of my Linux box is slow, taking 5-15 ms per fsync. Since we want to compare AMQP protocol implementations without being bottlenecked by a cheap disk, the tests in this section increase flow control settings:

[
 {rabbit, [
  {loopback_users, []},

  %% RabbitMQ internal flow control for AMQP 0.9.1
  %% Default: {400, 200}
  {credit_flow_default_credit, {5000, 2500}},

  %% Maximum incoming-window of AMQP 1.0 session.
  %% Default: 400
  {max_incoming_window, 5000},

  %% Maximum link-credit RabbitMQ grants to AMQP 1.0 sender.
  %% Default: 128
  {max_link_credit, 2000},

  %% Maximum link-credit RabbitMQ AMQP 1.0 session grants to sending queue.
  %% Default: 256
  {max_queue_credit, 5000}
 ]},

 {rabbitmq_management_agent, [
  {disable_metrics_collector, true}
 ]}
].

This configuration allows more Ra commands to be batched before RabbitMQ calls fsync. For production use cases, we recommend enterprise-grade high performance disks that fsync faster, in which case there is likely no need to increase flow control settings.

RabbitMQ flow control settings present a trade-off:

• Low values ensure stability in production.
• High values can result in higher performance for individual connections but may lead to higher memory spikes when many connections publish large messages concurrently.

RabbitMQ uses conservative flow control default settings to favour stability in production over winning performance benchmarks.

AMQP 1.0 in 4.0​

# quiver //host.docker.internal//queues/my-quorum-queue \
    --durable --count 1m --duration 10m --body-size 12 --credit 5000

RESULTS

Count ............................................. 1,000,000 messages
Duration ............................................... 12.0 seconds
Sender rate .......................................... 83,459 messages/s
Receiver rate ........................................ 83,396 messages/s
End-to-end rate ...................................... 83,181 messages/s

Latencies by percentile:

          0% ........ 9 ms       90.00% ....... 47 ms
         25% ....... 27 ms       99.00% ....... 61 ms
         50% ....... 35 ms       99.90% ....... 76 ms
        100% ....... 81 ms       99.99% ....... 81 ms

# quiver //host.docker.internal//queues/my-quorum-queue \
    --durable --count 1m --duration 10m --body-size 12 --credit 5000

RESULTS

Count ............................................. 1,000,000 messages
Duration .............................................. 100.2 seconds
Sender rate ........................................... 9,986 messages/s
Receiver rate ......................................... 9,987 messages/s
End-to-end rate ....................................... 9,983 messages/s

Latencies by percentile:

          0% ....... 10 ms       90.00% ....... 24 ms
         25% ....... 14 ms       99.00% ....... 30 ms
         50% ....... 18 ms       99.90% ....... 38 ms
        100% ....... 55 ms       99.99% ....... 47 ms

(15,493 - 1,244) * 5.9 ms = 84 seconds

AMQP 1.0 in 3.13​

# quiver //host.docker.internal//amq/queue/my-quorum-queue \
    --durable --count 1m --duration 10m --body-size 12 --credit 5000

---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
-----------------------------------------------------  -----------------------------------------------------  --------
     2.1        163,582      81,709       11     84.2       2.1         29,548      14,759        3      7.5       840
     4.1        336,380      86,356       12    185.3       4.1         29,840         146        0      7.5     2,331
     6.1        524,026      93,729       14    328.0       6.1         29,840           0        0      7.5         0
     8.1        687,864      81,837       11    462.3       8.1         31,302         730        1      7.5     6,780
    10.1        884,470      98,303       14    605.4      10.1         31,447          72        0      7.5     7,897
    12.1        999,924      57,669        7    687.5      12.1         31,447           0        0      7.5         0
    14.1        999,924           0        0    687.5      14.1         31,447           0        0      7.5         0
    16.1        999,924           0        0    687.5      16.1         31,447           0        1      7.5         0
    18.1        999,924           0        1    688.3      18.1         31,447           0        0      7.5         0
receiver timed out
    20.1        999,924           0        0    688.3      20.1         31,447           0        0      7.5         0

RabbitMQ 3.13 cannot handle this workload and the benchmark fails.

# quiver //host.docker.internal//amq/queue/my-quorum-queue \
    --durable --count 1m --duration 10m --body-size 12 --credit 5000

---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
-----------------------------------------------------  -----------------------------------------------------  --------
     2.1        130,814      65,342        9     70.0       2.1         26,915      13,437        6      7.5     1,213
     4.1        196,348      32,718        5     70.2       4.1         28,084         584        0      7.5     3,093
     6.1        261,882      32,734        7     70.2       6.1         30,131       1,022        1      7.5     4,952
     8.1        360,184      49,126        6     70.2       8.1         32,325       1,096        0      7.5     6,637
    10.1        425,718      32,734        6     70.2      10.1         34,225         949        1      7.5     8,089
    12.1        491,252      32,734        5     70.2      12.1         34,225           0        0      7.5         0
    14.1        589,554      49,102        7     70.2      14.1         34,225           0        0      7.5         0
    16.1        655,088      32,734        5     70.2      16.1         34,225           0        0      7.5         0
    18.1        720,622      32,734        6     70.2      18.1         34,225           0        0      7.5         0
receiver timed out

AMQP 0.9.1 in 4.0​

Since we set max_link_credit to 2,000, we allow for a maximum of 2,000 * 1.5 = 3,000 unconfirmed messages in the publisher.

$ java -jar target/perf-test.jar \
    --predeclared --exchange amq.default \
    --routing-key my-quorum-queue --queue my-quorum-queue \
    --flag persistent --flag mandatory \
    --pmessages 1000000 --size 12 --confirm 3000 --qos 5000 --multi-ack-every 5000

id: test-085526-136, sending rate avg: 70067 msg/s
id: test-085526-136, receiving rate avg: 70067 msg/s
id: test-085526-136, consumer latency min/median/75th/95th/99th 8803/33127/40424/53407/62883 µs
id: test-085526-136, confirm latency min/median/75th/95th/99th 8551/30323/38317/52103/63131 µs

$ java -jar target/perf-test.jar \
    --predeclared --exchange amq.default \
    --routing-key my-quorum-queue --queue my-quorum-queue \
    --flag persistent --flag mandatory \
    --pmessages 1000000 --size 12 --confirm 192 --qos 5000 --multi-ack-every 5000

id: test-084359-441, sending rate avg: 9931 msg/s
id: test-084359-441, receiving rate avg: 9931 msg/s
id: test-084359-441, consumer latency min/median/75th/95th/99th 7512/17054/26256/34249/38641 µs
id: test-084359-441, confirm latency min/median/75th/95th/99th 9432/16586/23918/32636/36858 µs

Summary​

Streams​

This sections benchmarks streams.

We declare a stream called my-stream:

deps/rabbitmq_management/bin/rabbitmqadmin declare queue \
    name=my-stream queue_type=stream durable=true

(We run with default RabbitMQ flow control settings.)

We want the receiver to start consuming from the very beginning of the stream. Quiver doesn't support passing a filter field to the source where we could specify a rabbitmq:stream-offset-spec value first. Therefore, for this benchmark it's easier to patch RabbitMQ to use stream offset spec first by default instead of next:

diff --git a/deps/rabbit/src/rabbit_stream_queue.erl b/deps/rabbit/src/rabbit_stream_queue.erl
index e36ad708eb..acd193d76f 100644
--- a/deps/rabbit/src/rabbit_stream_queue.erl
+++ b/deps/rabbit/src/rabbit_stream_queue.erl
@@ -344,7 +344,7 @@ consume(Q, Spec, #stream_client{} = QState0)
                        {term(), non_neg_integer()}) ->
     {ok, osiris:offset_spec()} | {error, term()}.
 parse_offset_arg(undefined) ->
-    {ok, next};
+    {ok, first};
 parse_offset_arg({_, <<"first">>}) ->
     {ok, first};
 parse_offset_arg({_, <<"last">>}) ->

AMQP 1.0 in 4.0​

# quiver //host.docker.internal//queues/my-stream \
    --durable --count 1m --duration 10m --body-size 12 --credit 5000
---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
-----------------------------------------------------  -----------------------------------------------------  --------
     2.1        278,782     139,321       25      8.0       2.1        215,185     107,539       22      7.6       224
     4.1        554,492     137,717       25      8.0       4.1        434,027     109,312       24      7.6       651
     6.1        825,082     135,160       25      8.0       6.1        650,236     107,997       26      7.6     1,079
     8.1        999,992      87,368       17      0.0       8.1        888,973     119,249       29      7.6     1,469
       -              -           -        -        -      10.1        999,993      55,455       13      0.0     1,583

RESULTS

Count ............................................. 1,000,000 messages
Duration ................................................ 8.9 seconds
Sender rate ......................................... 136,705 messages/s
Receiver rate ....................................... 112,587 messages/s
End-to-end rate ..................................... 112,196 messages/s

Latencies by percentile:

          0% ........ 7 ms       90.00% ..... 1553 ms
         25% ...... 519 ms       99.00% ..... 1612 ms
         50% ..... 1011 ms       99.90% ..... 1615 ms
        100% ..... 1616 ms       99.99% ..... 1616 ms

It is easy to observe a substantially higher throughput.

Note that end-to-end latencies are very high just because the sender can write into the stream at a higher rate than RabbitMQ being able to dispatch messages to the consumer ("receiver" in quiver terms).

AMQP 1.0 in 3.13​

# quiver //host.docker.internal//amq/queue/my-stream \
    --durable --count 1m --duration 10m --body-size 12 --credit 5000

---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
-----------------------------------------------------  -----------------------------------------------------  --------
     2.1        196,350      98,077       12     70.1       2.1          4,094       2,045        0      7.7       195
     4.1        392,956      98,205       13    138.5       4.1          4,094           0        0      7.7         0
     6.1        524,026      65,470       10    196.5       6.1          4,094           0        0      7.7         0
     8.1        655,096      65,470       11    259.4       8.1          4,094           0        0      7.7         0
    10.1        786,166      65,470       10    307.5      10.1          4,094           0        0      7.7         0
receiver timed out
    12.1        917,236      65,470        9    355.5      12.1          4,094           0        0      7.7         0

RabbitMQ 3.13 cannot handle this workload and the benchmark fails.

AMQP 0.9.1 in 4.0​

$ java -jar target/perf-test.jar \
    --predeclared --exchange amq.default \
    --routing-key my-stream --queue my-stream \
    --flag persistent --flag mandatory \
    --pmessages 1000000 --size 12 --confirm 192 --qos 5000 --multi-ack-every 5000

id: test-104223-225, sending rate avg: 88912 msg/s
id: test-104223-225, receiving rate avg: 88912 msg/s
id: test-104223-225, consumer latency min/median/75th/95th/99th 701/1340/1523/2500/4524 µs
id: test-104223-225, confirm latency min/median/75th/95th/99th 788/1983/2130/2437/2970 µs

Since streams store messages in AMQP 1.0 format, this workload requires RabbitMQ to translate each message between AMQP 0.9.1 and AMQP 1.0. This explains why stream throughput is lower when using AMQP 0.9.1 clients compared to AMQP 1.0 clients.

Summary​

Many Connections​

This section compares memory usage of connecting 40,000 clients with two AMQP 1.0 sessions / AMQP 0.9.1 channels per connection.

make run-broker \
    TEST_TMPDIR="$HOME/scratch/rabbit/test" \
    RABBITMQ_CONFIG_FILE="$HOME/scratch/rabbit/rabbitmq.conf" \
    PLUGINS="rabbitmq_amqp1_0" \
    RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 3000000 +S 6" \
    ERL_MAX_PORTS=3000000 

tcp_listen_options.sndbuf = 2048
tcp_listen_options.recbuf = 2048
vm_memory_high_watermark.relative = 0.95
vm_memory_high_watermark_paging_ratio = 0.95
loopback_users = none

package main

import (
	"context"
	"log"
	"time"

	"github.com/Azure/go-amqp"
)

func main() {
	for i := 0; i < 40_000; i++ {
		if i%1000 == 0 {
			log.Printf("opened %d connections", i)
		}
		conn, err := amqp.Dial(
			context.TODO(),
			"amqp://localhost",
			&amqp.ConnOptions{SASLType: amqp.SASLTypeAnonymous()})
		if err != nil {
			log.Fatal("open connection:", err)
		}
		_, err = conn.NewSession(context.TODO(), nil)
		if err != nil {
			log.Fatal("begin session:", err)
		}
		_, err = conn.NewSession(context.TODO(), nil)
		if err != nil {
			log.Fatal("begin session:", err)
		}
	}
	log.Println("opened all connections")
	time.Sleep(5 * time.Hour)
}

package main

import (
	"log"
	"time"

	amqp "github.com/rabbitmq/amqp091-go"
)

func main() {
	for i := 0; i < 40_000; i++ {
		if i%1000 == 0 {
			log.Printf("opened %d connections", i)
		}
		conn, err := amqp.Dial("amqp://guest:guest@localhost")
		if err != nil {
			log.Fatal("open connection:", err)
		}
		_, err = conn.Channel()
		if err != nil {
			log.Fatal("open channel:", err)
		}
		_, err = conn.Channel()
		if err != nil {
			log.Fatal("open channel:", err)
		}
	}
	log.Println("opened all connections")
	time.Sleep(5 * time.Hour)
}

The examples below directly invoke erlang:memory/0 on the node, a function that returns the memory size in bytes for each memory type.

rabbitmq-diagnostics memory_breakdown

# pipes the output to `jq` for more readable formatting
rabbitmq-diagnostics memory_breakdown --formatter=json | jq

AMQP 1.0 in 4.0​

Here are the runtime-reported memory footprint numbers:

1> erlang:memory().
[{total,5330809208},
 {processes,4788022888},
 {processes_used,4787945960},
 {system,542786320},
 {atom,999681},
 {atom_used,974364},
 {binary,194810368},
 {code,19328950},
 {ets,94161808}]

2> erlang:system_info(process_count).
360312

AMQP 1.0 in 3.13​

To compare, the runtime-reported memory footprint numbers in this test are:

1> erlang:memory().
[{total,12066294144},
 {processes,11156497904},
 {processes_used,11156461208},
 {system,909796240},
 {atom,1089809},
 {atom_used,1062780},
 {binary,192784464},
 {code,22068126},
 {ets,318872128}]

2> erlang:system_info(process_count).
1480318

We observe that the memory usage of processes in RabbitMQ 3.13 is 11.1 GB compared to only 4.8 GB in RabbitMQ 4.0 (a reduction of about 56%). As explained in the previous blog post, the RabbitMQ 3.13 implementation of AMQP 1.0 is resource heavy because each AMQP 1.0 session in the plugin includes an AMQP 0.9.1 client and maintains AMQP 0.9.1 state.

AMQP 0.9.1 in 4.0​

1> erlang:memory().
[{total,5409763512},
 {processes,4716150248},
 {processes_used,4715945080},
 {system,693613264},
 {atom,991489},
 {atom_used,962578},
 {binary,187229040},
 {code,19118766},
 {ets,235605424}]

2> erlang:system_info(process_count).
600314

Summary​

Conclusion​

This blog post demonstrated that the new native AMQP 1.0 implementation in RabbitMQ 4.0 performs multiple times better than AMQP 1.0 in RabbitMQ 3.13.

We also observed that AMQP 1.0 can perform better than AMQP 0.9.1. However, it’s challenging to provide a fair comparison. This blog post used an AMQP 1.0 client written in C and an AMQP 0.9.1 client written in Java. Therefore, we do not claim or promise that you will observe better throughput with your AMQP 1.0 workloads. The AMQP 0.9.1 implementation in RabbitMQ performs well since it has been stable and optimized for over 15 years.

Use cases where AMQP 1.0 will likely outperform AMQP 0.9.1 include:

• Sending to or receiving from a stream because a stream encodes messages in AMQP 1.0 format (as covered in this blog post).
• Leveraging queue locality using the RabbitMQ AMQP 1.0 Java client. (This feature will be covered separately.)
• Publishing to or consuming from other queues when one target queue reaches its limits on the same connection (as covered in the AMQP 1.0 flow control blog post).

1. On August 18, 2024, Team RabbitMQ's PackageCloud account will be discontinued
2. Cloudsmith mirror repositories now use *.rabbitmq.com domains, please update your repository definition files

Mirrors Now Use *.rabbitmq.com Domains​

The docs were updated to use *.rabbitmq.com for mirror subdomains, please update your repository files. The previously used domain names won't go away, however, this migration would make your setup more future-proof for large infrastructure changes our team expects to happen in the rest of 2024.

See Debian and RPM installation guides.

PackageCloud Will Be Discontinued on Aug 18th, 2024​

PackageCloud, a great package distribution service we've been using since 2016 IIRC, will be discontinued on Aug 18, 2024. This means that our PackageCloud repositories will become unavailable shortly after.

Please move to the mirrors used in Debian and RPM installation guides as soon as possible!

On behalf of the RabbitMQ Core Team I would like to thank the PackageCloud team for providing us with a solid service over the last eight years, during which we only have experienced one minor outage.

This migration has nothing to do with the quality of the service but rather reflects the fact that the core team is in a different corporate environment today compared to 2016-2023, plus we now have our own mirrors for backup.

• Modernized RabbitMQ to natively support the latest AMQP standard
• Support for more AMQP 1.0 features
• Significantly better AMQP 1.0 performance and scalability compared to RabbitMQ 3.13
• Greater interoperability with other AMQP 1.0 message brokers
• AMQP 1.0 is enabled by default in RabbitMQ 4.0

AMQP History​

The Advanced Message Queuing Protocol (AMQP) is an application layer protocol and a standard for business messaging.

AMQP was first developed by John O’Hara at JPMorgan in 2003. In his article Toward a Commodity Enterprise Middleware from 2007, John O’Hara describes the original motivation behind AMQP.

The financial services industry required a high performance messaging middleware (also known as service bus). This middleware must provide durability to avoid message loss and allow for store-and-forward and publish-subscribe techniques. Use cases include automated trading and buffering of market data events whose arrival rates temporarily exceed the transactional processing rate performed by backend services.

Some commercial third-party middleware providers were overly expensive. At one point, every major investment bank on Wall Street was building its own custom messaging middleware solution. However, they encountered the same problems repeatedly. Banks were not software companies, and messaging middleware is complex software that is hard to get right.

Banks have managed to collaborate in creating open technical standards such as:

• Society for Worldwide Interbank Financial Telecommunication (SWIFT).
• Financial Information eXchange (FIX) protocol
• FAST protocol (FIX Adapted for STreaming)
• Financial products Markup Language (FpML)

Therefore, the same organizations should also be capable of creating an open standard for business messaging: That's how AMQP was born.

The requirements for the AMQP specification were:

• Open standard
• Free of patents to enable anyone to implement a compatible implementation
• Appealing for commercial implementations (otherwise, the AMQP specification would not succeed)
• Should have more than one implementation (to qualify as a standard after all)
• AMQP software had to be proven in live systems. Middleware is a critical piece of any system and must be trusted. This trust has to be earned.
• Collective effort by many independent companies solving the same problem of connecting systems together

Fast forward to today, AMQP is used in critical messaging infrastructure by most of the largest corporations in the world across all industries, and RabbitMQ is the most popular open-source AMQP message broker.

AMQP Versions​

The following table shows when different AMQP specifications were published:

In his article from 2007, John O'Hara writes:

The AMQP Working Group is rapidly evolving the protocol and hopes to reach version 1.0 during 2008

Not only did the release of version 1.0 take three years longer than anticipated, but the protocol specification also changed completely. The result is that AMQP 1.0 is a very different protocol compared to its pre-release versions such as AMQP 0.9.1.

When "AMQP" is mentioned in conversations or documentation, the version is often omitted, creating a lot of confusion about which protocol is meant. Since RabbitMQ is the most popular message broker, many websites refer to AMQP 0.9.1. Other documentation, in turn, refers to AMQP 1.0.

The rest of this blog post omits AMQP versions 0.8, 0.9, and 0.10 since they are old specifications that aren't used anymore. Instead, we focus on the differences between AMQP versions 0.9.1 and 1.0, both of which are heavily used in today's message brokers.

On a very high level, the biggest difference between AMQP 0.9.1 and AMQP 1.0 is the following:

• AMQP 0.9.1 defines the protocol between client and server as well as server entities such as exchanges, queues, and bindings.
• AMQP 1.0 defines only the protocol between client and server.

AMQP 0.9.1​

A client publishes a message to an exchange, which routes the message to a queue. The queue stores the message until it gets consumed by a client.

More precisely, a publisher sends a message to an instance of an exchange type. The exchange type defines the routing algorithm. The AMQP 0.9.1 specification defines different types of exchanges which must be implemented by an AMQP 0.9.1 broker:

• Direct exchange
• Fanout exchange
• Topic exchange
• Headers exchange

Exchanges also offer an extension point. RabbitMQ provides various other exchange types, and third-party plugins can also easily be added to RabbitMQ.

Depending on the binding (including the binding key) from a queue to an exchange instance and the routing key (and possibly other headers) of the message, the exchange type algorithm decides to which queue(s) the message gets copied.

The model in AMQP 0.9.1 is therefore simple, yet flexible and powerful. If needed, complex routing topologies can be created, including exchange to exchange bindings.

Exchanges, queues, and bindings can be either pre-declared by an administrator or dynamically created via HTTP or, more commonly, directly by the client applications via AMQP 0.9.1. Specifically, AMQP 0.9.1 defines protocol frames such as exchange.declare, queue.declare, and queue.bind. Therefore, AMQP 0.9.1 client applications are aware of the server topology.

The RabbitMQ Summit 2023 talk Serving millions of clients with Native MQTT and MQTT 5.0 explained that the AMQ 0.9.1 model is so powerful that it provides much of what an MQTT broker needs. RabbitMQ uses the AMQ 0.9.1 topic exchange type to match topics in incoming MQTT packets against topic filters of MQTT subscriptions.

AMQP 1.0​

AMQP 1.0 is more generic than AMQP 0.9.1. AMQP 1.0 does not define a server model. In fact, AMQP 1.0 is so generic that it does not even mandate a central broker to be present. AMQP 1.0 defines peers exchanging messages with each other. They do so by sending messages to a target and by consuming messages from a source.

Both targets and sources contain an address. What internal object an AMQP 1.0 address refers to and how an address is resolved is not defined by the AMQP 1.0 specification because different AMQP 1.0 brokers have different models to receive, store, and send messages.

For example, in an AMQP 1.0 broker that implements the AMQP 0.9.1 model (such as RabbitMQ), an AMQP 1.0 target address refers to an AMQP 0.9.1 exchange, and an AMQP 1.0 source address refers to an AMQP 0.9.1 queue. Other AMQP 1.0 brokers might choose that an address refers to an immutable log (a stream), a topic, some in-memory data structure, or a SQL database. Yet another AMQP 1.0 implementation could be just a client library sending messages directly to another client without any central broker being involved where an address could refer to some operating system socket. It's up to the AMQP 1.0 implementation which decides how it resolves an address and how it stores messages.

Because the AMQP 1.0 specification is so generic, it's easier for a wide range of brokers to add support for AMQP 1.0 independent of the model they use to store and forward messages. That's why more AMQP 1.0 brokers exist compared to AMQP 0.9.1 brokers.

AMQP 0.9.1 vs. AMQP 1.0​

The following table compares AMQP 0.9.1 with AMQP 1.0:

The Pervasiveness of AMQP 0.9.1​

Naively thinking, one might expect that the newer AMQP 1.0 protocol version is better, more feature-rich, and more performant, and therefore would obsolete and replace older AMQP 0.9.1 implementations. However, this is not the case. In fact, new AMQP 0.9.1 client libraries are still being created, such as the C++ library open-sourced by Bloomberg just last year. Surprisingly, even entirely new AMQP 0.9.1 broker implementations, such as LavinMQ, have been developed from scratch years after AMQP 1.0 was released.

The reasons that AMQP 0.9.1 is so popular compared to AMQP 1.0 are RabbitMQ's strong ecosystem and community, and the fact that AMQP 0.9.1 is less complex. Simplicity usually wins because simplicity leads to developers and users understanding the technology, which in turn results in more client libraries being developed and higher adoption.

As stated previously, AMQP 0.9.1 client applications are aware of exchanges, queues, and bindings. John O’Hara even argued in his article from 2007 that interoperability cannot be achieved without the AMQP 0.9.1 protocol defining the model (exchanges, queues, and bindings):

AMQP [0.9.1] is split into two main areas: transport model and queuing model. AMQP [0.9.1] is unusual in that it thoroughly specifies the semantics of the services it provides within the queuing model; since applications have a very intimate relationship with their middleware, this needs to be well defined or interoperability cannot be achieved.

Seventeen years later, we can conclude that he was right.

Given that in 2024 AMQP 0.9.1 is still more pervasive compared to AMQP 1.0, has AMQP 1.0 failed? As shown in the table above, many brokers have become AMQP 1.0 compliant. Although no single broker implements the entire AMQP 1.0 specification due to its complexity, any AMQP 1.0 client can exchange messages with any of these AMQP 1.0 compliant brokers. Therefore, arguably, we can conclude that both AMQP 0.9.1 and AMQP 1.0 have succeeded.

Why does RabbitMQ 4.0 get an AMQP 1.0 upgrade?​

There are several reasons why we have invested in first-class AMQP 1.0 support:

1. Customer Demand: Both commercial customers and open-source users have requested better AMQP 1.0 support. While not the primary reason for our investment, sometimes conversations occur at such a high, non-technical level that prospective customers prefer to base their central messaging infrastructure on the latest AMQP standard rather than a pre-release AMQP specification. An example can be found here.
2. Protocol Features: The AMQP 1.0 specification has a better-defined type system and more protocol features, as partially shown in the table above. This blog post provides only an overview of AMQP 0.9.1 vs. AMQP 1.0, with subsequent posts covering AMQP 1.0 features in greater detail.
3. Extensibility: AMQP 1.0 is designed for extensibility. More thought has been put into the protocol specification, and we believe we can leverage this extensibility in future RabbitMQ versions. For example, thanks to this extensibility, RabbitMQ can inform AMQP 1.0 clients about queue locality, allowing clients to publish and consume locally to the RabbitMQ node they are connected to.
4. Improved Performance: The old RabbitMQ 3.13 plugin was limited in features, suboptimally designed, and slow due to its internal proxying via the AMQP 0.9.1 protocol. Numerous AMQP 1.0 implementation bugs have been fixed in RabbitMQ 4.0.
5. Interoperability: An open source message broker such as RabbitMQ is all about interoperability. AMQP messages should be able to traverse different brokers and organizations. Since many brokers implement AMQP 1.0, users can more easily migrate to (and from) RabbitMQ. Another interoperability use case was demonstrated by Alvaro Videla in his RabbitMQ Summit 2023 keynote, where messages were moved from RabbitMQ to Real-Time Intelligence in Microsoft Fabric to analyze application messages using various Azure cloud services.
6. Multi-Protocol Message Broker: RabbitMQ is the best multi-protocol message broker on the market. It supports AMQP 0.9.1, MQTT, Streams, and, since RabbitMQ 4.0, AMQP 1.0 natively. STOMP is supported via a plugin. RabbitMQ offers users full flexibility in how messages are published and consumed: Messages can be published to and consumed from the same queue using different protocols. RabbitMQ excels by performing well-documented message protocol conversions. Natively supporting AMQP 1.0 provides even greater flexibility to users.

AMQP 1.0 in RabbitMQ 3.13​

Since AMQP 1.0 was released in 2011, RabbitMQ has provided limited support for AMQP 1.0 via a plugin:

The AMQP 1.0 plugin up to RabbitMQ 3.13 proxies AMQP 1.0 internally via AMQP 0.9.1. In other words, the plugin translates each AMQP 1.0 message received from the client to an AMQP 0.9.1 message and then sends this translated message via the AMQP 0.9.1 protocol to RabbitMQ core. This approach has major drawbacks:

1. Limited Feature Support: Support for AMQP 1.0 is limited to the subset of features provided by the AMQP 0.9.1 protocol.
2. Slow Performance: Each message must be translated between AMQP 0.9.1 and AMQP 1.0.
3. Resource Heavy: This approach is resource-intensive in terms of memory and CPU usage, as each AMQP 1.0 session in the plugin includes an AMQP 0.9.1 client and maintains AMQP 0.9.1 state.

As explained in the Serving Millions of Clients with Native MQTT blog post, the MQTT plugin worked conceptually the same way up to RabbitMQ 3.11: it also proxied internally via the AMQP 0.9.1 protocol.

AMQP 1.0 in RabbitMQ 4.0​

With RabbitMQ 4.0, native AMQP 1.0 support replaces the AMQP 1.0 plugin:

AMQP 1.0 clients send messages directly to exchanges (AMQP 1.0 targets) and receive messages directly from queues (AMQP 1.0 sources). In other words, native AMQP 1.0 no longer proxies via the AMQP 0.9.1 protocol but continues to utilize the simple, flexible, and powerful AMQ 0.9.1 model. This transition brings several benefits:

1. Enhanced Feature Support: More AMQP 1.0 features are supported. This mirrors how the introduction of Native MQTT in RabbitMQ 3.12 facilitated the addition of more protocol features, such as MQTT 5.0 in RabbitMQ 3.13.
2. Fast Performance: Native AMQP 1.0 offers better throughput and lower latency.
3. Resource Light: Lower memory and CPU usage on the broker, with a single Erlang process used per AMQP 1.0 session compared to 15 Erlang processes in RabbitMQ 3.13.

Native AMQP 1.0 thus follows the success story of Native MQTT.

Obsoleting the AMQP 1.0 plugin, which seems straightforward in the diagram, actually required 12 months of engineering work. This extensive effort was due to several factors:

• Complexity of the AMQP 1.0 Protocol: AMQP 1.0 is the most complex protocol supported by RabbitMQ, more intricate than both AMQP 0.9.1 and MQTT.
• Queue modifications: AMQP 1.0 clients can directly consume from each queue type, necessitating changes to classic queues, quorum queues, and stream implementations.
• Rolling Upgrades: We support rolling upgrades from RabbitMQ 3.13 to 4.0, allowing you to upgrade your AMQP 1.0 workloads without downtime.

Even though RabbitMQ 4.0 supports AMQP 1.0 natively, this does not imply that RabbitMQ supports all AMQP 1.0 features. Like any other AMQP 1.0 broker, RabbitMQ's AMQP 1.0 implementation has documented limitations.

RabbitMQ AMQP 1.0 clients​

We have developed two new AMQP 1.0 client libraries specifically for RabbitMQ:

• RabbitMQ AMQP 1.0 Java client
• RabbitMQ AMQP 1.0 .NET client

These client libraries have the following characteristics:

• Thin Wrappers: They are thin wrappers containing RabbitMQ-specific logic built around existing open-source AMQP 1.0 client libraries.
• Opinionated API: They offer a simple and safe API designed to help application developers easily get started with sending and receiving messages via AMQP 1.0.
• RabbitMQ Model Integration: They enable applications to manage RabbitMQ’s AMQ 0.9.1 model (such as declaring and deleting exchanges, queues, and bindings) through AMQP 1.0. This feature leverages the HTTP over AMQP 1.0 extension specification.
• Best of Both Worlds: Applications can maintain an "intimate relationship with their middleware" by dynamically creating server topologies while enjoying a straightforward experience with the otherwise complex AMQP 1.0 protocol. Additionally, applications can optionally bypass the thin RabbitMQ-specific wrapper and interact directly with the underlying AMQP 1.0 client library to access more advanced AMQP 1.0 features.
• Reduced Network Traffic: In some scenarios, they can reduce intra-RabbitMQ cluster traffic by identifying the RabbitMQ nodes on which a given queue is located, allowing applications to publish and consume "locally" from these nodes.
• Optional: These client libraries are not required for interacting with RabbitMQ via AMQP 1.0. Any other AMQP 1.0 client should also be able to communicate with RabbitMQ.

RabbitMQ Modernizations​

RabbitMQ 4.0 completes its modernization journey, which began with RabbitMQ 3.8 in 2019. These modernizations include:

• Quorum Queues: Using the Raft consensus algorithm to provide a safe, replicated queue type resilient to network partitions.
• Prometheus Metrics and Grafana Dashboards: Offering first-class observability integrations and alerts.
• RabbitMQ on Kubernetes: Operators to run RabbitMQ anywhere.
• Streams: An immutable log providing message replay and high message throughput with the stream protocol.
• Classic Queue Storage Version 2: Providing higher message throughput and lower memory usage.
• Commercial Disaster Recovery Solution: Ensuring business continuity when the entire data center goes down.
• Native MQTT and MQTT 5.0: Transforming RabbitMQ into an IoT broker.
• Khepri: Using the Raft consensus algorithm to provide a safe and scalable metadata store resilient to network partitions.
• Native AMQP 1.0: Upgrading RabbitMQ's core protocol to the latest AMQP standard (as covered in this blog post).

We encourage you to try and test AMQP 1.0 in RabbitMQ 4.0 and provide feedback. You can use the latest rabbitmq:4.0 docker image. While we encourage you to build new applications with AMQP 1.0, we will continue to support and maintain AMQP 0.9.1 in RabbitMQ 4.x.

Watch our RabbitMQ Summit 2024 talk on Native AMQP 1.0 and read more technical blog posts on AMQP 1.0 features and performance benchmarks.

What is Community Support?​

Community support is defined as all questions, root cause analysis requests, issue reports, and other interactions the RabbitMQ core team has with open source RabbitMQ users on GitHub and our community forums.

Community Support Policy Changes​

Effective immediately, a new community support policy is adopted for Open Source distributions of RabbitMQ. The policy seeks to ensure that RabbitMQ remains a sustainable open source project.

Under the new policy, only the following groups of users will receive support directly from the RabbitMQ core team:

• Users with VMware Tanzu RabbitMQ commercial licenses
• Active RabbitMQ community contributors. A definition of a “Contribution" is provided in the community support eligibility document
• Users and community members who report certain types of RabbitMQ issues and provide all the details necessary to reproduce the reported issue

Release Policy Changes​

Effective June 1st 2024, the RabbitMQ team will adopt a new release policy for the Open Source distribution. The changes do not impact the current RabbitMQ Open Source License. The RabbitMQ core team will continue developing Open Source RabbitMQ under the Mozilla Public License 2.0 on GitHub.

With the new policy, older Open Source release versions (for example, RabbitMQ 3.12.x and older) will no longer receive patches through community support. Patches for older versions of both the Open Source and Commercial versions will be available to customers that have VMware Tanzu RabbitMQ commercial licenses.

In practical terms this means that once RabbitMQ 4.x is released, all new contributions will eventually ship as part of

• Open Source RabbitMQ 4.x
• VMware Tanzu RabbitMQ 4.x

Conclusion​

Users who contribute to the RabbitMQ open source distribution, and/or stay current with the latest RabbitMQ release will still have the ability to collaborate with the RabbitMQ team.

All other users seeking assistance with the RabbitMQ Open Source distribution will be required to obtain commercial licenses for RabbitMQ support or seek RabbitMQ-related help elsewhere.

Our team has discovered significant performance regressions on Erlang 27, as high as 30% lower message throughput in many common workloads.

We are investigating the root cause of the regressions. Please do not use Erlang/OTP 27 with RabbitMQ at this time.

We will announce support for Erlang/OTP 27 when we are confident that it works well with RabbitMQ.

Read dedicated blog posts for more details about these changes:

• support for version 5 of the MQTT protocol
• support for stream filtering
• performance improvements

RabbitMQ 3.13 is the final minor release in the 3.x series. The next release will be 4.0!

Experimental Support For Khepri (Mnesia Replacement)​

Apart from the new features mentioned in the first paragraph, RabbitMQ 3.13 includes experimental support for Khepri. Khepri is a new storage backend for RabbitMQ metadata that is designed to replace Mnesia. It is not yet ready for production use but we encourage users to try it out in test environments and provide feedback.

Our plan is to completely remove Mnesia in the future. This should significantly improve RabbitMQ's tolerance to network partitions. Once we switch to Khepri, there will be no partition handling strategy configuration (pause_minority, autoheal, etc) — Khepri is based on the Raft protocol, just like quorum queues and therefore the semantics of what to do when a partition occurs are well defined and not configurable.

A recorded talk about Khepri is available.

rabbitmqctl enable_feature_flag khepri_db

You shouldn't really notice any difference after enabling Khepri. The main difference is what happens internally when you declare exchanges, queues, bindings and so on. We encourage experimentation such as declaring your actual topology first and then enabling Khepri (to validate that everything works as expected), inducing failures to validate the cluster remains available (as long as the majority of the nodes is up and connected) and so on. Please report any issues you run into.

Feature Flags​

RabbitMQ 3.13.0 includes a few new feature flags. It doesn't however, set any older flags as required (apart from those that were already required in 3.12 of course). Therefore, if you have some feature flags disabled, upgrading from 3.12 to 3.13 will still work. In the 3.11 -> 3.12 upgrade, some users ran into issues if not all feature flags were enabled. Such issues won't happen when moving from 3.12 to 3.13.

Classic Queues: Version 1 Remains The Default​

We had intended to change the default version of classic queues to v2 in 3.13 but ultimately decided against it. Therefore, v1 is still the default and v2 remains an opt-in feature. However, classic queues v2 are highly recommended! You can upgrade your queues by setting x-queue-version=2 in a policy. To make sure new queues are created as v2 by default, you can set

classic_queue.default_version = 2

in rabbitmq.conf.

The reason v1 remains the default has nothing to do with any v2 shortcomings but rather with the fact that changing the node default led to some back and forth migrations between v1 and v2 in certain scenarios. In particular, a mirrored queue would be upgraded and downgraded back and forth between v1 and v2 during a rolling upgrade, since the default would be different on different nodes. To avoid any risk of such scenarios, we decided against this change.

Classic queues v2 will become the only option in the future. By then, queue mirroring will be removed and therefore there will be no risk of mirroring-related issues.

Message Containers​

Message Containers are a mostly invisible change in how messages are handled internally. RabbitMQ was originally built as an AMQP 0-9-1 broker. However, over the years, support for AMQP 1.0, MQTT, STOMP and Streams was added. This led to some internal message format conversions since different protocols have mostly similar concepts, but differ in the details such as available data types.

Message containers are based on a message format from AMQP 1.0 and modernize internal message representation with today's multi-protocol assumptions and makes all the conversions between protocols explicit.

These conversions are now documented.

That's A Wrap For 3.x!​

RabbitMQ 3.0.0 was released in November 2012. For various historical reasons, the major version has not been incremented since then. However, it's time to say goodbye to the 3.x series and move on to 4.0 later this year. Version 4.0 will include a number of breaking changes but most importantly, it will no longer support mirroring of classic queues. Policy keys related to mirroring will be ignored and queues will become single-node queues. This is a final call for users requiring highly available queues: migrate to quorum queues, or streams if applicable, as soon as possible. You will enjoy much higher data safety, reliability and better performance than mirrored queues ever offered.

• support for version 5 of the MQTT protocol
• support for stream filtering

This post focuses on the changes to the classic queues in this release:

• classic queue storage format version 1 is deprecated
• new implementation of the classic queue message store

Classic Queues Storage Primer​

Before we get into the changes, it's worth explaining briefly how classic queues store messages. For each message, we need to store the message payload and some metadata about the message (eg. whether this message was delivered to a consumer). It makes sense to separate the message data (an opaque binary blob, potentially of a significant size) from the metadata (small key-value map). However, for small messages, performing two separate writes, one for metadata and another one for the contents of the message, is wasteful. Therefore, classic queues handle small messages differently than large messages.

Historically, in what we now refer to as version 1 of the classic queues, this process is called embedding the message in the index and the property queue_index_embed_msgs_below controls what constitutes a small enough message to be embedded (the default is 4kB). Messages above this threshold are stored separately in the message store - a separate key-value structure with a different on-disk representation. For messages stored in the store, the index contains the metadata and the message store ID, which allows to retrieve the payload when needed. There is one message store per virtual host, while each queue has a separate index.

Version 2 of the classic queue storage, introduced in 3.10, is logically very similar: there's still the same per-vhost message store and a separate per-queue message store for metadata and small messages. However, the structure of what we store per-queue is completely different and therefore we don't call it just an index anymore - small messages are not embedded in the index, but stored in separate files within the per-queue message store.

The per-vhost message store is still there for larger messages, but version 3.13 changes its behaviour significantly.

For backwards compatibility queue_index_embed_msgs_below still controls whether a message is large enough to be stored in the per-vhost message store and the default is still 4kB.

Classic Queues Version 2 (CQv2)​

A few years ago, we started a journey to re-implement classic queues for better performance. A lot of things have changed since the original implementation, which is now almost 20 years old! Here's a rundown of the steps on this journey:

1. Since 3.10, queues with queue-version=2 use the new index storage format (we store per-queue data differently)
2. Since 3.12, classic queues (both v1 and v2) never store more than a small set of messages in memory
3. Since 3.12, messages below queue_index_embed_msgs_below (4kb by default) are handled much more efficiently

With 3.13, we are getting close to the end of this journey:

1. Starting from 3.13, messages above queue_index_embed_msgs_below are handled in a more efficient way
2. Starting from 3.13, classic queues v1 are deprecated

In RabbitMQ 4.0 we will remove the mirroring feature of classic queues. As we've said many times before, if you need highly available replicated queues, you should be using quorum queues that have been available since 3.8. Removing the mirroring feature will enable further optimisations to the implementation.

Moreover, in 4.0 we'll most likely remove the v1 implementation of the queue index (this may be delayed based on your feedback!). When you upgrade to 4.0 in the future, all the classic queues that are still using v1 will need to be converted to v2 during startup. This can take a long time if there are many messages and/or many queues. It's therefore a good idea to go through the conversion process deliberately.

What If I Don't Feel Ready for CQv2?​

Until version 1 of the index implementation is removed, you can still use it.

There's no such choice for the message store implementation - 3.13 contains significant improvements, especially when used in combination with a v2 index. However, there may also be minor regressions when combined with v1. Users are recommended to test their applications thoroughly and report situations where a v2 index is worse than v1.

CQv1 -> CQv2 Conversion​

Since v1 and v2 use a different file format, a conversion is needed if a queue is changed from v1 to v2 (or vice versa - downgrading is supported). If you have an existing classic queue v1 and apply a policy with x-queue-version=2, this queue will become unavailable for the duration of the conversion - the queue needs to have a moment to rewrite the files into the new format. Such conversions shouldn't take more than a few seconds - if you see it taking longer, please report this.

Since the queue version can be changed through a policy, it's also possible to gradually migrate from v1 to v2. You can declare a policy that only matches a subset of the queues and once they are converted, you can either extend the regular expression to match more queues or declare another policy matching a different subset of the queues. Even if the policy matches a lot of queues, the migration is strictly a per-queue operation - any queue that completed the conversion becomes available to serve client applications immediately after the conversion, even if other queues are still rewriting their files.

You can go through this conversion already on 3.12 (or even 3.10 or 3.11). If you do, the removal of v1 in 4.0 won't really affect you, since all your queues will be v2 already.

Performance Comparison​

Let's see the results comparing RabbitMQ 3.12.11 with 3.13.0-rc.4. Please refer to previous blog posts for details about the benchmarking setup and the way we run these tests or check out the repository, where we keep the environments configuration and the script with the workloads.

All tests were performed with 100B and 5kB messages.

Publishing And Consuming​

In this test, we have a single publisher and a single consumer and just try to deliver messages as quickly as possible through a single queue.

As you can see, classic queues v2 offer a significantly better performance compared with CQv1 and 3.13 improves the performance of both versions. For the users still using CQv1, moving to CQv2 on 3.13 may almost double the throughput for small messages!

Publish Only​

In this test, we publish to a queue at full speed, with 2 publishers, while not consuming the messages at all. The queue grows from empty to 5 million messages.

With 100B messages, CQv2 blows CQv1 out of the water with more than 250% higher throughput compared to 3.12.

The 5kB test is more nuanced. 3.13 with CQv2 wins by a large margin and CQv2 superiority is visible even in 3.12. However, the combination of the new message store with the old index doesn't perform consistently - it has a good throughput most of the time, but with significant slow downs (latency spikes). This is unfortunate but something we decided to keep this way, given the number of factors required to trigger this behaviour and the fact that users should move to CQv2 anyway. We saw this behaviour only in this test, so the following conditions need to be met: a 3.13 node running a CQv1 queue with messages larger than 4kb (or whatever the value of queue_index_embed_msgs_below is), publishers significantly faster than consumers (or consumers not present at all) and a high throughput of messages. If you have such a workload, moving to CQv2 should not only prevent this regression but give you a significantly better performance than CQv1 could ever achieve.

Consume Only​

In this test we consume the long backlog of messages from the previous test. There are 5 million messages to consume (hopefully your queues are much shorter!).

With 100B messages, you can see that CQv2 offers ~30% higher consumption rate early on. Over time, as the queue becomes shorter, CQv1 gets faster but CQv2 environment still empties the queue long before CQv1 (when consumption rate goes down to zero, that means the queue is empty):

With 5kB messages, handled by the per-vhost message store, you can see the main benefit of the 3.13 changes. Since the message store implementation is shared by v1 and v2, in this test both 3.13 environments are significantly ahead of 3.12, even a 3.12 with CQv2. We can see a ~50% higher throughput in 3.13:

Multiple Queues​

In this test, rather than pushing a single queue to its limit, we have 5 concurrent message flows: 5 publishers, each publishing to a different queue, and 5 consumers, one for each queue. Each publisher sends 10000 messages per second so the total expected throughput is 50000 messages. With 100B messages, all environments reached the expected throughput, while with 5kB messages, all oscillated around 27000 messages/s. The more interesting part here is the end-to-end latency - how long does it take from the moment a message is sent until the message is consumed.

For 100B messages, we can see that the CQv2 environments can deliver the messages much faster. For users moving from CQv1 on 3.12 to CQv2 on 3.13, that a 75% reduction of the mean latency and a 50% of reduction in memory usage.

For 5kB messages, the results are much closer and in fact, 3.12 wins this particular test (something we may look into in the future). However, 3.13 can still achieve similar results while using 100MB less memory:

Publisher Confirm Latency​

Lastly, let's take a look at a very different test. Rather than flooding queue(s) with messages, we only publish one message at a time, wait for the publisher confirm and then publish the next one (a consumer is present but not really relevant here, since it can easily consume the incoming messages).

With 100B messages, we can once again see how much faster CQv2s are, with well over 200% speed up compared to CQv1 on 3.12:

With 5kB messages, the benefits of the new per-vhost message store implementation in 3.13 are apparent, with more than 350% improvement:

You may notice that the throughput is actually a bit higher with 5kB messages than it is with 100B messages. This is counter-intuitive but actually not that strange. 5kB is still a tiny amount of data to send over the wire, while the message store is designed to handle such messages better. As with everything else, this difference may change in the future with further optimisations and perhaps a change to the default value of queue_index_embed_msgs_below that we will consider.

Caveat​

Both version 2 of the index and the new message store implementation should provide significant benefits for most users. However, these implementations already focus on non-mirrored use cases and the new message store implementation was designed with the v2 of the index in mind. While it is backwards compatible and can be used with classic queues v1, there are some cases where using v1 of the index with the new message store may provide worse performance or a different performance profile than in the past, as seen in the publish-only test. It is therefore highly recommended to:

1. Test and benchmark your application with 3.13
2. Compare performance with v1 and v2
3. If v1 behaves better without mirroring, please report that, so we can take a look

Remaining on v1 can be a workaround for the cases where it provides better performance. However, that old implementation will be removed in the future, so you can't rely on that workaround for long. Please report such cases so that you can upgrade to v2 in the future.

Final Word​

Redesigning and replacing such a core component of a widely used piece of software is a very hard task. Special thanks to Loïc Hoguin for embarking on this project, scavenging through the code, sometimes dating all the way back to the very first release of RabbitMQ, before the world heard about iPhones. As always, we welcome testing and feedback and we hope the upgrade will provide you similar benefits as shown above.

In this blog post, I will go over the reasons we are doing this and what improvements it will bring to you.

Limitations of the current website​

Only the latest RabbitMQ version is documented​

As of this writing, the www.rabbitmq.com website contains the documentation of the RabbitMQ server, some AMQP clients and it offers tutorials to get started with RabbitMQ.

The documentation covers the latest version of RabbitMQ only ‑ currently 3.12.x. That was ok-ish so far because new releases series usually come with new additions. However, with the upcoming RabbitMQ 4.0.x, significant deprecated features will be removed! If we continue like this, it means the documentation for these dropped features will go away. This is a real problem for people who will still be running RabbitMQ 3.12.x for some time.

This is the first and main reason we want to upgrade the website: we want the documentation to cover many versions of RabbitMQ: old, current and upcoming.

We could take a snapshot of the website and publish it somewhere else, like previous.rabbitmq.com in the past. But the Git repository/workflow and the home-grown framework don't make this easy, that's why we stopped keeping snapshots of older versions. We would make a better job if the workflow and the tooling had room for maintaining several versions easily and all of them would be published to www.rabbitmq.com in a way that it is easy for users to switch to whatever version they are running.

This brings me to the second reason behind the big upgrade.

In-house framework​

The website is statically generated and published to a Cloudflare worker. The static generator we used was developed by the RabbitMQ team a decade ago and barely evolved ever since.

Making or maintaining a website framework is definitely not the best way to invest our time. There is still plenty of work on RabbitMQ itself :-) So instead of adding multiple versions support to it, we looked at existing FOSS static website generators that had this feature built-in and evaluated a few of them.

We chose Docusaurus. It supports multiple versions ‑ our primary criteria ‑ and it uses Markdown as the markup language, thus we don't have to rewrite anything. This would make the conversion easier.

“What's in it for me?”​

With Docusaurus, we, the RabbitMQ team, make our life easier, yay \o/ But “are there any benefits for you”, you may ask?

Multiple versions documented​

Of course, you will be able to read documentation of past and new versions of RabbitMQ, even the future one while it is being worked on.

Initially, it will only cover RabbitMQ 3.13.x because we didn't convert previous versions to Docusaurus. A snapshot of RabbitMQ 3.12.x documentation will be published on previous.rabbitmq.com instead.

We plan to keep all supported RabbitMQ versions docs on the new website. When an old release series reaches end-of-life, its documentation will also move to previous.rabbitmq.com. This is to avoid that the version dropdown becomes too cluttered or that the search feature brings too many results which are less relevant.

Speaking of search...

Integrated search feature​

Docusaurus integrates several solutions to provide an internal search feature. We picked Algolia because it is very efficient and works well with multiple versions.

The entire website, including the blog, will be indexed and searchable. In the example above, I searched "MQTT" and Algolia returns blog posts and documentation pages in the results.

What is not obvious on the screenshot however is that it only returned docs from the current version I'm browsing! So if I selected an hypothetical RabbitMQ 5.2.x in the dropdown version menu, I would only get results relevant to that version.

Integrated blog​

As mentionned above, Docusaurus provides a blog. We will switch to it and stop using blog.rabbitmq.com. This way, all RabbitMQ-related content will be in a single place:

• It will be easier for you to browse and move between blog posts and docs.
• As said, the internal search will consider both docs and blog posts.
• For content writers, it will be easier to cross-reference between blog posts and docs.

Mobile version and dark mode​

Docusaurus being slightly more modern than our in-house framework, the new website should be browsable comfortably on a small screen and on mobile devices.

It also comes with a dark mode!

Conclusion​

We are very excited with this move to Docusaurus and everything that the new website will bring to you! Beside the conversion to a new framework, we also plan many improvements to the content itself! Things like... No, let's talk about this in a future blog post on the new website once we have something to show :-)

Meanwhile, we would love your feedback! We know there are some rough edges after the conversion, some parts we couldn't map easily to Docusaurus alternatives. So, do you find the new website comfortable? Anything you lose that you love on the old one?

Here are a couple links to help you with this:

• The new website
• The pull request to share comments and feedback

Thank you so much for any constructive positive or negative comments you might share!

Concepts​

The idea of stream filtering is to provide a first level of efficient filtering on the broker side, without the broker needing to interpret messages. This way consumers that need only a subset of the stream do not need to get all the data and handle all the filtering by themselves. This can drastically reduce the data transmitted to consumers.

With filtering a filter value can be associated to each message. It can be geographical information, like the region of the world each message comes from, as illustrated in the next figure:

So our stream has 1 AMER (green) message, 1 APAC (dark blue) message, 2 EMEA (purple) messages, then 2 AMER messages.

Message Publishing​

Publishers get to associate each outbound message with its filter value:

In the figure above, the publisher publishes 1 AMER (green) message and 2 EMEA (purple) messages that will be added to the stream.

Message Consuming​

When a consumer subscribes, it can specify one or several filter values and the broker is expected to dispatch only the messages with this or these filter value(s). We will see soon this is a bit different in practice but this is enough to understand the concepts.

In the figure below, the consumer at the top specified it only wants AMER (green) messages and the broker dispatches only those. Same thing for the consumer in the middle with EMEA messages and the consumer at the bottom with APAC messages.

That's it for the concepts, let's discover the implementation details now.

Structure of a Stream​

We need to know how a stream is structured to understand stream filtering internals. A stream is a directory with segment files in it. Each segment file has an associated index file (used to know where to attach a consumer at a given offset in the segment file, among others). Having several "small" segment files is better than having a large monolithic file for the whole stream: it is for example more efficient and safer to delete "old" segment files to truncate the stream than removing the beginning of a large file.

A segment file is made of chunks that contain messages. The number of messages in a chunk depends on the ingress rate (high ingress rate means many messages in a chunk, low ingress rate means fewer messages in a chunk). The number of messages in a chunk varies from a few (even 1) to a few thousands.

What is the deal with chunks? Chunks are the unit of work in streams: they are used for replication and, more importantly for our topic, for consumer delivery. The broker sends chunks to consumers, one at a time, using the sendfile system call (to send a whole chunk from the file system to the network socket, without copying the data into userspace).

The following figure illustrates the structure of a stream:

With this in place, let's see how the broker can know whether to dispatch a chunk or not.

Filtering On The Broker Side​

Imagine we have a consumer that wants only AMER (green) messages. When a broker is about to dispatch a chunk, it needs to know whether the chunk contains AMER messages or not. If it does, it can send the chunk to the consumer, if it does not, the broker can skip the chunk, move on to the next one, and re-iterate.

Each chunk has a header that can contain a Bloom filter, which tells the broker whether the chunk contains a message with a given filter value. A Bloom filter is a space-efficient probabilistic data structure, used to test whether an element is a member of a set. In our example the set contains AMER, EMEA, and APAC and the element is AMER.

The following figure illustrates the broker-side filtering process for our 3 chunks:

As shown in the figure above, the filter can return false positives, that is chunks that do not contain messages with the expected filter value(s). This is normal, as Bloom filters are probabilistic. They do not return false negative though: if the filter says there is no AMER (green) messages, we can be sure it is true. We have to live with this uncertainty: we may dispatch some chunks for nothing sometimes, but it is better than dispatching all the chunks.

Something certain is that a consumer can receive messages it does not want: look at our first chunk on the left, it contains AMER (green) messages that the consumer asked for, but also EMEA (purple), and APAC (dark blue) messages. This is why there must be filtering on the client side as well.

Filtering On The Client Side​

The broker handles a first level of filtering when delivering messages, but as the unit of delivery is the chunk, the consumer can still receive messages it does not want. So the client must do some filtering as well, which obviously must be consistent with the filter value(s) set at subscription time.

The following figure illustrates a consumer that wants only AMER (green) messages and that must do a last step of filtering:

Let's see how this translates into application code.

API Examples​

Filtering is not intrusive and can be handled as a cross-cutting concern, minimizing the impact on application code. Here is how to set the logic to extract the filter value from a message when declaring a producer with the stream Java client (filterValue(Function<Message,String>) method):

Producer producer = environment.producerBuilder()
  .stream("invoices")
  .filterValue(msg -> msg.getApplicationProperties().get("region").toString())  
  .build();

On the consuming side, the stream Java client provides the filter().values(String... filterValues) method to set the filter value(s) and the filter().postFilter(Predicate<Message> filter) method to set the client-side filtering logic. Both methods must be called when declaring the consumer:

Consumer consumer = environment.consumerBuilder()
  .stream("invoices")
  .filter()
    .values("AMER")  
    .postFilter(msg -> "AMER".equals(msg.getApplicationProperties().get("region")))  
  .builder()
  .messageHandler((ctx, msg) -> {
    // message processing code
  })
  .build();

As you can see, filtering does not change the publishing and consuming code, just the declaration of producers and consumers.

Let's see now how to configure stream filtering in the most appropriate way for a use case.

Stream Filtering Configuration​

The first post on stream filtering provided some numbers (about 80% bandwidth savings with filtering compared to no filtering). Benefits of stream filtering depend heavily on the use case: ingress rate, cardinality and distribution of filter values, but also filter size. The larger the filter, the better (error rate gets smaller). It is possible to set a value between 16 and 255 bytes for the size of the filter used in chunks, the default being 16 bytes.

The stream Java client provides the filterSize(int) method to set the filter size when creating a stream (it sets the stream-filter-size-bytes parameter internally):

environment.streamCreator()
  .stream("invoices")
  .filterSize(32)
  .create()

How to estimate the size of the filter? There are many Bloom filter calculators available online. The parameters are the number of hash functions (2 for RabbitMQ stream filtering), the number of expected elements, the error rate, and the size. You usually have an idea of the number of elements, so you need to find a trade-off between the error rate and the filter size.

Here are some examples:

• 10 values, 16 bytes => 2 % error rate
• 30 values, 16 bytes => 14 % error rate
• 200 values, 128 bytes => 10 % error rate

So, the larger the filter, the better? Not exactly: even though a Bloom filter is very efficient in terms of storage, as it does not store elements, just whether elements are in the set, the filter size is pre-allocated. If you set the filter size to 255 and each chunk contains at least one message with a filter value, there will be 255 bytes allocated in each chunk header. This is fine if chunks contain many large messages, as the filter size is negligible compared to the chunk size. But with a degenerated case like a single-message chunk with a 10-byte message and a 10-byte filter value, you end up with a filter larger than the actual data.

You'll have to experiment with your own use cases to estimate the impact of the filter size on your stream size. The first post on stream filtering provides a trick to estimate the size of a stream with Stream PerfTest (read the whole stream without filtering and consult the rabbitmq_stream_read_bytes_total metric).

Bonus: Stream Filtering On AMQP​

Even though the preferred way to access streams is the stream protocol, other protocols are supported, like AMQP. Stream filtering is also supported with any AMQP client library:

• Declaration: set the x-queue-type argument to stream and use the x-stream-filter-size-bytes to set the filter size when you declare a stream.
• Publishing: use the x-stream-filter-value header to set the filter value for outbound messages.
• Consuming: use the x-stream-filter consumer argument to set the expected filter value(s) (string or array of strings) and optionally the x-stream-match-unfiltered consumer argument to receive messages without any filter value as well (default is false). Client-side filtering is still necessary!

Wrapping Up​

This blog post provided an in-depth description of stream filtering in RabbitMQ 3.13. It complements a first post that gives an introduction on the usage and a demonstration of stream filtering.

Stream filtering is easy to use and to benefit from, but some knowledge on the internals can be useful to optimize its usage, especially for tricky use cases. Remember that client-side filtering is necessary and must be consistent with the configured filter value(s). This is usually straightforward to implement. It is also possible to set the filter size in the most appropriate way for a given use case.

Keep reading to find out how stream filtering works and see it in action.

Concepts of Stream Filtering​

Imagine you have a stream containing data from all around the world and an application that needs to process only a subset of this data, let's say messages for a given region. The application can read the whole stream and filter out the data to only process the messages it is interested in. This works but it means the whole stream content will go through the network.

Stream filtering provides a first level of efficient filtering on the broker side, without the broker needing to interpret messages. It can dramatically reduce the amount of data exchanged on the network for some use cases. Let's discover the semantics of this new exciting feature.

On The Publishing Side​

Stream filtering is based on a filter value: a publishing application can associate each message with a string value. Filter values can be anything, but they should meet some criteria for filtering to work properly. A defined set of values shared across the messages is a good candidate: geographical locations (e.g. countries, states), document types in a stream that stores document information (e.g. payslip, invoice, order), categories of products (e.g. book, luggage, toy).

How a message is associated to a filter value depends on the client library. Here is an example with the stream Java client:

Producer producer = environment.producerBuilder()
  .stream("invoices")
  .filterValue(msg -> msg.getApplicationProperties().get("region").toString())  
  .build();

In this example the application developer provides some logic to extract the filter value from the message application properties. Using filtering is as simple as this: no need to change the actual message publishing code, you just need to provide the filter value logic when creating a Producer.

Let's see now how it works for consumers.

On The Consumer Side​

Here is a Java code snippet to declare a consumer that is only interested in messages from the emea region:

Consumer consumer = environment.consumerBuilder()
  .stream("invoices")
  .filter()
    .values("emea")  
    .postFilter(msg -> "emea".equals(msg.getApplicationProperties().get("region")))  
  .builder()
  .messageHandler((ctx, msg) -> {
    // message processing code
  })
  .build();

Filtering is configured in two places:

• filter().values(String... filterValues) tells the broker we are interested in messages associated with these values (we can specify several values, not only one)
• filter().postFilter(Predicate<Message> filter) provides some client-side logic to filter out messages that would not be associated with the expected filter value(s)

Why the need for this client-side filtering logic? The broker-side filtering logic uses a Bloom filter:

A Bloom filter is a space-efficient probabilistic data structure, used to test whether an element is a member of a set.

A Bloom filter is very efficient in terms of storage and speed, but it is probabilistic: it can return false positives. Because of this, the broker can send messages it believes match the expected filter values whereas they do not. That's why some client-side filtering logic is necessary.

This is something to be aware of, but that is a minor caveat compared to the benefits stream filtering brings.

A subsequent blog post covers the internals of stream filtering for those interested in technical details. You can also have a look at the stream Java client documentation on filtering for more information. It covers among others that a message does not always have to be associated to a filter value and a consumer can choose to receive messages with given filter value(s) and messages without a filter value (with filter().matchUnfiltered()).

Trying It Out​

Let's see stream filtering in action. Start a RabbitMQ 3.13+ node:

docker run -it --rm --name rabbitmq -p 5552:5552 \
    -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-rabbitmq_stream advertised_host localhost' \
    rabbitmq:3.13

Enable the stream plugin:

docker exec rabbitmq rabbitmq-plugins enable rabbitmq_stream

Download Stream PerfTest (it requires Java 11 or more to run):

cd /tmp
wget -O stream-perf-test.jar \
  https://github.com/rabbitmq/rabbitmq-java-tools-binaries-dev/releases/download/v-stream-perf-test-latest/stream-perf-test-latest.jar

Let's publish messages for 10 seconds:

java -jar stream-perf-test.jar --producers 1 --consumers 0 --rate 100 --filter-value-set 1..50 --size 10000 --time 10

The messages are 10 KB long and --filter-value-set 1..50 means a random filter value between "1" and "50" is associated with each message.

Let's consume all the messages (without any filtering):

java -jar stream-perf-test.jar --producers 0 --consumers 1 --offset first --prometheus

The output should stop after a few seconds, when the consumer reaches the end of the stream. Do not stop the application, open another terminal tab instead, and query Stream PerfTest metrics to see how much data it read:

curl --silent localhost:8080/metrics | grep rabbitmq_stream_read_bytes_total

You should get something like the following:

# HELP rabbitmq_stream_read_bytes_total
# TYPE rabbitmq_stream_read_bytes_total counter
rabbitmq_stream_read_bytes_total 1.0046894E7

This is about 10 MB. The client had to transfer the entire stream.

Now stop Stream PerfTest (Ctrl-C) and start it again, this time with filtering enabled:

java -jar stream-perf-test.jar --producers 0 --consumers 1 --offset first --prometheus --filter-values 5

Here we ask to get only messages with the "5" filter value (--filter-values 5). Again, wait for the output to stop and check the number of bytes read:

curl --silent localhost:8080/metrics | grep rabbitmq_stream_read_bytes_total

You should get something like:

# HELP rabbitmq_stream_read_bytes_total
# TYPE rabbitmq_stream_read_bytes_total counter
rabbitmq_stream_read_bytes_total 1957641.0

This is less than 2 MB. It is 8 MB of bandwidth saved, about 80%, not bad!

Of course this is somewhat artificial: Stream PerfTest is not a real application and it is unlikely it distributes messages and filter values the way real applications do. But still, it gives an idea of what the bandwidth savings can be with stream filtering.

Wrapping Up​

We had a quick overview of stream filtering in RabbitMQ 3.13. It allows to save bandwidth when messages are dispatched from the broker to consuming applications. Not all use cases can benefit from stream filtering, but the benefits in terms of bandwidth for those that can are quite compelling.

Stay tuned for a subsequent blog post that will cover the internal details of stream filtering to help you use it and configure it in the most optimal way.

RabbitMQ 3.13 will support MQTT 5.0 and will therefore be the next big step in our journey to make RabbitMQ one of the leading MQTT brokers.

This blog post explains how the new MQTT 5.0 features are used in RabbitMQ.

MQTT Overview​

MQTT is the standard protocol for the Internet of Things (IoT).

IoT remote devices can have a poor network quality when connecting to the broker. Therefore, MQTT is lightweight: MQTT protocol headers are small to save network bandwidth.

Since IoT devices might often disconnect and reconnect - imagine a car driving through a tunnel - MQTT is also efficient: Clients are connected and authenticated with a handshake that is shorter compared to other messaging protocols.

The MQTT protocol has been around for many years. As shown in the following table, the latest MQTT protocol version is 5.0.

It is worth mentioning that there is a difference between the user facing protocol version and the "internal" protocol version (also known as the protocol level). The latter is sent from client to server in the CONNECT packet. Because user facing protocol version 3.1.1 maps to internal protocol version 4, to avoid further confusion, the MQTT committee decided to skip user facing version 4.0 such that user facing version 5.0 maps to internal protocol version 5.

MQTT 5.0 Features​

Appendix C. Summary of new features in MQTT v5.0 provides a complete list of new MQTT 5.0 features.

Since you find great MQTT 5.0 resources including illustrative diagrams and usage patterns on the web, this blog post focuses solely on RabbitMQ specifics. This section explains the most significant features implemented in PR #7263. For each feature, we provide an example how it is used with RabbitMQ or outline a high level description of how it is implemented in RabbitMQ.

To run the examples by yourself, start RabbitMQ server 3.13, for example, using this Docker image tag:

docker run -it --rm --name rabbitmq -p 1883:1883 -p 15672:15672 -p 15692:15692 rabbitmq:3.13.0-management

In another terminal window, enable the MQTT plugin:

docker exec rabbitmq rabbitmq-plugins enable rabbitmq_mqtt

Since the MQTT plugin got enabled dynamically, feature flags defined by the MQTT plugin are disabled. Enable all feature flags including feature flag mqtt_v5:

docker exec rabbitmq rabbitmqctl enable_feature_flag all

Listing the feature flags should now show that all feature flags are enabled:

docker exec rabbitmq rabbitmqctl list_feature_flags --formatter=pretty_table

Below examples use MQTTX CLI version 1.9.4. We use a CLI rather than a graphical UI so that you can easily run the examples by copy pasting the commands.

All new features also apply to the RabbitMQ Web MQTT Plugin.

Here is the list of MQTT 5.0 features covered in this blog post:

• Feature 1: Message Expiry
• Feature 2: Subscription Identifier
• Feature 3: Subscription Options
• Feature 4: Reason code on all ACKs
• Feature 5: User properties
• Feature 6: Payload Format and Content Type
• Feature 7: Request / Response
• Feature 8: Assigned Client Identifier
• Feature 9: Topic Alias
• Feature 10: Flow control
• Feature 11: Maximum Packet Size
• Feature 12: Server initiated DISCONNECT
• Feature 13: Session Expiry
• Feature 14: Will delay
• Feature 15: Optional Server feature availability

Feature 1: Message Expiry​

Description​

An expiry interval in seconds can be set for each message being published to the broker. If the message is not consumed within that expiry interval, the message is discarded or dead lettered.

Example​

Create a subscription for topic t/1. This creates a queue in RabbitMQ. Disconnect the client by typing Ctrl+C in the terminal. Since we use a Session Expiry Interval of 600 seconds, this queue will exist for 10 more minutes.

mqttx sub --client-id sub-1 --topic t/1 --session-expiry-interval 600 --qos 1
…  Connecting...
✔  Connected
…  Subscribing to t/1...
✔  Subscribed to t/1
^C

Publish a message to the same topic with a message expiry interval of 30 seconds:

mqttx pub --topic t/1 --message m1 --message-expiry-interval 30 --qos 1
…  Connecting...
✔  Connected
…  Message publishing...
✔  Message published

Within the next 30 seconds, list the queues:

docker exec rabbitmq rabbitmqctl --quiet --formatter=pretty_table list_queues name type messages
┌─────────────────────────────┬─────────┬──────────┐
│ name                        │ type    │ messages │
├─────────────────────────────┼─────────┼──────────┤
│ mqtt-subscription-sub-1qos1 │ classic │ 1        │
└─────────────────────────────┴─────────┴──────────┘

Wait for 30 seconds, and list the queues again:

docker exec rabbitmq rabbitmqctl --quiet --formatter=pretty_table list_queues
┌─────────────────────────────┬─────────┬──────────┐
│ name                        │ type    │ messages │
├─────────────────────────────┼─────────┼──────────┤
│ mqtt-subscription-sub-1qos1 │ classic │ 0        │
└─────────────────────────────┴─────────┴──────────┘

The message expired since client sub-1 has not connected to the broker to consume the message. If a dead lettering policy is set up, the message will be dead lettered to an exchange. In our case, dead lettering is disabled. Querying the Prometheus endpoint proves that 1 message expired from a classic queue.

curl --silent localhost:15692/metrics | grep rabbitmq_global_messages_dead_lettered_expired_total
# TYPE rabbitmq_global_messages_dead_lettered_expired_total counter
# HELP rabbitmq_global_messages_dead_lettered_expired_total Total number of messages dead-lettered due to message TTL exceeded
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_classic_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_classic_queue",dead_letter_strategy="disabled"} 1
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_least_once"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="at_most_once"} 0
rabbitmq_global_messages_dead_lettered_expired_total{queue_type="rabbit_quorum_queue",dead_letter_strategy="disabled"} 0

Another interesting feature is the following requirement:

The PUBLISH packet sent to a Client by the Server MUST contain a Message Expiry Interval set to the received value minus the time that the Application Message has been waiting in the Server.

Send a second message to the broker with a Message Expiry Interval of 60 seconds:

mqttx pub --topic t/1 --message m2 --message-expiry-interval 60 --qos 1

Wait for 20 seconds before reconnecting the subscribing client:

mqttx sub --client-id sub-1 --topic t/1 --no-clean --session-expiry-interval 0  --qos 1 --output-mode clean
{
  "topic": "t/1",
  "payload": "m2",
  "packet": {
    ...
    "properties": {
      "messageExpiryInterval": 40
    }
  }
}

As mandated by the MQTT 5.0 protocol specification, the client receives the second message with a Message Expiry Interval set to 40 seconds: 60 seconds as received by the broker minus the 20 seconds the message has been waiting in the broker.

Implementation​

MQTT 5.0 Message Expiry is implemented in RabbitMQ using per-message TTL similar to the expiration field in AMQP 0.9.1 publishers.

Feature 2: Subscription Identifier​

Description​

A client can set a subscription identifier in a SUBSCRIBE packet. If the client is being sent a message due to that subscription, the broker will include that subscription identifier into the PUBLISH packet.

Use cases for subscription identifiers are listed in section SUBSCRIBE Actions.

Example​

Send 3 separate SUBSCRIBE packets from the same client to the server, each with a different topic filter and different subscription identifier:

mqttx sub --client-id sub-2 --topic t/1 --subscription-identifier 1 --session-expiry-interval 600
^C
mqttx sub --client-id sub-2 --topic t/2 --subscription-identifier 2 --session-expiry-interval 600 --no-clean
^C
mqttx sub --client-id sub-2 --topic "t/#" --subscription-identifier 3 --session-expiry-interval 0 --no-clean --output-mode clean

In a 2nd terminal window, we see 3 bindings from the same queue to the same topic exchange, each with a different routing key:

docker exec rabbitmq rabbitmqctl --quiet --formatter=pretty_table list_bindings \
    source_name source_kind destination_name destination_kind routing_key
┌─────────────┬─────────────┬─────────────────────────────┬──────────────────┬─────────────────────────────┐
│ source_name │ source_kind │ destination_name            │ destination_kind │ routing_key                 │
├─────────────┼─────────────┼─────────────────────────────┼──────────────────┼─────────────────────────────┤
│             │ exchange    │ mqtt-subscription-sub-2qos0 │ queue            │ mqtt-subscription-sub-2qos0 │
├─────────────┼─────────────┼─────────────────────────────┼──────────────────┼─────────────────────────────┤
│ amq.topic   │ exchange    │ mqtt-subscription-sub-2qos0 │ queue            │ t.#                         │
├─────────────┼─────────────┼─────────────────────────────┼──────────────────┼─────────────────────────────┤
│ amq.topic   │ exchange    │ mqtt-subscription-sub-2qos0 │ queue            │ t.1                         │
├─────────────┼─────────────┼─────────────────────────────┼──────────────────┼─────────────────────────────┤
│ amq.topic   │ exchange    │ mqtt-subscription-sub-2qos0 │ queue            │ t.2                         │
└─────────────┴─────────────┴─────────────────────────────┴──────────────────┴─────────────────────────────┘

The first entry is the implicit binding to the default exchange.

Each MQTT subscription with its MQTT topic filter corresponds to one AMQP 0.9.1 binding with a binding key. To be precise, table column routing_key is misnamed: It should be called binding_key instead. The topic level separator in MQTT is the "/" character whereas the topic level separator in AMQP 0.9.1 topic exchanges is the "." character.

Again in the 2nd terminal window, send a message to topic t/1:

mqttx pub --topic t/1 --message m1

The 1st terminal window (of the subscribing client) receives the following PUBLISH packet:

{
  "topic": "t/1",
  "payload": "m1",
  "packet": {
    ...
    "properties": {
      "subscriptionIdentifier": [
        1,
        3
      ]
    }
  }
}

It contains subscription identifiers 1 and 3 because both topic filters t/1 and t/# match topic t/1.

Likewise, if you send a 2nd message to topic t/2, the subscribing client will receive a PUBLISH packet containing subscription identifiers 2 and 3.

Implementation​

Subscription identifiers are part of the MQTT session state. Therefore, subscription identifiers must be persisted in the server's database while the client is disconnected and until the MQTT session ends. RabbitMQ stores the subscription identifiers in the binding arguments:

docker exec rabbitmq rabbitmqctl --quiet --formatter=pretty_table list_bindings routing_key arguments
┌─────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────┐
│ routing_key                 │ arguments                                                                         │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ mqtt-subscription-sub-2qos0 │                                                                                   │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ t.#                         │ {mqtt_subscription_opts,0,false,false,0,3}{<<"x-binding-key">>,longstr,<<"t.#">>} │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ t.1                         │ {mqtt_subscription_opts,0,false,false,0,1}{<<"x-binding-key">>,longstr,<<"t.1">>} │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ t.2                         │ {mqtt_subscription_opts,0,false,false,0,2}{<<"x-binding-key">>,longstr,<<"t.2">>} │
└─────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘

The exact structure of the binding arguments is not important and will likely change in a future RabbitMQ version. However, you can see integers 1, 2, and 3 in the binding arguments which correspond to the subscription identifiers.

When the topic exchange routes a message, the publishing Erlang process will include all matched binding keys into the message. The Erlang process of the subscribing MQTT client compares the matched binding keys with the MQTT topic filters it knows about and includes the subscription identifiers into the PUBLISH packet sent to the MQTT client.

The publishing Erlang process can be an MQTT connection process or a AMQP 0.9.1 channel process. As always, RabbitMQ excels in cross protocol interoperability: When an AMQP 0.9.1 (or STOMP or AMQP 1.0) client sends a message to the topic exchange, the correct subscription identifier will be included in the PUBLISH packet sent to the MQTT client.

Feature 3: Subscription Options​

Description​

MQTT 5.0 comes with 3 new subscription options:

1. No Local
2. Retain as Published
3. Retain Handling

All subscription options are implemented by RabbitMQ. Here, we focus only on the Retain Handling option:

This option specifies whether retained messages are sent when the subscription is established.
The values are:
0 = Send retained messages at the time of the subscribe
1 = Send retained messages at subscribe only if the subscription does not currently exist
2 = Do not send retained messages at the time of the subscribe

Example​

Send a retained message:

mqttx pub --topic mytopic --message m --retain

Retain Handling value 0 will receive the retained message while value 2 will not:

mqttx sub --topic mytopic --retain-handling 0
…  Connecting...
✔  Connected
…  Subscribing to mytopic...
✔  Subscribed to mytopic
payload: m
retain: true
^C

mqttx sub --topic mytopic --retain-handling 2
…  Connecting...
✔  Connected
…  Subscribing to mytopic...
✔  Subscribed to mytopic

Feature 4: Reason code on all ACKs​

Description​

Packets CONNACK, PUBACK, SUBACK, UNSUBACK, and DISCONNECT contain a reason code.

Implementation​

One implementation example is that RabbitMQ will reply with a reason code No matching subscribers in the PUBACK packet if the message is not routed to any queues. MQTT 5.0 reason code No matching subscribers corresponds conceptually to the mandatory message property and BasicReturn handler in AMQP 0.9.1.

Feature 5: User properties​

Description​

Most MQTT packets can contain user properties. The meaning of user properties is not defined by the MQTT specification.

Example PUBLISH packet​

User properties in the PUBLISH packet are defined by client applications and forwarded unaltered by the server.

Subscribe in the 1st terminal window:

mqttx sub --topic t/5

Publish a message with user properties in the 2nd terminal window:

mqttx pub --topic t/5 --message m --user-properties "key1: value1"

The 1st terminal window will receive the user properties unaltered:

payload: m
userProperties: [ { key: 'key1', value: 'value1' } ]