Shovel Plugin
Overview
This guide provides an overview of RabbitMQ Shovel, a core RabbitMQ plugin that unidirectionally moves messages from a source to a destination. Two more guides cover the two flavors of Shovels, dynamic shovels and static shovels, respectively. This one focuses on explaining the concept, how do shovels work and what they can do.
Sometimes it is necessary to reliably and continually move messages from a source (typically a queue) in one cluster to a destination (an exchange, topic, etc) in another cluster.
The rabbitmq_shovel plugin allows you to
configure a number of shovels (transfer workers), which do just that
and run as part of a RabbitMQ cluster.
The source and destination can be in the same cluster (typically in different vhosts) or distinct ones. Shovels support AMQP 0.9.1 and AMQP 1.0 sources and destinations. The source and destination do not have to use the same protocol, so it is possible to move messages from an AMQP 1.0 broker to RabbitMQ or vice versa.
A shovel behaves like a well-written client application, which connects to its source and destination, consumes and republishes messages, and uses acknowledgements on both ends to cope with failures.
A Shovel uses Erlang AMQP 0-9-1 and Erlang AMQP 1.0 clients under the hood.
Why Use Shovel
Shovel is a minimalistic yet flexible tool in the distributed messaging toolkit that can accommodate a number of use cases. Below are some of its key features and design goals.
Loose Coupling
A shovel can move messages between brokers (or clusters) in different geographic or administrative domains that
- may have different loosely related purposes
- may run on different versions of RabbitMQ
- may use different messaging products or protocols
- may have different users and virtual hosts
WAN-friendly
The Shovel plugin uses client connections under the hood. Acknowledgements and publisher confirms are used to ensure data safety in case of connection and node failures.
Cross-protocol and Product Message Transfers
Modern Shovel versions support multiple protocols: AMQP 0.9.1 and AMQP 1.0.
This means it is possible to shovel, e.g. from and AMQP 1.0 broker source to a RabbitMQ destination and vice versa. More protocols may be supported in the future.
Flexibility
When a shovel connects (either to the source or the destination) it can be configured to predeclare a certain topology it needs.
There is no requirement to run the shovel on the same broker (or cluster) as its source or destination, although that's the most typical approach; the shovel can run on an entirely separate node or cluster.
A comparison between clustering, federation is provided in the Distributed Messaging guide.
What Does a Shovel Do?
In essence, a shovel is a minimalistic message pump. Each shovel:
- Connects to the source and destination clusters
- Consumes messages from a queue
- Re-publishes to a destination
- Uses data safety features on both ends and handles failures
The shovel configuration allows each of these processes to be tailored.
Connection
After connection to a source or a destination broker a series of configured topology declaration operations can be issued. For example, on an AMQP 0-9-1 endpoint, queues, exchanges and bindings can be declared.
A shovel will attempt to reconnect if a failure occurs and multiple brokers can be specified for the source and destination so that another broker may be selected (at random) to reconnect to. A reconnection delay can be specified to avoid flooding the network with reconnection attempts, or to prevent reconnection on failure altogether.
All configured topology declaration operations for that source or destination are re-issued upon reconnection.
Consumption
The Shovel's consumer will acknowledge messages automatically on receipt, after (re-)publication, or after confirmation of its publication from the destination server.
Re-publishing
Most publishing and message properties are controlled by the operator.
Getting started
The Shovel plugin is included in the RabbitMQ distribution. To enable it, use rabbitmq-plugins:
rabbitmq-plugins enable rabbitmq_shovel
Management UI users may also wish to enable the rabbitmq_shovel_management plugin
for Shovel status monitoring.
There are two distinct ways to define shovels: dynamic shovels are defined using
runtime parameters and static shovels are defined in the advanced.config file.
The pros and cons with each approach are covered below. Most users should consider dynamic shovels first for their ease of reconfiguration and management.
| Static Shovels | Dynamic Shovels |
|---|---|
Defined in the broker advanced configuration file. | Defined using runtime parameters. |
Creation and deletion require a node restart. | Creation and deletion do not require a node restart. Can be created and deleted at any time. |
Less opinionated, less automation-friendly: any queues, exchanges or bindings can be declared manually at startup. | More opinionated, more automation-friendly: the queues, exchanges and bindings used by the shovel will be declared automatically. |
Note that when using AMQP 1.0 the "nodes" may still need to be created outside of the shovel as the protocol does not include topology creation.
Authentication and authorisation for Shovels
The plugin uses Erlang AMQP 0-9-1 and Erlang AMQP 1.0 clients under the hood to open connections to its source and/or destination. Just like any other client library connection, a Shovel connection must successfully authenticate and be authorized to access the virtual host and resources it is trying to use. This is true for both sources and destinations.
Authentication and authorisation failures of shovel connections will be logged by the node that's running the shovel.
Shovel Failure Handling in Clusters
It's normally desirable to ensure that shovels are resilient to failure of any node in the source or destination clusters, or the cluster hosting the shovel.
A shovel can be provided a list of both source and destination endpoints. In this case the shovel will connect to the first reachable endpoint.
Dynamic shovels are automatically defined on all nodes of the hosting cluster on which the shovel plugin is enabled. Each shovel will only start on one arbitrarily chosen node, but will be restarted on another node in case of node failure.
Static shovels should be defined in the configuration file for all nodes of the hosting cluster on which the shovel plugin is enabled. Again each shovel will only start on one node and be restarted on another cluster node when a node failure is detected.
Securing Shovel Connections with TLS
Shovel connections can use TLS. Because Shovel uses client libraries under the hood, it is necessary to both configure the source broker to listen for TLS connections and the Shovel to use TLS when connecting.
To configure Shovel to use TLS, one needs to
- Specify CA certificate and client certificate/key pair, as well as other parameters (e.g. peer verification depth) via URI query parameters
- Configure Erlang client to use TLS
Just like with "regular" client connections, server's CA should be trusted on the node where Shovel runs, and vice versa. The same TLS troubleshooting methodology that is recommended for application connections applies to shovels.
Monitoring Shovels
There are two ways of discovering the status of shovels.
Using Management UI
Shovel status can be reported on the Management plugin user interface
in the administrative section.
This requires the rabbitmq_shovel_management plugin to be enabled
on the node used to access management UI.
Using CLI Tools
Shovel status can be obtained by direct query of the Shovel plugin app using rabbitmqctl:
# use the -n switch to target a remote node
rabbitmqctl shovel_status
The result will return a list of statuses, one per Shovel running.
Each element of the list is a map with several properties:
- Name
- Type
- Status
- Last state change timestamp
| Property | Description |
| Name | Name of the shovel |
| Type | static for static shovels, dynamic for dynamic ones |
| Status | Current state of the shovel |
| Timestamp | Time when the shovel has last entered this state (e.g. successfully connected, lost connection, ran into an exception) |
Timestamp will return a local calendar time of the form
of {{YYYY, MM, DD}, {HH, MM, SS}}.
Key states of a shovel are
| State | Description |
| starting | Shovel is starting or trying to connect to its configured endpoints |
| running | Shovel has successfully connected and running (consuming from the source and republishing to the destination). This state will report some basic endpoint and protocol information. |
| terminated | Shovel has stopped or ran into an exception. A reason will be provided. |