RabbitMQ Methodology¶

Not many companies give much thought to the structure they create within RabbitMQ. Typically, the focus is on ensuring that each application receives its data, without considering the flow of data in each queue. As a result, queues are often named based on the system and its function, such as whether it's an "input," "retry," or "dead-letter" queue. For example, in my case, for my CBL (Common Business Logic) application, I might create queues like cbl.queue.input, cbl.queue.retry, and cbl.queue.dlq. All the data would initially flow into the input queue, and from there, the application would determine which process handles it.

At first glance, this approach seems inefficient. The biggest issue is that it prevents parallel processing of messages, as each message is stacked behind the previous one, leading to potential bottlenecks. The lack of concurrent processing means that even if multiple entities could be handled in parallel, the application struggles to scale.

For these reasons, I prefer a different method of defining queues in RabbitMQ, one that is more efficient and allows for greater flexibility in processing. By designing queues and message flows with more thoughtful granularity and consideration for parallelization, we can greatly improve the performance and scalability of our system. Below, I will describe the approach I take and how it can optimize message handling in a RabbitMQ-based system.

Naming Conventions¶

Let's take a look at the singular naming conventions

Exchange¶

Exchange names should be descriptive of the type of data they handle. Typically, I use domain names such as customer or coupon. However, in most cases, it’s better to provide subdomain-like names, such as customer.purchase, customer.profile, or coupon.event. This naming convention makes it easier to identify the type of data being processed and adds a level of structure that enhances both scalability and maintainability. By organizing exchanges with clear, meaningful names, you can quickly determine the purpose and contents of each exchange without needing to dig into its configuration or contents.

Queue¶

Queue names should be based on the exchange name followed by the operation being performed. For example, if the exchange is named coupon, the queue could be named coupon.change or coupon.assign to reflect the specific operation related to coupons. This convention helps clearly define the purpose of each queue and what type of message it will process.

In cases where there are multiple consumers for the same type of data, it's a good practice to append the application name to the queue name. For instance, if you have multiple applications consuming data related to coupons, you might create queues like coupon.change.app1 or coupon.assign.app2, where app1 and app2 represent the respective consumer applications. This ensures that each consumer has its own dedicated queue while still reflecting the core data and operation being processed.

By using this approach, you maintain clarity and flexibility in managing queues, particularly as the number of consumers or operations increases.

Routing Key¶

Routing keys should be based on the operation being performed. For example, if the operation is to change a coupon, the routing key could be change. Similarly, for assigning a coupon, the routing key might be assign. This approach ensures that messages are routed to the correct queues based on the specific action or operation required.

By defining routing keys in this way, it becomes easier to maintain a logical flow of messages through the system, as each operation has its own dedicated routing key. It also helps in ensuring that messages are processed by the correct consumers and systems, providing greater control over message routing and reducing the risk of errors.

In cases where there are multiple operations related to the same data (e.g., change and assign), this strategy helps keep the system organized and easily extensible, as new operations can be added simply by defining new routing keys.

Topology example¶

flowchart LR
    IS.PosPurchase -- transaction --> customer.purchase
    IS.LoyaltyProfile -- change --> loyalty.profile
    CBL.CuponService -- urgent --> coupon.event
    CBL.CuponService -- non - urgent --> coupon.event
    DWH -- change --> customer.segmentation
    ECHO --> product-catalogue.product
    ECHO --> product-catalogue.price
    ECHO --> product-catalogue.category
    ECHO --> product-catalogue.stock

    subgraph RabbitMQ
        customer.purchase -- transaction --> custpomer.purchase.transaction.cdp
        coupon.event -- urgent --> coupon.event.urgent.cdp
        coupon.event -- non - urgent --> coupon.event.non-urgent.cdp
        customer.segmentation -- change --> customer.segmentation.cdp
        loyalty.profile -- change --> loyalty.profile.cdp
        product-catalogue.product --> product-catalogue.product.ssp
        product-catalogue.product --> product-catalogue.product.wms
        product-catalogue.price --> product-catalogue.price.ssp
        product-catalogue.category --> product-catalogue.category.ssp
        product-catalogue.stock --> product-catalogue.stock.ssp
        subgraph direct
            customer.purchase
            coupon.event
            customer.segmentation
            loyalty.profile
        end
        subgraph fanout
            product-catalogue.product
            product-catalogue.price
            product-catalogue.category
            product-catalogue.stock
        end
    end

Types of Exchanges¶

When designing a RabbitMQ-based system, it’s essential to understand the different types of exchanges available and how they can be used to optimize message routing and processing. By selecting the appropriate exchange type for each use case, you can ensure that messages are delivered efficiently and reliably to the correct consumers.

Direct Exchange¶

A direct exchange routes messages to queues based on an exact match between the routing key and the queue binding key. This means that messages are delivered to queues where the routing key exactly matches the queue binding key. Direct exchanges are ideal for scenarios where messages need to be delivered to a single consumer based on specific criteria, such as the type of data or operation being performed.

Fanout Exchange¶

A fanout exchange routes messages to all queues that are bound to it, regardless of the routing key. This means that messages are broadcast to all consumers bound to the exchange, ensuring each consumer receives a copy of the message. Fanout exchanges are ideal for scenarios where the same message should be processed by multiple consumers, such as broadcasting updates or notifications to multiple systems or applications.

Topic Exchange¶

A topic exchange routes messages to queues based on pattern matching between the routing key and the queue binding key. This allows for more flexible routing, enabling you to send messages to queues based on topics or categories. Topic exchanges are ideal when messages need to be delivered to multiple consumers based on specific criteria, such as filtering messages by topic, region, or action type.

By understanding the differences between these exchange types and how they can be used to route messages effectively, you can design a more efficient and scalable RabbitMQ-based system that meets the needs of your application.

Types of Queues¶

When designing a RabbitMQ-based system, it's essential to understand the different types of queues available and how they can be used to optimize message processing and delivery. By selecting the appropriate queue type for each use case, you can ensure that messages are handled efficiently and reliably by the correct consumers.

Quorum Queue¶

A quorum queue is a highly available and durable queue designed to ensure that messages are not lost during failures. Quorum queues replicate messages across multiple nodes in a RabbitMQ cluster, offering fault tolerance and reliability. Quorum queues are ideal for use cases where message durability and availability are critical, such as processing financial transactions or critical system updates.

Classic Queue¶

A classic queue is a simple and lightweight queue that provides basic message handling capabilities. Classic queues are ideal for situations where message processing is straightforward, and advanced features or durability guarantees are not required. They are well-suited for scenarios where message loss is acceptable or where messages are processed quickly and do not need to be stored long-term.

Stream Queue¶

A stream queue is a persistent and append-only queue that stores messages in a log-like structure. Stream queues are ideal for scenarios where messages need to be stored and processed in the order they were received, such as event streams or logs. Stream queues ensure that messages are processed in the correct order and are not lost during failures, providing durability and reliability for ordered message processing.

By understanding the differences between these queue types and how they can be used to optimize message processing and delivery, you can design a more efficient and reliable RabbitMQ-based system that meets the needs of your application.

Queue Additional Properties¶

Dead Letter Exchange¶

A dead-letter exchange (DLX) is an exchange to which messages are sent when they cannot be delivered to their intended queue. This could happen for various reasons, such as the queue being full, the message being rejected, or the message expiring. By defining a dead-letter exchange, you ensure that messages are not lost but routed to a designated exchange for further processing or handling.

Message TTL (Time-to-Live)¶

The message TTL (time-to-live) property defines how long a message can remain in a queue before it expires. By setting a message TTL, you ensure that messages are processed within a specific timeframe, preventing old or stale messages from accumulating in the queue. This is useful in scenarios where timely processing is essential and helps avoid potential bottlenecks caused by outdated messages.

Delivery Limit¶

The delivery limit property defines the maximum number of times a message can be delivered to a consumer before it is considered undeliverable. By setting a delivery limit, you can prevent messages from being repeatedly delivered to a consumer that is unable to process them. This reduces the risk of message processing errors and ensures that resources are not wasted on undeliverable messages.

Manage RabbitMQ with Terraform¶

Managing RabbitMQ with Terraform allows you to define and automate the provisioning, configuration, and scaling of your RabbitMQ infrastructure. With Terraform, you can ensure consistency and reproducibility across environments, making it an ideal choice for maintaining RabbitMQ deployments in both cloud and on-premises setups.

One important thing to note is that the RabbitMQ provider for Terraform is not officially maintained by Broadcom, but it is a community-supported provider that works well in practice. You can find the documentation for this provider here.

This approach is particularly suitable for on-premises solutions, as Kubernetes operators for RabbitMQ already provide similar functionality for managing RabbitMQ clusters in containerized environments.

Example Terraform Configuration¶

resource "rabbitmq_exchange" "coupon-event" {
  name  = "coupon.event"
  vhost = "/"
  settings {
    type        = "direct"
    durable     = true
    auto_delete = false
  }
}

resource "rabbitmq_queue" "coupon-event-non-urgent-cdp" {
  name  = "coupon.event.non-urgent.cdp"
  vhost = "/"

  settings {
    durable     = true
    auto_delete = false
    arguments = {
      "x-queue-type" = "quorum"
    }
  }
}

resource "rabbitmq_binding" "coupon-event-non-urgent" {
  source           = "${rabbitmq_exchange.coupon-event.name}"
  vhost            = "/"
  destination      = "${rabbitmq_queue.coupon-event-non-urgent-cdp.name}"
  destination_type = "queue"
  routing_key      = "non-urgent"
  depends_on = [
    rabbitmq_exchange.coupon-event,
    rabbitmq_queue.coupon-event-non-urgent-cdp
  ]
  lifecycle {
    replace_triggered_by = [rabbitmq_queue.coupon-event-non-urgent-cdp]
  }
}

resource "random_password" "password" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "rabbitmq_user" "coupon-service" {
  name     = "coupon-service"
  password = random_password.password.result
  lifecycle {
    ignore_changes = [
      password
    ]
  }
}

resource "rabbitmq_permissions" "coupon-service" {
  user  = "${rabbitmq_user.coupon-service.name}"
  vhost = "/"

  permissions {
    configure = ""
    write     = "coupon.*"
    read      = ""
  }
}