The line between queue and log is where this decision lives

If you treat everything as a stream of immutable events, Kafka wins; if you treat everything as directed tasks that need smart handling, RabbitMQ wins and will hurt less in day-two operations. The hard part is not picking a tool once, but understanding what failure modes you’re buying in your architecture: backpressure shape, replay behaviour, delivery guarantees, and how much work it takes to route “the weird stuff” correctly.

By the end of this piece you should be able to look at a workload and say, with reasons, “this belongs on Kafka”, “this belongs on RabbitMQ”, or “this needs both, and here is the line between them”.

RabbitMQ is a router with queues; Kafka is a log with consumers

RabbitMQ implements AMQP 0‑9‑1 semantics: producers publish to exchanges, exchanges apply routing rules, and messages land in queues that consumers drain. Kafka exposes an append-only log: producers write records to partitions, consumers track offsets per partition and read forward independently.

RabbitMQ’s core abstraction is the queue. The queue is where you apply per-message TTL, priority, dead-lettering, and concurrency limits, and the exchange fabric decides which queue each message goes to. Kafka’s core abstraction is the topic partition: ordering and throughput are controlled by how you partition, and processing behaviour comes from consumer-group design and offset management.

If you need “do this job once, somewhere” with rich per-message handling, think RabbitMQ. If you need “record every state change and let multiple systems consume and replay later”, think Kafka.

Message model and routing semantics

RabbitMQ gives you multiple exchange types out of the box: direct, fanout, topic, headers, plus plugin-provided types such as x-delayed-message and consistent-hash exchanges. The practical difference is that you can encode a lot of routing logic in the broker itself: per-tenant queues, per-region delays, error channels, and complex “fan-out then filter” patterns without writing much bespoke glue code.

Kafka’s routing is simple by design: messages have a key; the partitioner maps that key to a partition; records within a partition are strictly ordered. Fan-out is done by having multiple consumer groups subscribe to the same topic; there is no “routing” in the AMQP sense, only topic choice and partitioning strategy.

If your routing rules depend on message headers, wildcards and binding keys, RabbitMQ matches that mental model directly. If your routing rules depend on event type and entity identity (user id, order id) and you want every service to see the same ordered history, Kafka matches that mental model.

Ordering and delivery guarantees

RabbitMQ offers at-least-once delivery by default: messages are persisted (depending on durability settings), delivered to consumers, and requeued on negative acknowledgement or consumer failure. Ordering is per-queue, but can be broken by priority queues, dead-lettering, and multiple consumers draining the same queue.

Kafka’s guarantees are expressed as at-most-once, at-least-once, and exactly-once semantics, depending on producer and consumer configuration. Ordering is per partition; within one partition you get strict order, but cross-partition you must treat records as concurrent and reconcile in your processing logic.

Exactly-once semantics in Kafka come from idempotent producers and transactional processing: the producer writes records and commits consumer offsets atomically, so a pipeline either processes a batch once or not at all. RabbitMQ does not expose a broker-level “exactly-once” mode; if you need that behaviour you implement idempotency at the consumer side, usually by deduplicating against a database or cache keyed by message id.

I would treat Kafka’s EOS pipeline as the default for financial, billing, and compliance workloads that need strict replay and no double-processing, and treat RabbitMQ plus application idempotency as the default for job queues and task orchestration.

Throughput versus latency under real workloads

Benchmarks across vendors and studies are noisy, but the shape is consistent: Kafka dominates raw throughput; RabbitMQ offers excellent tail latency for many task-queue workloads.

Confluent’s 2024 benchmarking against Kafka 3.7 measured roughly 605,000 messages per second sustained on a three-broker cluster with 1 KB messages and three-way replication. On the same hardware, RabbitMQ classic mirrored queues peaked around 38,000 messages per second before queue depth grew unbounded, a roughly 16× gap.

RabbitMQ’s classic queues on small instances regularly hit tens of thousands of messages per second; a 2026 managed RabbitMQ benchmark on modest hardware measured around 120,000 messages per second without publisher confirms and about 108,000 per second with confirms enabled. Kafka single-node benchmarks on similar small footprints tend to land above 100,000 messages per second, and cluster benchmarks routinely pass 500,000 per second.

Latency tells a different story. RabbitMQ delivers sub-millisecond median latency when routing in memory with local consumers; Kafka’s median latency is typically in the 5–15 ms range due to batched, durable writes and replication. In other words, Kafka wins when the workload is throughput-bound and you care about aggregate volume; RabbitMQ often wins when the workload is latency-bound and you care about “how fast does this job start” for human-facing operations.

Persistence, replay, and retention

Kafka’s design goal is durable, replayable logs. Topics can be configured with retention by time or size, and compacted topics keep only the latest record per key, making them suitable for state snapshots. Consumers can rewind offsets and reprocess from any point in the retained history, which is the foundation for event sourcing and “rebuild a projection from the log”.

RabbitMQ’s queues are not designed for long-term retention or replay in the same way. Messages can be persisted and survive broker restarts, but queues are generally drained; dead-letter queues hold failed messages, but are treated as exception channels, not permanent logs. If you want replay, you build it on top of RabbitMQ by copying messages to audit queues or an external store such as S3 or a database.

RabbitMQ Streams narrow this gap by providing append-only storage with Kafka-style semantics inside RabbitMQ, and benchmarks show they can reach hundreds of thousands of messages per second when tuned. That said, Kafka’s ecosystem — connectors, stream processing frameworks, and tooling for managing long-lived topics — makes it the default choice when “we will replay this five years from now” is part of the requirement.

Smart routing, priorities, retries, and dead letters

RabbitMQ’s strength is “smart routing at the broker”. Exchanges bind to queues with routing keys and arguments, and you can stack behaviours without writing code every time. Priority queues let urgent jobs jump ahead of bulk work; delayed-message exchanges provide scheduled delivery; dead-letter exchanges capture failures for inspection and replay.

The official dead-letter exchange behaviour is explicit. RabbitMQ will republish messages to a dead-letter exchange when:

The message is negatively acknowledged;

The message expires due to per-message TTL;

The queue exceeds its length limit and drops the message;

A quorum queue returns the message more times than its delivery limit.

That quoted list is the broker’s view of failure. It lets you set up “jobs that failed three times go here”, “expired jobs go there”, and handle them differently without modifying producers or consumers.

Kafka has no native concept of dead-letter queues. The usual pattern is to have consumers write problematic records to a dedicated “dead-letter” topic, with metadata about the error, and then treat that topic as a separate stream for analysis and remediation. Retries are usually implemented with backoff inside the consumer, often alongside idempotent writes to avoid duplicate side effects.

If your routing and error handling rules change frequently and you want to express them at the broker layer with policies, RabbitMQ gives you the knobs out of the box. If your error handling is tightly coupled to domain logic and you already have a stream-processing framework around Kafka, keeping retries and dead-letter handling in code may be cleaner.

Delivery semantics and exactly-once processing

Kafka’s delivery semantics are documented explicitly: at-most-once, at-least-once, and exactly-once modes, each with trade-offs around performance and complexity. The exactly-once story depends on idempotent producers and transactional APIs, which allow a pipeline to consume records, transform them, and produce new records while committing both outputs and offsets atomically.

Conduktor’s explanation of EOS is a good summary: idempotent production avoids duplicates on retries, and transactional processing ensures that consumers see a consistent history of what has been processed. In practice, the most reliable pattern is idempotent producers plus transactional writes plus idempotent consumers, so that either the whole batch is visible once or can be retried without double application.

RabbitMQ does not expose an EOS mode; instead, you combine durable queues, acknowledgements, and idempotent consumer behaviour. For job queues, this is usually enough: the job is either in the queue, being processed, or finished, and failures result in retries or dead-lettering with clear visibility into what went wrong.

I would reserve Kafka EOS for pipelines where double-processing causes real financial or compliance damage, and treat RabbitMQ’s at-least-once semantics plus consumer-level idempotency as the standard for operational and batch workloads.

Cluster behaviour, scaling, and failure modes

Kafka scales horizontally through partitioning and replication. Topics are split into partitions; partitions are allocated across brokers; each partition has a leader and followers for replication. Throughput grows by adding partitions and brokers; availability is maintained by replication factors and leader election within the cluster.

RabbitMQ historically relied on classic mirrored queues for replication, which had well-known scaling issues. The modern answer is quorum queues, a Raft-based replicated queue type “considered the default choice when needing a replicated, highly available queue”. Quorum queues trade some throughput for better data safety and leader election behaviour, and are explicitly designed for high availability under upgrades and node failures.

A 2026 Kubernetes-based broker benchmark evaluated brokers against throughput, low latency, and multifunction requirements; Kafka won on high-throughput workloads, while other brokers, including RabbitMQ, showed favourable latency characteristics and richer functionality for certain patterns. That matches production experience: Kafka scales out as a log infrastructure; RabbitMQ scales adequately for many task workloads, but you must pay attention to queue types, mirror strategies, and backpressure.

For high-cardinality workloads with thousands of small queues — per-user or per-device queues — RabbitMQ’s routing model can carry the load, provided you design for connection limits and use modern queue types. For high-volume event streams with long retention and multiple independent consumers, Kafka’s partitioned topics and consumer groups are the safer base.

When RabbitMQ is the right choice

RabbitMQ is the right choice when you have:

Task queues where each message represents a unit of work: send an email, resize an image, run a report.
Complex routing rules: route by tenant, by severity, by feature flag, or by geographic region.
Rich per-message handling requirements: per-message TTL, priority, delayed delivery, dead-lettering, and rate limiting.

The broker gives you tools to express these behaviours declaratively. For example, a delayed queue for retries:

bash

# Declare a delayed exchange
rabbitmqadmin declare exchange name=retry-exchange type=x-delayed-message \
  arguments='{"x-delayed-type":"direct"}'

# Bind a queue to receive retries
rabbitmqadmin declare queue name=jobs-retry \
  arguments='{"x-dead-letter-exchange":"jobs-exchange"}'

rabbitmqadmin declare binding source=retry-exchange destination=jobs-retry routing_key=jobs.retry

Here the broker handles “wait N seconds, then put it back on the main exchange” for you; your consumer only needs to set the delay and routing key.

RabbitMQ is also a better fit when:

You want low median latency for human-facing operations.
You accept that long-term replay belongs in a separate storage layer (database, object store), not in the queue.
You prefer to keep operational complexity lower; a RabbitMQ cluster with quorum queues is simpler to reason about than a large Kafka deployment for many teams.

I would default to RabbitMQ for monolith-plus-worker architectures, microservices that need straightforward request/worker semantics, and systems where “one job, somewhere, soon” is the primary concern.

When Kafka is the right choice

Kafka is the right choice when you have:

High-volume, append-only event streams: click events, transaction logs, telemetry from millions of devices.
Multiple independent consumers of the same data: fraud detection, analytics, ETL, real-time dashboards subscribing to the same topic.
A need for replay and time travel: rebuild aggregates, re-run models, rehydrate caches from past events.

A typical Kafka configuration for a core event stream might look like this:

bash

# Create a topic with 12 partitions and 3-way replication
kafka-topics.sh --bootstrap-server kafka:9092 \
  --create --topic orders-events \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=compact,delete \
  --config retention.ms=604800000   # 7 days

Here you get both log retention and compaction, so consumers can either replay the last week of events or read the latest state per key.

Kafka is also the right choice when:

Your throughput requirements are in the hundreds of thousands to millions of messages per second, and you want scalability with predictable characteristics.
You want exactly-once or “effectively once” semantics inside streaming pipelines, which Kafka’s transactional APIs support directly.
You plan to integrate with a broad ecosystem of connectors, stream processors, and warehouses; most tooling assumes Kafka or Kafka-compatible interfaces.

I would default to Kafka for event-sourced systems, data platforms, and cross-service communication where every event is part of a shared history multiple teams care about.

Mixed architectures: using both without tripping over yourself

Many systems benefit from both: Kafka as the system-of-record log, RabbitMQ as the task queue for doing work off those events.

A clean pattern is:

Kafka topics hold canonical events: order-created, payment-authorised, invoice-issued.
A small service consumes Kafka, derives work units, and publishes jobs into RabbitMQ: “send invoice email”, “generate PDF”, “sync to CRM”.
RabbitMQ workers handle retries, priorities, and dead-lettering; Kafka retains the history.

The boundary should be explicit in code and configuration. For example, a Kafka consumer that bridges to RabbitMQ:

python

import json
from kafka import KafkaConsumer
import pika

consumer = KafkaConsumer(
    "invoice-issued",
    bootstrap_servers=["kafka:9092"],
    group_id="invoice-jobs",
    enable_auto_commit=False,
)

rabbit = pika.BlockingConnection(pika.ConnectionParameters("rabbitmq"))
channel = rabbit.channel()
channel.exchange_declare(exchange="invoice-jobs", exchange_type="direct")

for message in consumer:
    event = json.loads(message.value)
    job = {
        "invoice_id": event["invoice_id"],
        "user_id": event["user_id"],
    }
    channel.basic_publish(
        exchange="invoice-jobs",
        routing_key="send-email",
        body=json.dumps(job),
        properties=pika.BasicProperties(
            delivery_mode=2,  # persistent
            headers={"x-retry-count": 0},
        ),
    )
    consumer.commit()

Kafka keeps the history of invoice-issued; RabbitMQ carries the operational workload of sending and tracking emails with smart retries.

The main thing to avoid is “Kafka for everything, RabbitMQ for everything”, with no clear division. Pick one as the log of record, and treat the other as a work engine fed from that log.

Failure modes engineers actually hit

With RabbitMQ, the common failure modes are:

Queues growing without bound because consumers are slower than producers and you haven’t set length limits or backpressure.
Priority queues breaking ordering in ways that surprise downstream logic.
Dead-letter exchanges misconfigured, so failed messages disappear instead of going to inspection queues.

With Kafka, the common failure modes are:

Hot partitions: a small subset of keys dominates traffic, causing uneven load and hurting throughput.
Consumer lag: consumer groups fall behind publishers, and you discover that retention or disk isn’t sized for sustained lag.
Misunderstood EOS: teams believe they have exactly-once semantics but haven’t configured idempotent producers or transactional processing correctly.

A 2026 benchmark study on Kubernetes brokers noted that Kafka’s throughput increases with message size up to a point and then stabilises, while RabbitMQ’s throughput continues to rise with larger messages; that matters when you move from small telemetry events to larger payloads. In practice, I would treat “What happens when our traffic doubles?” as the primary question for Kafka, and “What happens when our consumers stall?” as the primary question for RabbitMQ.

The one question to ask before you pick

The single most useful question is: do you need a durable, replayable log of everything that happened, or do you need a reliable system for doing work off events?

If you need the log, Kafka is the default, and you should design around topics, partitions, consumer groups, and EOS or at-least-once semantics depending on risk. If you need the work engine, RabbitMQ is the default, and you should design around exchanges, queue types (quorum, classic, streams), dead-lettering, and backpressure policies.

You can glue them together cleanly, but only if you draw the line early: Kafka as the spine of your data, RabbitMQ as the muscles that move jobs around it.

Asynchronous Event Driven Architecture: RabbitMQ vs. Apache Kafka