Idempotency & Atomic Transactions For LRT

What is the idempotence?

Idempotency in a microservices architecture means that processing the same operation multiple times has the same effect as processing it once — whether that operation arrives as a repeated synchronous request (for example over HTTP/REST) or as a redelivered message from a broker (for example a Kafka record). This ensures that operations are safe to retry without causing unintended side effects, which is especially important in distributed systems where failures, retries, and message redeliveries are common.

Why is Idempotence Important in Microservices?

In microservices architecture, services are often distributed and communicate over unreliable networks. As a result:

  • Failures and Retries: When an operation fails due to network issues or system crashes, the caller may retry it. Without idempotence, multiple retries could lead to inconsistent data or unexpected behavior.

  • Asynchronous Messaging: When using message queues, brokers may send the same message multiple times if acknowledgment is not received, or due to delivery guarantees such as "at-least-once."

  • Concurrency: Multiple instances of a service might attempt to process the same request or data at the same time, and idempotence ensures that this doesn’t lead to incorrect results. For example, imagine a microservice responsible for billing. If a charge is applied twice due to a retry, the customer could be billed multiple times without idempotence in place.

Examples of Idempotent and Non-Idempotent Operations

Idempotency is a property of the operation, not of the protocol that carries it. The same operation can travel over a synchronous protocol such as HTTP/REST or as a message consumed from a broker such as Kafka; what matters is its effect on state when it is applied more than once.

Idempotent Operations

  • Reading an order — fetching the same order multiple times does not change the system state.
    (REST: GET /orders/123 — Messaging: a get-order query message)

  • Updating an order to a fixed value — applying the same update repeatedly results in the same final state.
    (REST: PUT /orders/123 — Messaging: an order-updated event carrying the full new state)

  • Deleting an order — deleting it again has the same effect as deleting it once.
    (REST: DELETE /orders/123 — Messaging: an order-deleted event keyed by the order id)

Non-Idempotent Operations

  • Creating a new order on every call — each delivery creates another order.
    (REST: POST /orders — Messaging: a create-order command consumed twice produces two orders)

  • Relative changes such as "add $100 to the balance" — each delivery shifts the state again.
    (REST: POST /wallets/123/add?amount=100 — Messaging: an add-funds command consumed twice doubles the amount)

These non-idempotent operations are the ones that must be given an idempotency mechanism before they can be safely retried or redelivered.

Implementing Idempotency

Use idempotency keys: The caller attaches a unique identifier to every operation so duplicates can be recognised. In REST this is typically a header such as Idempotency-Key; in messaging it is usually the message key or a header (for example a Kafka record key, or a messageId carried in the record headers).

Check before processing: The receiver keeps a record of already-processed keys and ignores (or replays the stored result of) any operation it has seen before.

Design operations to be naturally idempotent: Prefer setting an absolute state over applying a relative change. In REST this often means preferring PUT (set this resource to this value) over POST (create another one); in messaging it means modelling events as full-state, keyed records and using upserts rather than blind inserts.

Use database constraints: Enforce unique constraints (for example a unique business key) so that a duplicate write fails fast instead of creating a second record.

Why Idempotency is Important with Retries?

  1. Prevents Duplicate Transactions

    Suppose a payment service handles a make-payment operation (an HTTP POST /payments call, or a make-payment command consumed from a topic). If the caller does not receive an acknowledgement due to a network failure and retries — or the broker redelivers the message — the same payment might be processed twice, leading to double charges for the customer.

    With idempotency, the service recognises the repeated operation and ensures it is processed only once.

  2. Ensures Data Integrity
    Without idempotency, retrying a non-idempotent operation (like adding money to a wallet) might result in incorrect balances.

    Example (delivered over any protocol):

    First delivery: add-funds(amount=100) → Balance: $100

    Redelivery after failure: add-funds(amount=100) → Balance: $200 (Incorrect!)

    With idempotency, the service checks whether the operation has already been applied and ensures the balance remains correct.

  3. Avoids Partial Updates & Inconsistencies

    In distributed transactions, a microservice might update multiple databases or services. If retries happen without idempotency, it could lead to partially completed operations.

    Example:

    Step 1: Service A debits an account.

    Step 2: Service B fails before updating the order status.

    Step 3: The caller retries (or the broker redelivers), causing another debit.

    Idempotency ensures only one successful debit happens, even with multiple retries.

  4. Improves System Resilience

    Retries happen automatically almost everywhere — client-side libraries such as Spring Retry or Resilience4j for synchronous calls, and at-least-once delivery in message brokers such as Kafka, RabbitMQ, or SQS for asynchronous communication.

    If a service is not idempotent, these automatic retries and redeliveries might overload the system with unintended duplicate operations.

How to Ensure Idempotency for Retried Operations in General?

  1. Use an Idempotency Key

    The caller attaches a unique identifier to the operation (e.g., an Idempotency-Key header over HTTP, or a message key / messageId header on a Kafka record).

    The receiver checks whether an operation with that key has already been processed.

    If it has, the receiver skips re-execution — and, for request/response styles, returns the previously stored result instead of running the operation again.

  2. Store Processed Keys

    Maintain a record of processed keys in a database or cache to avoid reprocessing.

  3. Use Database Constraints

    Enforce unique constraints in the database (e.g., a unique business key such as the order id for payments).

  4. Prefer State-Setting Operations over Relative Ones

    A state-setting operation reaches the same final state no matter how many times it is applied, whereas a relative or create-style operation does not. In REST this means preferring PUT /orders/123 over POST /orders; in messaging it means modelling events as full-state, keyed records and applying them as upserts.

Understanding Atomic Operations and Atomic Transactions

Before looking at how idempotency is applied inside a long-running transaction, you first need a clear picture of the unit that idempotency actually protects: the atomic operation.

An atomic transaction is a set of operations that must either all succeed or all fail as a single unit. Within microservices, such an all-or-nothing unit is referred to as an atomic execution.

atomic operations of the place-order long-running transaction

For instance, in the placing-order scenario above, there are 4 atomic operations. Each atomic execution can itself consist of multiple service-level operations. It is perfectly acceptable to have multiple service-level operations inside a single atomic execution, as long as they all belong to the same logical unit of work.

Fetching the user’s details is quite different from the other atomic executions, because it is a read-only operation that does not change any state in the user-service’s database. It can therefore be considered an atomic operation, but it is not an atomic transaction. The others are atomic transactions, because each of them changes state in its respective service’s database.

In short, every atomic transaction is an atomic operation, but not every atomic operation is an atomic transaction — only the state-changing ones are.

Understanding where these atomic boundaries lie is crucial, because there is no rollback mechanism once an atomic transaction has been committed to its service’s database. Once a local transaction commits, it stays committed; the only way back to a consistent state is a compensating action, not a rollback.

How Idempotency and Atomic Transactions Relate

Idempotency and atomicity are two different ideas that work together inside a long-running transaction:

  • An atomic transaction is about granularity — it is the all-or-nothing unit of work that the saga advances one step at a time.

  • Idempotency is about repeatability — it is the property that lets a single unit be applied many times with the same effect as applying it once.

The two ideas meet because of how a saga keeps data consistent without a global transaction. Each atomic transaction commits to its own service’s database independently, and — as noted above — once it commits there is no rollback. To stay consistent under failure, the saga engine relies on two safety nets:

  1. Retry — for transient failures (a network blip, a temporarily unavailable service), the same atomic execution is invoked again.

  2. Compensation — for permanent failures, the atomic transactions that already committed are undone by their compensating actions in reverse order.

Retry is exactly where idempotency becomes mandatory. When the engine re-invokes a state-changing atomic transaction, the operation may already have been applied on the first attempt. Without idempotency, the retry would apply it a second time — charging the customer twice, decrementing the stock twice, or creating a duplicate order.

The atomic transaction defines what gets retried, and idempotency guarantees that retrying it is safe — one is the unit of work, the other is the property that makes that unit safe to repeat.

This pairing also explains the read-only case. A read-only atomic operation (such as fetching the user’s details) is naturally idempotent — repeating it changes nothing — which is precisely why it never needs a compensating action. It is the state-changing atomic transactions that must be made idempotent and given compensations.