Transaction Retry Architecture With Retry Coordinator

This document focuses exclusively on the transaction retry subsystem — the mechanism by which StackSaga detects paused transactions and re-invokes them safely, without duplication or conflict, across a cluster of microservice instances.

Topics covered:

Identifying the nature of errors is crucial for effective transaction management.

StackSaga distinguishes two classes of errors:

At the first glance, it may seem easy to manage retrying by re-invoking the transaction on the same node that originally processed it. However, this approach can lead issues.

the main concern is that the standard nodes are ephemeral and can become unresponsive or fail by nature, especially in a dynamic microservices environment where instances can be scaled up or down based on traffic demands. This can lead to a situation where the transaction remains paused indefinitely. for instance, when the time has come to be exposed for retrying the respective transaction, the node that originally processed it might not be available anymore, leading to a situation where the transaction remains paused indefinitely.

For instance, just imagine there are 3 instances are running on, and there are 7 transactions have been saved for retrying by them due to some network issues. If each instance takes care of the transactions by themselves, After a while, the scheduler is triggered for replaying transactions on each instance. At that moment it can have different instances count due to scaling up or scaling down based on the traffic at that particular time due to the scaling up and down. Just imagine an instance that has made some transactions is not running on when the scheduler is triggered for retrying called order-service-3400001. At that moment, other instances do not touch to those transactions that made by that respective instance. hence, those transactions will not be exposed for retrying ever, like the diagram shows below.

As a solution, StackSaga proposes a retry architecture that decouples transaction retrying from the original processing node. it consists of 3 components and let’s begin to explore them one by one.

As mentioned above, the standard nodes are ephemeral and that are short-lived, temporary, and designed to be created, destroyed, or replaced on-demand. based on the ephemeral term, in Stacksaga architecture, it can be deployed nodes in tow different modes:

Definition:

For instance, in a typical deployment, you might have 5 or 1000 standard nodes that handle transaction processing, and 2 retry-nodes that manage retries while handle transaction processing as well. the standard nodes can be scaled up or down based on traffic demands, while the 2 retry-nodes remain stable to ensure that transaction retries are managed effectively. because, it is not necessary increase or decrease the retry-nodes based on the traffic. and also there is no barrier scale up or down the retry-nodes. retry-nodes are designed to be more stable and long-lived than standard nodes. because the retrying is the worst case scenario for the transaction, not the regular flow. read the article of Proportional Analysis of Failure Modes in Long-Running Transactions Using the Saga Pattern to understand more about need of retrying and the weight .

The next challenge is how to manage the retrying across the cluster of retry-nodes without risking duplicate processing or conflicts. because they have no any aware about which transactions should be retried.

Here the Token Ring Partitioning comes to the picture. it is the key mechanism that allows StackSaga to safely distribute retry responsibility across multiple Orchestrator instances without any risk of overlap or conflict.

StackSaga uses the Murmur3 consistent hashing algorithm to assign every transaction to a deterministic position in a 64-bit token ring. This token is computed when the transaction is initialised (by any orchestrator) and stored alongside the transaction record in the database, together with the cluster name and region of the originating Orchestrator. The retry subsystem uses this stored token, cluster, and region to determine which orchestrator instance is responsible for retrying a given transaction at any point in time.

The full token space spans from \(-2^{63}\) to \(2^{63}-1\):

The below diagram illustrates how the retry-nodes are assigned non-overlapping segments of the token ring. There are 4 retry-nodes in this example, each responsible for a distinct quarter of the token space:

The full Murmur3 hash space is divided equally across 4 Retry-Node instances.

As we have seen in the previous section, a slot in the token ring is assigned to each Orchestrator instance. the next concern is how the token ring is partitioned and how the Orchestrator instances receive their assigned token ranges.

Two components are responsible for the token ring partitioning and distribution:

The Master is the single authority that divides the full token ring among registered Slaves. The slaves take their assigned slice and further divide it among their connected Orchestrator instances.

Before diving into each component, let’s have a small preview of how they collaborate to manage transaction retries.

All components are working to safely share the token range of the Murmur3 token ring, which is used to partition the transaction space and ensure that retries are distributed without conflict.

Now that each component is clear on its own, here is the simplest possible summary of how they collaborate:

Each component does its own small job cleanly. No component reaches into another’s responsibility. The result is a retry system that scales horizontally, tolerates partial failures, and guarantees that every paused transaction is eventually retried — exactly once at a time.

StackSaga uses the Murmur3 consistent hashing algorithm to assign every transaction to a deterministic position in a 64-bit token ring. This token is computed when the transaction is initialised and stored alongside the transaction record in the database, together with the cluster name and region of the originating Orchestrator. The retry subsystem uses this stored token, cluster, and region to determine which orchestrator instance is responsible for retrying a given transaction at any point in time.

The full token space spans from \(-2^{63}\) to \(2^{63}-1\):

The Master divides the full token ring into equal segments, one per registered Slave node. With three Slave nodes the ring is divided as follows:

Each Slave further divides its received token range equally among all Orchestrator instances registered with it.

The following example shows how Slave Node 1 divides its range among two Order Service instances:

All other Slave nodes apply the same logic for their own registered Orchestrator instances.

The Retry-Coordinator-Master runs a repeating timer that fires at the 30th second of every minute. At each trigger the Master:

The published range is labelled as valid for the next full minute (minute T+1), not the current minute.

Publishing at the 30th second and marking the range as valid for the next minute creates a 30-second preparation window. This window is intentionally allocated to account for worst-case communication delays: Slave distribution, Orchestrator updates, and network latency in large clusters.

By the time minute T+1 begins, all Orchestrators are guaranteed to have received their new sub-ranges and are ready to start polling.

Because each Orchestrator holds a non-overlapping sub-range for a defined time window, two Orchestrator instances can never claim responsibility for the same transaction simultaneously. This eliminates the need for distributed locking when retrying transactions.

When a transaction is created it inherits the region of the Orchestrator that created it. That region value is stored in the transaction record in the database.

During the retry polling loop, each Orchestrator adds region as a mandatory filter alongside the token range:

This guarantees that an Orchestrator in us-central never picks up and retries a transaction that was originally created by an Orchestrator in asia-south — even if both regions share the same database (e.g., a globally replicated Cassandra cluster).

In very large deployments the single-master-per-domain model, while operationally straightforward, presents two concerns as the system grows:

Virtual clusters address both concerns by allowing you to divide a single physical deployment into multiple independent master-slave groups, each operating as a completely separate retry coordination unit, all within the same region.

A virtual cluster is a named group of Retry-Coordinator-Master, Agent Slave, and Orchestrator instances that operate in isolation from other groups. The group identity is declared via the stacksaga.instance.cluster property.

All three components that share the same cluster name form one logical retry cluster:

Two virtual clusters running in the same physical Kubernetes cluster are completely isolated from each other — they share no state, no connections, and no retry responsibility.

The simplest multi-region setup is one virtual cluster per region. This is the baseline configuration for geographic distribution.

For the highest levels of scale and fault isolation, each physical region can host multiple virtual clusters. In this topology, if any single Master fails, only the fraction of transactions belonging to that virtual cluster are affected — the remaining virtual clusters in the same region continue retrying normally.

A transaction is permanently bound to the region and cluster of the Orchestrator that created it. No other virtual cluster — even within the same region — will ever attempt to retry it.

The retry query always includes both region and cluster as mandatory filters:

This makes the system particularly well-suited to Apache Cassandra deployments where region and cluster form part of the partition key, enabling the database to route the query directly to the correct nodes with zero full-table scans.