Stacksaga Cassandra Reactive Support

Overview

stacksaga-cassandra-reactive-support is the Cassandra reactive (non-blocking) implementation of the Stacksaga Event Store. It provides all the necessary facilities for accessing the Cassandra database for the Stacksaga engine, and exposes endpoints for accessing the event store to display tracing details in the StackSaga Trace-Window.

It is used in the orchestrator service alongside any StackSaga orchestrator starter. See Saga Orchestrator Engines for the full list of available implementations and their respective starters.

Retry Ordering Guarantee (No Strict FIFO in Cassandra):
Unlike SQL implementations where retrying is processed strictly in chronological First-In, First-Out (FIFO) order (ORDER BY paused_time ASC), the Cassandra event-store implementation fetches transactions by token order (token(transaction_id)), not by insertion or paused timestamp.

Consequently, Cassandra does not guarantee strict chronological retry ordering within the same schedule window. If your business domain strictly requires exact FIFO retry ordering, Cassandra is not recommended as the event-store.

Architectural Recommendation for Strict FIFO Requirements:
As a general best practice, StackSaga recommends using the same database technology for both your primary domain data and the event-store to eliminate the operational overhead of running multiple database engines. However, if your system uses Cassandra as its primary database and has a strict requirement for exact FIFO retry ordering, StackSaga recommends using one of the SQL database implementations specifically for the event-store. In this architecture, your high-throughput domain operations execute against Cassandra, while StackSaga manages its event store and retry queue in SQL to guarantee strict FIFO ordering. See Retry & Recovery Ordering Guarantee: Token Order vs. Strict FIFO for technical details.

Adding cassandra-support to your orchestrator application

Adding cassandra-support to your orchestrator application consists of 3 simple steps.

Step-1

Add stacksaga-cassandra-reactive-support as a dependency to your orchestrator application.

Step-2

Execute the schema creation script provided by the stacksaga-cassandra-reactive-support module.

Step-3

Configure the Cassandra connection configurations.

The following sections will explain each step in detail.

Adding stacksaga-cassandra-reactive-support as a dependency

Here is the way that you can add the library into your existing orchestrator application as a dependency.

Adding stacksaga-cassandra-reactive-support as a dependency
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.stacksaga</groupId>
            <artifactId>stacksaga-bom</artifactId>
            <version>1.0.0-SNAPSHOT</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependency>
    <groupId>org.stacksaga</groupId>
    <artifactId>stacksaga-cassandra-reactive-support</artifactId>
</dependency>
It is recommended to use StackSaga Initializer to get the dependency snippets for your project for StackSaga related dependencies, as it will ensure you have the correct versions and configurations for your project setup.

Generate Schema

Because StackSaga creates some dynamic tables based on the service name, region, and cluster for saving transaction data, a schema creation tool is provided for your convenience.

Understanding Region and Virtual Cluster

Before generating the schema, it is important to understand what region and cluster mean in the StackSaga context — because these two inputs directly determine the names of the generated tables.

What is "Region"?
A region maps to your physical deployment location (e.g. us-central, asia-south). Every transaction is stamped with the region of the orchestrator that created it. This ensures that recovery and retry operations in one region never interfere with those in another — even when both share the same Cassandra cluster.

What is a "Virtual Cluster"?
By default, a region contains exactly one cluster (a one-to-one relationship — for example, region us-central maps to a single cluster also named us-central).

However, for large-scale deployments you can create multiple virtual clusters within the same physical region (e.g. us-central-c1, us-central-c2, us-central-c3). Each virtual cluster gets its own independent set of generated tables (es_rt_* and es_irt_*) and its own dedicated Retry Ring Coordinator deployment. This lets you split the retry subsystem into isolated streams — if one virtual cluster’s coordinator fails, the others continue operating normally.

See Virtual Clusters for the full conceptual explanation and Ring-Coordinator Deployment Topologies to understand how the Ring-Coordinator and Retry-Nodes are deployed in practice across single and multiple virtual clusters.

How does this affect table names?
The generated tables are named using a hash suffix derived from the combination of service name, region, and cluster. This hash-based approach keeps table names short and avoids issues with special characters in region or cluster identifiers. You never need to construct these names manually — the schema generator tool produces the correct names for you based on your inputs.

Schema Generator

Enter your service name, region, and cluster name to generate and download the .cql schema script for your Cassandra database.

If you deploy the application in multiple regions or clusters, you must run the generator separately for each combination of service name, region, and cluster. Each run produces a distinct set of Generated Tables (with a different hash suffix), while the Constant Tables are the same across all runs.

→ Open the Cassandra Schema Generator

Configuring Cassandra connection

How Transaction Data Is Stored in Cassandra

The primary responsibility of the Cassandra database support implementation is providing the facilities for saving transaction data in the event store. The approach to saving transaction data is quite different from SQL support implementations due to Cassandra’s data modeling characteristics — Denormalized Schema, clustering, partitioning, and the absence of centralized indexing.

Table Overview

All tables used by StackSaga fall into two categories:

Constant Tables — table names never change regardless of service, region, or cluster. These are created once as part of the schema script and are shared across all deployments:

Table Purpose

es_transaction

Main transaction metadata store

es_transaction_tryout

Transaction execution step (tryout) records

es_execution_markers

Write protection and duplicate detection for Kafka event streaming

es_frozen_transaction

Compensation failed transactions requiring manual intervention

Generated Tables — table names include a hash suffix derived from your service name, region, and cluster. A separate set is created for each combination of service, region, and cluster:

Table Pattern Purpose

es_rt_1_{hash}, es_rt_2_{hash}, es_rt_3_{hash}

Transaction recovery retention (crash / missing transaction tracking)

es_irt_1_{hash}, es_irt_2_{hash}, es_irt_3_{hash}

Immediate retry retention (paused transaction tracking)
The {hash} suffix is computed by the schema generator tool from your service name, region, and cluster name.

es_transaction table

es_transaction is the main table where transaction metadata is saved in the event store, using the transaction ID as the Partition Key. All transactions are saved in this table as Single-Row Partition. The Single-Row Partition approach enables handling millions of transactions without creating database hotspots across Cassandra nodes, because transactions are distributed across all available nodes efficiently as shown below.

StackSaga cassandra managing throughput

Even though Single-Row Partition helps overcome the database hotspot problem, data cannot be fetched without knowing the exact transaction key. This makes it impossible to bulk-scan for transactions needing recovery or retry. This is addressed by the es_rt_* recovery tables and es_irt_* retry tables described below.

es_transaction_tryout table

es_transaction_tryout is the table where transaction tryout data is saved, using the transaction ID as the Partition Key. Transaction tryouts are saved under the transaction ID with a Multi-Row Partition approach. Because tryout data is keyed by the same transaction ID, it is stored on the same Cassandra node as the parent transaction in es_transaction. This colocation optimizes network latency — all related data flows to and from the same node.

StackSaga cassandra es_transaction_tryout table

As shown in the diagram above, tryout data is saved on the same node as the parent transaction. If the tx-8dKz7LpJ2Q transaction metadata is saved on Node-1, the tx-8dKz7LpJ2Q tryout data is also saved on Node-1.

es_execution_markers table

es_execution_markers is the table used by StackSaga’s Cassandra reactive event store to track individual executed records/spans and enforce write protection. For complete details on how this table prevents duplicate execution in Kafka event streaming and guards against concurrent race conditions, see Write Protection & Duplicate Prevention.

es_frozen_transaction table

es_frozen_transaction is a constant table that stores Compensation Failed Transactions — transactions where a compensation span itself failed with a permanent, non-retryable error, leaving the transaction stuck in a partially compensated state. These transactions cannot be retried automatically and require manual developer intervention: the team diagnoses the root cause, deploys a fix, and manually restores the transaction via the restore endpoint.

See Compensation Failed Transactions for the full explanation of this failure category.

Two Tracking Problems: Recovery vs. Retry

Cassandra’s partition-based distribution prevents bulk filtering of transactions directly from the main transaction table (see Why Bulk Filtering is Not Possible in Cassandra for details). To track transactions that need attention, StackSaga uses two separate sets of generated tables — each solving a fundamentally different problem:

Problem Table Set When a record is added / deleted

Transaction paused due to transient error
A downstream resource was temporarily unavailable.

es_irt_1_{hash} / es_irt_2_{hash} / es_irt_3_{hash}
(immediate retry retention)

Added only when the transaction stops due to a retryable error.
Deleted immediately after it is fetched by a Retry-Node for re-invocation.

Transaction crashed / went missing
The orchestrator node died unexpectedly mid-execution.

es_rt_1_{hash} / es_rt_2_{hash} / es_rt_3_{hash}
(recovery retention)

Added when the transaction is initialized.
Deleted only when it completes successfully.

Both table sets use the same three-table rotation mechanism, but with very different time windows — typically hours for recovery and minutes for retry.

Why Bulk Filtering is Not Possible in Cassandra

  • Partition-Based Distribution
    Cassandra shards data across multiple nodes using the transaction ID as the partition key. Because rows are distributed across the entire cluster, querying transactions based on a field (such as status or timestamp) without knowing the exact partition key requires contacting multiple nodes, making global status queries inefficient and impractical at scale.

  • No Centralized Indexing for Status-Based Queries
    In a relational database, finding pending transactions is as simple as running WHERE status = 'PENDING'. In Cassandra, secondary indexes on low-cardinality columns (like status) cause severe performance degradation across cluster nodes. Consequently, status-based bulk filtering directly on the main transaction table is not feasible.

The Alternative: Dedicated Rotating Tables as Status Queues

Because Cassandra cannot filter transactions by status in-place, StackSaga adopts an architectural solution where table membership itself represents the transaction’s state:

  • Presence in es_irt_* = The transaction paused due to a transient error and needs retry.

  • Presence in es_rt_* = The transaction was initialized and has not yet completed. If it remains in the table past the retention window, it is considered crashed or missing and needs recovery.

  • State Transition via Deletion: Once a transaction is picked up and referred for execution by a Retry-Node, its row is immediately deleted from the table. Deletion acts as the status update.

Why Are There Three Tables per Category?

Both recovery (es_rt_*) and retry (es_irt_*) use a rotating set of three tables (_1, _2, and _3):

  • Each table is assigned to a specific scheduled time slot in a round-robin rotation.

  • Dividing the retention period across three tables allows the system to drain and process batches on a predictable schedule without requiring expensive time-range scans across Cassandra partitions.

Why Tables Are Generated per Service, Region, and Virtual Cluster

Generated tables (es_rt_* and es_irt_*) carry a hash suffix derived from the service name, region, and virtual cluster:

  • Regional and Cluster Isolation: Retry-Nodes only access tables matching their own service, region, and virtual cluster. A Retry-Node in us-east-1 never touches data originating from us-west-1, even in a globally shared Cassandra cluster.

  • Independent Scaling: Virtual clusters can scale up or down independently and deploy dedicated Retry Ring Coordinators without cross-cluster interference.

  • Clean Partition Scopes: Schema scripts generate distinct table sets for each cluster, keeping individual tables compact and manageable.

How Retry-Nodes Process These Tables (Token Range-Based Batch Fetching)

Why the table itself acts as the status indicator
In a SQL database you can simply filter WHERE status = 'PENDING' to find work. In Cassandra, that is not possible without an expensive full cluster scan, because data is sharded by partition key across nodes. StackSaga’s solution is elegant: the presence of a row in the table IS the status. If a row exists in es_rt_*, that transaction needs recovery. If a row exists in es_irt_*, that transaction needs retry. No status field is needed — membership in the table is the signal.

The actual fetching and re-invocation is done by Retry-Nodes — orchestrator instances that have the Ring-Coordinator Connector added and enabled. The Retry Ring Coordinator (Master + Slave) is not responsible for touching the database. Its only role is to assign each Retry-Node a token range — a unique, non-overlapping slice of the Cassandra token ring — so that no two Retry-Nodes ever fetch the same rows simultaneously.

The way Ring-Coordinator instances and Retry-Nodes are deployed together depends on the scale and virtual cluster configuration of your system. See Ring-Coordinator Deployment Topologies for the available deployment options — from an embedded single-instance setup to fully clustered multi-virtual-cluster deployments.

Here is how a Retry-Node processes a table (the same flow applies to both es_rt_* and es_irt_*):

  1. Receive a token range from the coordinator
    The Retry Ring Coordinator pushes the Retry-Node its assigned token range (e.g., token 1 to 1000) for the current time window. The Ring Coordinator itself never connects to the Cassandra database; it only manages and assigns token ranges across active Retry-Nodes.

  2. Determine the target table
    Based on the configured schedule, the Retry-Node identifies which table slot is currently active (e.g., es_irt_1_{hash} at 00:00, es_irt_2_{hash} at 00:02, etc.).

  3. Fetch a batch by token range (without specifying partition keys)
    Because we need to fetch a batch and do not know in advance which transaction IDs are in the table, we query using Cassandra’s token() function across our assigned range rather than filtering on specific partition keys or statuses:

    SELECT * FROM es_irt_1_{hash}
WHERE token(transaction_id) >= 1 AND token(transaction_id) <= 1000
LIMIT 100;  -- controlled by stacksaga.cassandra.transaction.stream.batch-size

    Targeted Node Retrieval (No Full Cluster Scan): Cassandra’s broker knows exactly which physical nodes hold partitions within token range [1, 1000]. If the data within that range spans across 10 nodes in the cluster, the broker routes requests strictly to those 10 nodes and aggregates the results — it never queries all nodes in the cluster. The Retry-Node receives up to batch-size (e.g., 100) records.

  4. Re-invoke the transactions
    Each fetched transaction is immediately referred for execution, starting from the last successfully recorded span.

  5. Delete the rows immediately after referral
    Once a transaction is handed over for re-invocation, its row is deleted from the table:

    DELETE FROM es_irt_1_{hash} WHERE transaction_id = :transaction_id;

    Why immediate deletion is mandatory: In Cassandra, tables do not support in-place status mutation without incurring heavy secondary indexing penalties. Instead, table membership itself represents status. Deleting the row immediately upon batch retrieval marks it as processed and prevents that same transaction from being exposed again in the subsequent batch fetch.

  6. Continuous long-polling with configured delay
    Even if a batch query returns empty (no records found), the Retry-Node does not stop. It maintains a continuous long-polling loop with a small configured delay throughout the active time window, ensuring that any new transactions arriving in the table partition are picked up promptly. Once the time window expires, the Retry-Node advances to the next scheduled table slot.

stacksaga-cassandra-retry-node-processing-flow
Figure 1. Retry-Node Token-Range Batch Processing Flow

Why is this safe with multiple Retry-Nodes running in parallel?
Each Retry-Node holds a non-overlapping token range. Token range 1–1000 and token range 1001–2000 can never produce the same rows. This means multiple Retry-Nodes can fetch from the same table concurrently without coordination, locks, or conflicts — a major architectural advantage of this design over SQL-based implementations.

Retry & Recovery Ordering Guarantee: Token Order vs. Strict FIFO

When designing a distributed transaction retry strategy, understanding how the database driver orders retrieved batches is critical.

How SQL Implementations Achieve Strict FIFO Ordering

In SQL implementations, identifying transactions to retry or recover is executed via indexed queries with explicit sorting:

SELECT * FROM saga_retry_queue
WHERE status = 'PAUSED'
ORDER BY paused_time ASC
LIMIT 100;

Because the SQL database sorts by timestamp, transactions are retried in strict First-In, First-Out (FIFO) order: a transaction that paused at 00:02:10 is guaranteed to be retried before a transaction that paused at 00:02:40.

How Cassandra Processes Batches: Murmur3 Token Order

In Cassandra, rows are partitioned across cluster nodes using a hashing function (by default, Murmur3Partitioner). As explained in the batch retrieval steps above, Retry-Nodes do not query by timestamp; they query by the Murmur3 token hash of the transaction ID:

SELECT * FROM es_irt_1_{hash}
WHERE token(transaction_id) >= :start_token AND token(transaction_id) <= :end_token
LIMIT 100;

Cassandra returns matching rows ordered by the numerical value of token(transaction_id), not by the timestamp when the transaction was paused or created.

Example: Out-of-Chronological-Order Retry Execution

Consider two transactions that encountered transient errors within the same schedule window (both saved into es_irt_1_{hash}):

  • tx-1: Paused at 00:02:10 → token(tx-1) = 850

  • tx-2: Paused at 00:02:40 → token(tx-2) = 150

When the Retry-Node queries token range [1, 1000]:

  1. Because token(tx-2) (150) is smaller than token(tx-1) (850), Cassandra returns tx-2 before tx-1.

  2. As a result, tx-2 is fetched and retried before tx-1, even though tx-1 failed 30 seconds earlier.

This reordering occurs only within the same scheduled time window (i.e. transactions co-located in the same table slot). Transactions assigned to different table slots (for example, es_irt_1 at 00:00 vs. es_irt_2 at 00:02) are strictly processed in their respective schedule windows.
Architectural Guidance: When to Pair Cassandra with an SQL Event Store

In the vast majority of microservice architectures, transient errors (such as network blips or temporary resource limits) resolve within seconds or minutes, making minor non-FIFO execution variations within a 1-to-2 minute window completely acceptable.

However, if your business domain contains strict sequential requirements (for example, financial order books or sequential inventory allocation where tx-1 must strictly precede tx-2 under all circumstances):

  • Default Rule: StackSaga strongly recommends keeping the event-store technology aligned with the system’s primary database to avoid the operational overhead of managing multiple database engines.

  • The FIFO Exception: If your application uses Cassandra as its primary database but strictly requires exact FIFO retry ordering, deploy StackSaga with one of the SQL database support modules for the event-store. This hybrid pattern provides the best of both worlds: Cassandra delivers massive write throughput for your primary domain operations, while the SQL event-store guarantees strict chronological FIFO ordering for saga retries and recovery.

es_irt_* tables (Immediate Retry Retention)

In StackSaga, asynchronous transaction retrying is an essential feature that ensures transactions are retried when they fail due to transient errors (such as resource unavailability or temporary network timeouts).

When a transaction fails with a retryable error, it is stored in one of the active es_irt_* tables. At the next scheduled time window, a Retry-Node fetches it by token range, re-invokes it, and deletes the row immediately using the token range batch processing flow described above.

The key difference between immediate retry retention (es_irt_*) and transaction recovery retention (es_rt_*) is the time window: retry retention uses a much shorter delay (minutes) compared to recovery retention (hours), since paused transactions are expected to be retried quickly once the downstream resource becomes available again.

es_irt_* table selection formula

stacksaga diagram stacksaga cassandra how transactions saved for retry

Suppose you configure the Transaction retry retention time to be 2 minutes. The actual time a transaction waits before being retried will naturally oscillate between 1 minute and 3 minutes.

Why does this oscillation happen?

When the retry delay is set to 2 minutes, 3 scheduled rotation slots run in round-robin fashion throughout the day:

  1. 00:00 → es_irt_1_{hash}

  2. 00:02 → es_irt_2_{hash}

  3. 00:04 → es_irt_3_{hash}

  4. 00:06 → es_irt_1_{hash}

  5. 00:08 → es_irt_2_{hash}

  6. and so on…​

The Challenge: Preventing Premature Retries

Imagine a transaction encounters a transient error and pauses at 00:03:58. The next scheduled batch runs at 00:04:00 — just 2 seconds later. If we placed that transaction into es_irt_3_{hash} (the 00:04:00 table), it would be retried after only 2 seconds! Retrying immediately would defeat the purpose of the 2-minute cooldown and risk overwhelming an already struggling downstream service. Every transaction must be guaranteed a fair minimum cooldown period (at least half of the configured delay).

The Solution: The Midpoint Boundary Rule

To solve this, StackSaga divides each 2-minute scheduling interval in half using the midpoint as a boundary point:

  • For the active interval between 00:02:00 and 00:04:00, the configured duration is 2 minutes, making 00:03:00 the midpoint boundary.

Transaction Placement Rule:

  1. First Half (Before Midpoint — 00:02:00 to 00:03:00):
    The transaction paused early enough in the interval. By the time the next schedule (00:04:00) runs, it will have waited at least 1 minute (the minimum threshold). Therefore, it is placed in the table for the immediate next schedule (es_irt_3_{hash} at 00:04:00).

  2. Second Half (After Midpoint — 00:03:00 to 00:04:00):
    The transaction paused too close to the 00:04:00 schedule (less than 1 minute remaining). To protect it from premature retry, it skips 00:04:00 and is placed into the table for the schedule after next (es_irt_1_{hash} at 00:06:00).

Table 1. Example Scenario: Transactions Pausing Between 00:02:00 and 00:04:00
Transaction Stopped Time Time Remaining until 00:04:00 Position in Interval Assigned Table & Schedule Total Actual Wait Time

T4

00:02:50

1 min 10 sec (>= 1 min)

First half (Before 00:03:00)

es_irt_3_{hash} (Runs at 00:04:00)

1 min 10 sec

T5

00:03:08

52 sec (< 1 min)

Second half (After 00:03:00)

es_irt_1_{hash} (Runs at 00:06:00)

2 min 52 sec

T6

00:03:58

2 sec (< 1 min)

Second half (After 00:03:00)

es_irt_1_{hash} (Runs at 00:06:00)

2 min 02 sec

Why Wait Time Fluctuates Between 1 and 3 Minutes:

  • Minimum wait (1 minute): A transaction stopping exactly at the midpoint (00:03:00) waits 1 minute until execution at 00:04:00.

  • Maximum wait (nearly 3 minutes): A transaction stopping just after the midpoint (00:03:01) waits 59 seconds until 00:04:00, plus the full 2-minute cycle until 00:06:00 (total 2 minutes 59 seconds).

es_rt_* tables (Transaction Recovery Retention)

While es_irt_* tables track transactions paused due to transient errors, the es_rt_* tables track crashed or missing transactions — scenarios where an orchestrator node died unexpectedly mid-execution (e.g., node crash, hardware failure, or unhandled network partition).

When a transaction is initialized, its data is stored in one of the active es_rt_* tables. If the transaction completes its full saga journey successfully, its row is deleted from the table. However, if a transaction fails to complete due to an unexpected crash, its row persists in the es_rt_* table, making it a candidate for recovery.

Identifying Missing Transactions

Consider a scenario where 1,000 transactions are executed within a specified time period across multiple nodes. Ideally, all transactions that complete their journey are deleted from the es_rt_* table. However, if even one transaction fails to complete — due to a node crash, power failure, or unexpected interruption — its row persists in the table. When the scheduled time window arrives, a Retry-Node fetches that row (along with any others in its token range) from the respective es_rt_* table, re-invokes the transaction from the last successfully completed span, and then deletes the row so it is not picked up again.

es_rt_* table selection formula

stacksaga diagram stacksaga cassandra how transactions saved for recovery

Suppose you configure the Transaction recovery retention time to be 8 hours (480 minutes). The actual time a transaction is allowed to run before being considered missing will naturally oscillate between 4 hours and 12 hours.

Why does this oscillation happen?

When recovery retention is configured as 8 hours, 3 schedulers run in round-robin fashion throughout the day:

  1. 1st schedule at 00:00 → es_rt_1_{hash}

  2. 2nd schedule at 08:00 → es_rt_2_{hash}

  3. 3rd schedule at 16:00 → es_rt_3_{hash}

The Challenge: Preventing False Positives for Healthy In-Flight Transactions

Imagine a transaction is initialized at 15:59:00. The next recovery scheduler triggers at 16:00:00 — just 1 minute later. If we placed that transaction into es_rt_3_{hash} (the 16:00:00 table), the recovery scheduler would find it still in the table at 16:00, assume it crashed, and trigger a duplicate recovery attempt — even though the transaction is perfectly healthy and only started 1 minute ago! Every transaction must be granted at least half of the configured retention window (4 hours) as a minimum completion buffer before it can be flagged as missing.

The Solution: The Midpoint Boundary Rule

To prevent false alarms, StackSaga divides each 8-hour scheduling window in half using the midpoint as a boundary point:

  • For the active interval between 08:00:00 and 16:00:00, the configured duration is 8 hours, making 12:00:00 the midpoint boundary.

Transaction Placement Rule:

  1. First Half (Before Midpoint — 08:00:00 to 12:00:00):
    The transaction was initialized early in the window. By the time the next recovery schedule (16:00:00) arrives, the transaction will have been running for at least 4 hours (the minimum threshold). Therefore, it is placed into the table for the immediate next schedule (es_rt_3_{hash} at 16:00:00).

  2. Second Half (After Midpoint — 12:00:00 to 16:00:00):
    The transaction was initialized after the midpoint (less than 4 hours before the 16:00 schedule). To give it adequate time to complete naturally, it skips the 16:00 schedule and is placed into the table for the schedule after next (es_rt_1_{hash} at 00:00:00).

Table 2. Example Scenario: Transactions Initialized Between 08:00:00 and 16:00:00
Transaction Initialization Time Time Remaining until 16:00:00 Position in Interval Assigned Table & Schedule Total Grace Period Before Recovery

T4

11:30

4 hours 30 min (>= 4h)

First half (Before 12:00)

es_rt_3_{hash} (Runs at 16:00:00)

4 hours 30 min

T5

12:30

3 hours 30 min (< 4h)

Second half (After 12:00)

es_rt_1_{hash} (Runs at 00:00:00)

11 hours 30 min

T6

15:59

1 minute (< 4h)

Second half (After 12:00)

es_rt_1_{hash} (Runs at 00:00:00)

8 hours 01 min

Why Grace Period Fluctuates Between 4 and 12 Hours:

  • Minimum grace period (4 hours): A transaction initialized exactly at the midpoint (12:00:00) is checked at 16:00:00 (4 hours elapsed).

  • Maximum grace period (nearly 12 hours): A transaction initialized just after the midpoint (12:00:01) waits 3 hours 59 minutes until 16:00, plus the full 8-hour cycle until 00:00 (total 11 hours 59 minutes).

Handling False Positives in Transaction Recovery & the Role of Idempotency

While transactions that remain in the es_rt_* table are generally considered missing, this is not always the case.

Scenario: Transactions Delayed but Not Missing

Consider a situation where a transaction is still in the queue, waiting for execution because the respective orchestrator service is too busy. In this case:

  1. The system mistakenly assumes the transaction is missing since it has not been removed from the es_rt_* table within the expected time frame.

  2. As a result, the system triggers a recovery process, re-invoking the transaction.

  3. This can lead to the transaction (or certain atomic executions) being executed multiple times, causing unintended duplicate operations.

This is one of the possible ways transactions can be executed multiple times. To prevent these kinds of unintended duplicate executions, idempotency should be implemented at the atomic execution level of the transaction.
Time Window Manager Visualizer

⏱ Time Window Manager Visualizer

—
Input Time
—
Rest in Window
—
Effective Win Index
—
★ Effective Window Label
—
Windows Within Day
—
⏳ Exposed Within
📅 Full Day Timeline  (00:00:00 → 24:00:00)
🔍 Zoomed View  (around current time ± context windows)

Write Protection & Duplicate Prevention (Kafka Event Sourcing)

When using Cassandra reactive support with event-driven streaming (such as Apache Kafka), guarding against duplicate execution and concurrent race conditions is essential.

Why Execution Markers are Needed

StackSaga’s Kafka implementation uses at-least-once delivery semantics. Because of this, duplicate event deliveries are naturally expected in distributed environments,
for example:

  • Consumer group rebalances when pods scale up or down.

  • Consumer node restarts or crashes before offset commits are acknowledged.

  • Network timeouts and automated message retries.

To ensure idempotency, StackSaga saves an execution marker in the es_execution_markers table in Cassandra. Before or during processing, the framework inspects this table so that already-processed events are detected and not executed again.

Database Safety vs. Event Listener Side Effects (The Real Issue with Races)

Technically, a concurrent race condition does not corrupt the Cassandra database itself because Cassandra write queries are idempotent. Writing the same event-store record twice will not break the database state.

However, the real issue is what happens after the database write:

StackSaga publishes state-changed events to registered event listeners after successfully updating the event store:

  • org.stacksaga.api.listener.KafkaTransactionEventListener

  • org.stacksaga.api.listener.ReactiveKafkaTransactionEventListener (for Kafka event streams)

If a concurrent race condition causes the same record to be inserted multiple times, these state-changed listeners will also be invoked that same number of times.

If the listener triggers external side effects — such as sending a confirmation email or SMS notification to a customer, calling a third-party payment gateway, or dispatching an external API call — the customer would receive duplicate notifications unless the developer explicitly built custom idempotency into their listener code.

Choosing Between Consistency Modes & Overhead Trade-offs

StackSaga provides two write-protection modes to balance strictness and throughput:

1. STRICT Mode

  • When to use: Recommended when you must prevent duplicate listener calls (such as customer-facing notifications or non-idempotent third-party APIs) and you do not want to build custom idempotency logic into every listener.

  • How it works:

    1. LWT Lease Acquisition: Before running the execution, a worker acquires an exclusive temporary lock in Cassandra using Lightweight Transactions (LWT: INSERT INTO es_execution_markers …​ IF NOT EXISTS USING TTL <leaseDuration>).

    2. Active Lease Wait: If another worker is already processing the same record (marker exists with TTL > 0), the second worker waits for the remaining TTL duration and retries.

    3. Permanent Marker Commitment: When the first worker successfully commits its batch, it writes a permanent marker (TTL = 0). When concurrent workers retry, they see TTL = 0 and terminate cleanly with OptimisticLockConflictException.

    4. Automatic Crash Recovery: If the first worker crashes mid-flight, the temporary lease expires automatically when its TTL finishes, permitting another worker to retry and safely complete the saga step without deadlocks.

stacksaga-cassandra-write-protection-flow
Figure 2. STRICT Mode Execution Flow
STRICT Mode Lifecycle Breakdown

  • Claim the Lease (INSERT …​ IF NOT EXISTS USING TTL): Before executing business logic, the worker attempts an atomic conditional insert with a temporary lease TTL (default-lease-duration or domain override). If another node crashes mid-flight, this lease automatically expires after the TTL window, preventing permanent distributed deadlocks without manual intervention.

  • Evaluate Acquisition ([applied] == true):

    • Lease Won (applied == true): The worker exclusively owns the step. It executes the saga step and persists the event store batch along with a permanent execution marker (TTL = 0). Once persisted, registered transaction event listeners are safely dispatched.

    • Collision Detected (applied == false): A marker already exists. The worker queries the record to check whether processing is still active or already completed.

  • Collision Inspection & Safe Deduplication:

    • Lease Active (TTL > 0): Another worker is currently executing the step, or a crashed worker’s lease has not expired yet. The waiting worker backs off for the remaining lease duration and retries.

    • Permanently Finalized (TTL == 0): The step was already completed by another worker. The current message is a duplicate Kafka delivery and is safely discarded without re-executing logic or triggering duplicate listener events.

2. RELAXED_WITH_DEDUP Mode

  • When to use: Recommended when duplicate listener calls are harmless (e.g. idempotent downstream consumers, cache invalidations, simple loggers), OR when you already handle idempotency inside your listener logic.

  • How it works: Before running, the framework checks if a completed execution marker exists (checkExists). If found, it skips execution. This avoids Cassandra LWT locks, maximizing throughput, but does not prevent the rare race condition where two workers execute at the exact same millisecond.

stacksaga-cassandra-write-protection-relaxed-flow
Figure 3. RELAXED_WITH_DEDUP Mode Execution Flow
RELAXED_WITH_DEDUP Mode Lifecycle Breakdown

  • Fast Duplicate Check (checkExists): Before executing the step, the worker performs a lightweight read against es_execution_markers in Cassandra using standard read consistency (LOCAL_QUORUM). This query does not trigger Cassandra Paxos consensus (LWT), completely avoiding coordination round-trips.

  • Evaluate Existence:

    • Marker Exists (YES): A previous execution has already processed this record. The framework recognizes this as a duplicate Kafka event redelivery, aborts further processing with SpanAlreadyExecutedException, and safely terminates without re-running business logic or triggering duplicate listener notifications.

    • Marker Does Not Exist (NO): The worker proceeds to execute the saga step’s business logic.

  • Atomic Batch Persistence & Listener Dispatch: Upon business logic completion, the worker writes the saga event store state changes and the permanent ExecutionMarker (TTL = 0) together in an atomic logged batch. After the batch write successfully completes, registered transaction event listeners are dispatched.

Overhead Comparison: Managing idempotency manually in application code (such as using Redis distributed locks or custom database tables) incurs roughly the same overhead as having the framework do it, and the framework’s Cassandra LWT lease mechanism is built-in and highly optimized.

However, if duplicate listener invocations do not matter for your business use case, choosing RELAXED_WITH_DEDUP eliminates the LWT coordination overhead completely and delivers maximum write throughput.

Domain-Level Configuration

Write protection can be configured globally and fine-tuned per domain entity. Because different saga workflows have distinct consistency and performance requirements, you can maintain both the consistency mode and the lease TTL duration differently for each saga based on its domain.

For example:

  • Critical Sagas (such as order-saga or payment-saga) that trigger external side effects like payments or customer notifications can be configured with STRICT mode and an appropriate lease duration (e.g., 10s or 5s) to completely prevent duplicate executions.

  • High-Throughput Sagas (such as notification-saga or internal auditing) where operations are naturally idempotent can use RELAXED_WITH_DEDUP to bypass LWT locking overhead and maximize write throughput.

The domain key configured under domains corresponds to the domain entity name defined by @SagaDomainEntity#name() (matched case-insensitively). Any domain that is not explicitly overridden will automatically inherit the global default values (default-mode and default-lease-duration).

Example application.yml
stacksaga:
  cassandra:
    write-protection:
      default-mode: RELAXED_WITH_DEDUP      # Default High-throughput mode
      default-lease-duration: 5s            # Default LWT lease TTL (only for mode=STRICT mode)
      domains:
        order-saga:
          mode: STRICT                      # Strict mutual exclusion for order saga
          lease-duration: 10s               # Custom lease TTL override
        payment-saga:
          mode: STRICT                      # Strict mode using default 5s lease
        notification-saga:
          mode: RELAXED_WITH_DEDUP          # High-throughput mode for notifications

Configuring Properties

Property Name

Default Value

Type

Property Description

stacksaga.cassandra.read-consistency-level

LOCAL_QUORUM

SagaSupportConsistencyLevel[QUORUM,LOCAL_QUORUM]

Consistency level for read operations. Default is LOCAL_QUORUM, which provides a good balance between consistency. The consistency level that provided via stacksaga-cassandra.conf will override this value internally by the framework. Stacksaga only supports QUORUM and LOCAL_QUORUM consistency levels for read.

stacksaga.cassandra.write-consistency-level

LOCAL_QUORUM

SagaSupportConsistencyLevel[QUORUM,LOCAL_QUORUM]

Consistency level for write operations. Default is LOCAL_QUORUM, which provides a good balance between consistency and availability. The consistency level that provided via stacksaga-cassandra.conf will override this value internally by the framework. Stacksaga only supports QUORUM and LOCAL_QUORUM consistency levels for write.

stacksaga.cassandra.config

classpath:stacksaga-cassandra.conf

Resource

Location of the configuration file to use. See configuration options for more details about how to provide the .conf file and the available configuration options in the .conf file.

Write Protection Properties: The following properties configure duplicate protection, LWT lease TTL durations, and per-saga consistency modes under stacksaga.cassandra.write-protection.*. See Write Protection & Duplicate Prevention for the conceptual explanation.

stacksaga.cassandra.write-protection.default-mode

RELAXED_WITH_DEDUP

ConsistencyMode [STRICT, RELAXED_WITH_DEDUP]

Default write-protection mode applied to sagas that do not specify a per-domain override. RELAXED_WITH_DEDUP provides fast duplicate detection without LWT locks. STRICT uses Cassandra LWT for linearizable mutual exclusion.

stacksaga.cassandra.write-protection.default-lease-duration

5s

Duration

Default temporary marker lease duration for STRICT mode. Specifies how long an in-flight worker holds an exclusive lock in Cassandra before expiring via TTL, and the retry backoff duration for racing workers.

stacksaga.cassandra.write-protection.domains.<domain-name>.mode

null

ConsistencyMode [STRICT, RELAXED_WITH_DEDUP]

Optional per-domain consistency mode override, keyed by @SagaDomainEntity#name() (case-insensitive).

stacksaga.cassandra.write-protection.domains.<domain-name>.lease-duration

null

Duration

Optional per-domain lease duration override for STRICT mode. If null, falls back to default-lease-duration.

Transaction Properties: The following properties configure transaction lifetime, retry/restore scheduling, and the in-memory streaming buffer used by the recovery pipeline. See Transaction Recovery and Transaction Retry for the conceptual explanation.

stacksaga.cassandra.transaction.lifetime

24h

Duration

How long a transaction is kept as live. After reaching the lifetime the transaction will not be exposed for retrying even if the transaction is still in processing mode.
default: 24 hours

stacksaga.cassandra.transaction.retry.delay-in-minutes

1

int

How long the transaction should be kept waiting before the next retry exposure. This ensures that transactions are not retried too frequently within a short period. The transaction is frozen for some period of time even if it fails again after a recent retry. The value must evenly divide 1440 (24*60). The delay time can fluctuate within delayInMinutes to delayInMinutes+(delayInMinutes/2). For example, if delayInMinutes is 2, the transaction can be exposed for retrying after 2 to 3 minutes.
read more
default: two minutes

stacksaga.cassandra.transaction.restore.delay-in-minutes

480

int

How long before a transaction is considered crashed and exposed for recovery. In rare cases, transactions may crash without acknowledging the failure (e.g. application crash). This value determines after how long the transaction is considered crashed and exposed for restoring and retrying. The value must evenly divide 24. The delay time can fluctuate within delayInMinutes to delayInMinutes+(delayInMinutes/2). For example, if delayInMinutes is 540, the transaction can be exposed for restoring after 540 to 810 minutes.
read more
default: 480 minutes (8 hours)

stacksaga.cassandra.transaction.stream.batch-size

100

int

Controls how many transactions are fetched from the database at once into the in-memory buffer. This maps to the highTide parameter of Reactor’s limitRate(). A higher value reduces the number of database round trips but increases memory usage. A lower value keeps memory footprint small but increases database pressure.
default: 100

stacksaga.cassandra.transaction.stream.refill-threshold

75

int

Controls when the framework triggers the next database fetch to refill the buffer. This maps to the lowTide parameter of Reactor’s limitRate(). When the number of buffered transactions drops to this value, a new fetch is triggered in the background — so processing never stalls waiting for data. Should always be less than batch-size. A common rule of thumb is 75% of batch-size.
default: 75

stacksaga.cassandra.transaction.stream.concurrency

100

int

Controls how many transactions are processed simultaneously in the recovery pipeline. This maps to the concurrency parameter of Reactor’s flatMap(concurrency). Each concurrent slot handles one transaction’s full retry/restore pipeline, which may involve multiple database calls per transaction. Therefore, the concurrency value should always be less than or equal to the dedicated recovery connection pool size to avoid connection starvation and pipeline stalls.
Recommended: Set concurrency to 50-70% of the recovery connection pool size when the recovery pipeline performs multiple database operations per transaction. If the recovery datasource is isolated (dedicated pool), concurrency can be pushed closer to pool-size - 2 to reserve connections for background fetch operations.
Warning: Setting concurrency equal to or greater than the connection pool size risks starving the background limitRate refill fetch query, which silently stalls the entire recovery pipeline.

stacksaga.cassandra.transaction.stream.pre-fetch

70

int

Controls how many transactions are pre-fetched and buffered in memory before processing begins. This maps to the prefetch parameter of Reactor’s flatMap(concurrency, prefetch). Prefetching allows the framework to have a ready supply of transactions to process without waiting for the completion of the previous batch, thus improving throughput and reducing latency.