Stacksaga-Kafka Framework (Asynchronous)

Overview

Stacksaga-Kafka Framework is a reactive saga orchestration framework built on top of Spring Boot, designed to manage long-running transactions using the saga pattern and Kafka for internal communication.

It offers a robust and scalable solution for handling distributed transactions in microservices architecture, ensuring data consistency and reliability across multiple services. By leveraging Kafka’s event-driven capabilities, Stacksaga-Kafka enables seamless communication between services and simplifies the implementation of the saga pattern, allowing developers to focus on business logic rather than the complexities of distributed transactions.

Key features include:

Support for both reactive and non-reactive programming models, providing flexibility for various use cases.
Built-in compensation actions, timeouts, and retry mechanisms to handle failures and maintain data consistency.
Scalability and high performance, utilizing Kafka’s distributed architecture to process large volumes of transactions with low latency.
Comprehensive monitoring and logging tools to track saga progress and facilitate troubleshooting.
Idempotency management to ensure that repeated or duplicate events are processed safely and consistently, preventing unintended side effects.

Stacksaga-Kafka Framework is optimized for Spring Boot applications, making it easy to integrate and extend within your existing microservices-ecosystem.

Why Stacksaga-Kafka Framework?

In modern microservices architecture, managing distributed transactions can be complex and error-prone. The Stacksaga-Kafka Framework addresses these challenges by providing a robust framework for implementing the saga pattern, which breaks down long-running transactions into smaller, manageable steps.

Centralized Transaction State Management

Kafka

Just delivers messages asynchronously; it doesn’t track the state of a business transaction.

Stacksaga-Kafka Framework

Maintains the full transaction lifecycle:
Started → In Progress → Completed → Failed → Compensating → Compensated

Keeps a single source of truth for the saga’s status, even if events arrive out of order.

Advantage: No need to build complex state management into each microservice; orchestrator handles it.
Resilience and Fault Tolerance

Kafka

Provides message durability and delivery guarantees, but doesn’t handle transaction failures.

Stacksaga-Kafka Framework

Implements retry logic, timeouts, and compensation actions to recover from failures. +Ensures that the overall transaction can be completed or rolled back gracefully.

Advantage: Reduces the risk of data inconsistencies and partial updates across services.
Scalability and Performance

Kafka

Scales well for high-throughput messaging but doesn’t optimize for transaction workflows.

Stacksaga-Kafka Framework

Designed to handle large volumes of transactions efficiently, leveraging Kafka’s distributed architecture. +Optimizes the processing of saga steps to minimize latency and maximize throughput.

Advantage: Can handle complex transaction workflows without becoming a bottleneck.
Simplified Development and Maintenance

Kafka

Requires developers to implement saga logic and state management in each service.

Stacksaga-Kafka Framework

Provides a clear framework and abstractions for defining sagas, steps, and compensation actions.

Reduces boilerplate code and promotes best practices for distributed transactions.

Advantage: Speeds up development time and reduces the likelihood of errors.
Enhanced Monitoring and Observability

Kafka

Offers basic metrics and logging but lacks detailed saga tracking.

Stacksaga-Kafka Framework

Includes built-in monitoring tools to track saga progress, failures, and performance metrics.

Provides insights into transaction workflows and helps identify bottlenecks or issues.

Advantage: Improves operational visibility and aids in troubleshooting.

Components of Stacksaga-Kafka Framework

The Stacksaga-Kafka Framework consists of several key components that work together to facilitate saga orchestration and management.

Saga Orchestrator Service related components:
- Stacksaga kafka starter
- Stacksaga Database-Support
- Stacksaga Environment-Support
Target Service related components
- Stacksaga kafka client
Stacksaga Agent service related components
- stacksaga-agent

The usage of each component as follows,

As mentioned above, the Stacksaga-Kafka Framework consists can be divided into three main parts,
the main service get as the orchestrator service, you have to add the stacksaga-kafka-starter dependency to your Spring Boot application to make it the orchestrator service.
it provides the core functionality for managing and executing sagas using Kafka as the messaging backbone.
and to provide the event sourcing capabilities, you need to include the stacksaga-database-support module. This module allows the orchestrator to persist saga states and events in a database of your choice.

Optionally, the stacksaga-environment-support module helps integrate the orchestrator with various deployment environments, such as Kubernetes or Eureka, ensuring smooth operation in cloud-native setups. it helps to gathering environment-specific metadata like Region, Zone etc. if you don’t configure the module for the relevant environment, the default metadata is used.

after configuring the orchestrator service, you need to add the stacksaga-kafka-client to the target services (utility services) that will participate in the sagas. finally, to enable the asynchronous retry mechanism, you need to deploy the stacksaga-agent service in your infrastructure by adding the stacksaga-agent-$database dependency. it can be chosen based on the database you are using for event sourcing (the database that used for the orchestrator service).

Architecture

The following diagram illustrates the high-level architecture of the Stacksaga-Kafka Framework and how its components interact to manage and execute sagas.

As per the above diagram, as the first step, you have to create an aggregator class. it helps to maintain the state of the saga transaction each time the saga progresses through its steps. it works as the payload of primary executions. that means the aggregator objects is serialized and sent as the message to the target service via Kafka by the framework.
Next, you habe to create an enum for storing the saga steps and the topics of the respective events that should be triggered.
Next, you can configure how your flow should be executed based on the aggregator state by implementing the Saga Event Navigator interface.
Now you can trigger the SEC by using the SagaKafkaTemplate or ReactiveSagaKafkaTemplate. it provides methods to start a new saga transaction and also to send events to the SEC for progressing the saga.
Once the transaction is initiated, the SEC takes over and manages the saga’s lifecycle. it coordinates between different microservices by sending messages to the appropriate Kafka topics based on the configured saga steps one by one. in case of a failure, it triggers the compensating actions in the reverse order of execution.
Each target service listens to its designated Kafka topic using the stacksaga-kafka-client. upon receiving a message, the target service processes the request, performs the necessary business logic, and sends a response back to the SEC via the aggregator root topic that created by the framework.
Throughout the process, the SEC maintains the state of the transaction and persists events to the configured event store for future-retrying and traceability.
If any transaction fails due to temporary issues like network problems or service unavailability, the stacksaga-agent service periodically checks the event store for failed transactions and retries them until they succeed or reach a maximum retry limit.