Mysql Agent

stacksaga-agent-mysql-starter

stacksaga-agent-mysql is one of the Stacksaga Agent implementations that for supporting Mysql based Orchestrator services for retrying the transactions. stacksaga-agent-mysql is a ready to use dependency. you can create your own spring boot project and add the stacksaga-agent-mysql as a dependency and run your application with few configurations.

Adding stacksaga-agent-mysql-starter as a dependency
<dependency>
    <groupId>org.stacksaga</groupId>
    <artifactId>stacksaga-agent-mysql-starter</artifactId>
    <version>${org.stacksaga.version}</version>
</dependency>

After adding the dependency, update the configuration properties of the application as needed.

Transaction retrying with stacksaga-agent-mysql

As mentioned above, the agent acquires a token range for retrying from the entire transaction set in the event store. A scheduler is triggered with the configured time, and the retrying process is started by retreating the transactions from the event store. Retrieving the transactions from the event store is processed batch-wise. For instance, if you have configured batch size as 100 and if there are 1000 transactions to be retried in the event store, the loop is run 10 times sequentially like fetches 10 transactions by updating their retry-retention time and shares that 10 transactions to the available services to process. We discussed the process of how one thread is involved in the process. But the reality is that multiple threads do the same task in parallel in the application. There is a configured thread pool called StackSagaRetryExecutorPool. For instance, if the thread pool’s thread size is 3 as per the diagram, the token range that has been assigned to the respective node is divided into sub-ranges again. The sub-ranges are assigned for each thread. Then they do the same task individually based on their respective token range. After fetching the data from the event store by each thread, the transactions are transferred to another thread pool called StackSagaPublisherExecutorPool. It is responsible for collecting the transactions that each thread collects from the event store.

Let’s have a look at the process step by step.

stacksaga diagram stacksaga service agent mysql how retry is done
While fetching the retryable transactions from the event-store, the expired transactions also are retrieved for re-invoking if the configured expiry time (configured expiry time is set from the Mysql database support) is exceeded.

The steps are as follows:

  1. The scheduler is triggered

  2. Once the scheduler is triggered, the retrying process is started. At this moment, the node knows about their respective token range. And it has been divided again into sub-rangers identical to the configured pool size. As per diagram, the pool size is 3. Just imagine only one instance is running on the given region. Then this node acquires the entire token range -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807. And again, the range is divided into 3 sub-ranges due to the pool size is 3 like below.

    • Thread-1: -9,223,372,036,854,775,808 To -3,074,457,345,618,258,603

      Thread-1 is responsible for fetching the transactions from the event store between the above range.

    • Thread-2: -3,074,457,345,618,258,602 To 3,074,457,345,618,258,602

      Thread-2 is responsible for fetching the transactions from the event store between the above range.

    • Thread-3: 3,074,457,345,618,258,603 To 9,223,372,036,854,775,807

      Thread-3 is responsible for fetching the transactions from the event store between the above range.

      If you deploy 2 agent nodes in the region, the token rage is divided into two like and 3 sub ranges for each like below
      Node Thread Name Start Token End Token

      Node 1

      Thread-1

      -9,223,372,036,854,775,808

      -6,148,914,691,236,517,206

      Node 1

      Thread-2

      -6,148,914,691,236,517,205

      -3,074,457,345,618,258,603

      Node 1

      Thread-3

      -3,074,457,345,618,258,602

      0

      Node 2

      Thread-1

      1

      3,074,457,345,618,258,603

      Node 2

      Thread-2

      3,074,457,345,618,258,604

      6,148,914,691,236,517,206

      Node 2

      Thread-3

      6,148,914,691,236,517,207

      9,223,372,036,854,775,807

  3. Each thread adds all the fetching transactions to the StackSagaPublisherExecutorPool's queue.

  4. StackSagaPublisherExecutorPool's threads will send the transactions to the available services to process them. and finally, each orchestrator service will receive the transactions and execute them.

The Transaction retrying process is the same for both eureka and k8s profiles.

Profiles

Based on your environment, you can choose one of the profiles in stacksaga-agent-mysql-starter. There are two profiles as follows:

  1. eureka - Eureka based environment

  2. k8s - Kubernetes based environment

eureka Profile

If your application is deployed in the Eureka environment, you can use the eureka profile when the agent application is deployed. Under the eureka profile, the agent nodes can be scaled horizontally as per requirement. The stacksaga-agent-mysql-starter can acts as the leader and also as the follower. If there are multiple agent nodes in the region, one node should be deployed as a leader and other nodes should be deployed as the followers. The leader node is responsible for updating the token range based on the running nodes count.

Agent-service as Leader and Follower

A mentioned above after adding the stacksaga-agent-mysql-starter as a dependency, the application should be configured as a leader and als as a follower. Let’s see how the configuration looks like for both.

  • Leader Instance configuration:

    stacksaga.agent.mysql.eureka.instance-type=LEADER (1)
eureka.instance.instance-id=order-service-agent-us-east-leader (2)
    1 Set the instance-type as LEADER to run the node as the leader.
    2 Set a Eureka instance ID as a fixed (Static ID) one.
    It is recommended to use this format for the leader instance ID.
    Format: ${service-name}-agent-${region}-leader
    Using the service name in the leader instance ID helps to avoid the collision if you are using same event-store for multiple services. Because the followers identify the leader instance in the database by the leader instance ID. and adding the region to the leader instance ID guarantees the region-based uniqueness.

  • Follower Instance configuration:

    stacksaga.agent.mysql.eureka.instance-type=FOLLOWER (1)
stacksaga.agent.mysql.eureka.follower.leader-id=order-service-agent-us-east-leader (2)
eureka.instance.instance-id=${spring.application.name}:${random.uuid} (3)
    1 Set the instance-type as the FOLLOWER.
    2 Set the leader’s Static ID.this value should be the same exactly with the leader’s id that we configured in the leader node in the same region.
    3 Set the instance-id as a random ID.

Token range allocation for nodes

All agent applications are registered with the eureka server in eureka environment. So the leader service will have all other agent instances' details through the eureka server. The leader server periodically checks the changes of the instance based on the local eureka service registry cache and updates the database with the relevant token range for each instance. The position of each instance is sored based on the instance started time. For instance, if there are five StackSaga-agent instances in the cluster, the token range is divided with the help of Murmur3 Partition algorithm as follows:

How token range is shared with the available agents in Eureka

Steps:

1 Leader node uses the eureka client’s cache to get the list of all instances in the region. (It can be a single eureka server or peers)
2 Leader node calculates the range for each instance periodically based on their timetamps and updates the ranges is sent to each nodes.

k8s Profile

When Stacksaga agent is deployed in the kubernetes environment, the deployment architecture is a bit different from the eureka environment. In the kubernetes environment, the nodes are deployed as StatefulSet. The reason for using StatefulSet is that the token range of the node is calculated by itself based on the position (index of the node) and the total number of nodes. All nodes continuously monitor changes of respective StatefulSet’s changes in real-time. If one instance goes down or added, all the nodes will be notified the update in real-time and then the token range will be updated accordingly by themselves.

Deploy stacksaga-agent-mysql in kubernetes environment.

First you have to create a user account due to stacksaga-agent-mysql access the kubernetes API in k8s profile. And should create and bind the role with the created service account as follows.

ServiceAccount Manifest
apiVersion: v1
kind: ServiceAccount
metadata:
  name: stacksaga-agent-mysql-service-account #the name of the service account.
  namespace: default #the namespace the application is deployed.
ClusterRole Manifest
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: default
  name: stacksaga-agent-mysql-access
rules:
  # Grant read access to pods
  - apiGroups: [ "" ]
    resources: [ "pods" ]
    verbs: [ "get", "list", "watch" ]
  # Grant access to watch StatefulSets
  - apiGroups: [ "apps" ]
    resources: [ "statefulsets" ]
    verbs: [ "watch", "get", "list" ]
  # Grant access to nodes
  - apiGroups: [ "" ]
    resources: [ "nodes" ]
    verbs: [ "get", "list" ]
ClusterRoleBinding Manifest
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: stacksaga-agent-mysql-access-binding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: stacksaga-agent-mysql-service-account
    namespace: default
roleRef:
  kind: ClusterRole
  name: stacksaga-agent-mysql-access
  apiGroup: rbac.authorization.k8s.io

Create the service-agent StatefulSet to deploy the agent-service.

RoleBinding Manifest
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: your-app
spec:
  serviceName: "your-app"
  replicas: 3
  selector:
    matchLabels:
      app: your-app
  template:
    metadata:
      labels:
        app: your-app
    spec:
      serviceAccountName: stacksaga-agent-mysql-service-account #assign the service-account
      containers:
        - name: your-app-container
          image: your-app-image:latest
          ports:
            - containerPort: 8080
Headless Service Manifest
apiVersion: v1
kind: Service
metadata:
  name: your-app
spec:
  clusterIP: None
  selector:
    app: your-app
  ports:
    - port: 8080
      name: http

StackSaga Mysql-Agent Configuration Properties

Stacksaga MYSQL agent does support both eureka based and kubernetes based environments. There is a list of common configuration properties for both and as well as there are some configuration properties specific to eureka and kubernetes.

Common Configuration Properties

Property Name

Default Value

Type

Description

spring.profiles.active

-

String

There are two profiles available as eureka and k8s. You can choose one of them based on the deployment environment.

spring.application.name

-

String

The name of the agent application.

server.port

8080

int

The port of the agent service.

NOTE: Due to StackSaga Mysql-Agent internally uses spring default datasource and HikariCP for managing the connection pool, It can be configured the datasource properties in the same way that spring boot provides.
The prefix is stacksaga.datasource.

stacksaga.agent.mysql.target-service

-

String

The name of the target service. (The transactions are fetched based on this name from the event-store)

stacksaga.agent.mysql.target-service-host

-

String

The host name of the target service that retry tasks should be submitted to.

stacksaga.agent.mysql.act-leader-as-follower

true

boolean

Whether the leader service acts as the follower or not. If the cluster is small, you can run one instance and run as the leader and as well as the follower.

Communication Thread-Pool: It is responsible for communicating with other services for sharing the transactions to the target orchestrator services.

stacksaga.agent.mysql.thread-pool.communication.core-size

available processors * 5

int

core number of threads in the communication pool.

stacksaga.agent.mysql.thread-pool.communication.max-size

available processors * 50

int

maximum number of threads in the communication pool.

stacksaga.agent.mysql.thread-pool.communication.allow-core-thread-timeout

true

Boolean

Whether core threads are allowed to time out. This enables dynamic growing and shrinking of the communication pool.

stacksaga.agent.mysql.thread-pool.communication.queue-capacity

Integer.MAX_VALUE

int

Queue capacity. An unbounded capacity does not increase the pool and therefore ignores the "max-size" property.

stacksaga.agent.mysql.thread-pool.communication.keep-alive

60 seconds

Duration

Time limit for which threads may remain idle before being terminated.

Retry Thread-Pool: It is responsible for processing the transactions that should be retried and recovered(The expired transactions).

stacksaga.agent.mysql.thread-pool.retry.size

available processors * 5

int

core number of threads in the communication pool.

stacksaga.agent.mysql.thread-pool.retry.allow-core-thread-timeout

true

Boolean

Whether core threads are allowed to time out. This enables dynamic growing and shrinking of the communication pool.

stacksaga.agent.mysql.thread-pool.retry.queue-capacity

Integer.MAX_VALUE

int

Queue capacity. An unbounded capacity does not increase the pool and therefore ignores the "max-size" property.

stacksaga.agent.mysql.thread-pool.retry.keep-alive

60 seconds

Duration

Time limit for which threads may remain idle before being terminated.

stacksaga.agent.mysql.retry.batch-size

1000

int

how much data should be fetched from the database per time for retrying as a bulk.

stacksaga.agent.mysql.retry.delay-in-minutes

2

int

how often the retry task should be executed.

Eureka profile’s Configuration Properties

If you are in the Eureka environment, then you have to configure the following configuration properties.

Property Name

Default Value

Type

Description

stacksaga.agent.eureka.instance-type

-

InstanceType[master,slave]

The type of this agent. Whether this agent is master or slave. in the cluster, it should have at least one master node.

stacksaga.agent.eureka.token-range-update-delay

300_000

if the instance is a master instance, the delay time to update the token range. (Master will send the token range to the slave agent in withing this delay time.)

stacksaga.agent.eureka.token-range-update-initial-delay

300_000

long

if the instance is a master instance, the initial delay for updating the token range by the master.

stacksaga.agent.eureka.token-range-valid-duration

420_000

long

How long time the token range should be valid. This valid time is sent to the slave instance, and its executions are based on this time. The default value is tokenRangeUpdateDelay + 2 minutes = 7 minutes. The extra 5 minutes are added to avoid the network delay. The tokenRangeValidDuration should be greater than tokenRangeUpdateDelay all the time.

NOTE: In addition to the target eureka service details, The following meta-data should be added in the eureka.instance.metadata-map section.

eureka.instance.metadata-map.stacksagaRegion

-

String

The region that the application is deployed.

Kubernetes profile’s Configuration Properties

If you are in the kubernetes environment, then you have to configure the following configuration properties.

Property Name

Default Value

Type

Description

stacksaga.agent.k8s.zone-topology-name

topology.kubernetes.io/zone

String

the topology name of the zone in the kubernetes cluster.

stacksaga.agent.k8s.region-topology-name

topology.kubernetes.io/region

String

the topology name of the region in the kubernetes cluster.

stacksaga.agent.k8s.namespace

default

String

the namespace that the application is deployed in the kubernetes cluster.