Aggregator Versioning & Aggregator Casting

Aggregator Versioning

Aggregator versioning is the most important thing in StackSaga. All the applications are being updated with new features time to time. Any kind of changes that you make regarding the entire transaction, it caused for a version update of the particular aggregator.

Why it’s necessary to update the aggregator version?

You already know that Stacksaga does support event sourcing and event retrying to overcome the eventual consistency. To identify the event’s version is most important when the event is retried. Because all the time the aggregator state that comes to the executor is not equal to the current version of the aggregator. Therefore, You should know sometimes what is the aggregator version that transaction has been initialized.

For instance, just imagine that a transaction is initiated with aggregator version 1.0.0 and the transaction is failed while executing due to a network issue. Then the transaction is temporally stopped for a while until the scheduler is triggered. just think a new version will be

What is the event’s version?

The first event of the aggregator is saved with the version of the aggregator when the transaction is initialized. It is called as event’s version. That means what was the aggregator version that was when the transaction is initialized.

For instance, let’s have a look at the placing order example.

The aggregator version can be updated due to two main reasons. Changing the aggregator structure, Changing the executor structure

Not only the aggregators, even if you make some changes in one of the executors, It also causes a version update. Because every change you make related to the aggregator, it will change the aggregator state. If the aggregator-state is changed, the version must be updated.

As an example, if you have PlaceOrderAggregator, the following changes can be caused to a version update.

When the aggregator version should be updated

Aggregator Structure Changing
- Adding new data
- Removing existing data
Executor Changing
- Changing something inside the existing executor
- Changing Executors count
  - Adding new executor(s)
  - Removing existing executor(s)

Aggregator Structure Changing

If the aggregator’s data set is changed, the aggregator version should be updated.

Changing the Structure is the reason for the event-casting.

Adding new data to the aggregator

If some new data is added to the aggregator object, the aggregator version should be updated. Changing the structure of the aggregator by adding new data is the reason for event-upcasting

stacksaga diagram aggregator change add new data

Removing existing data from the aggregator

If some existing data is removed from the aggregator object, the aggregator version should be updated. Changing the structure of the aggregator by removing the existing data is the reason for event-down-casting

stacksaga diagram aggregator change remove data

Executor Changing

Without changing the aggregator structure, if it makes some changes related to the executors, at this time also the aggregator version should be updated. It can be happened both changing the count of executors or changing something in existing executor(s).

It is possible to update an existing query-executor to command-executor. But it is not possible to update command-executor to query-executor.
Even it is possible to update query-executor to command-executor, The StackSaga team does not recommend changing the existing executor type.

Changing something inside the existing executor

If you make some changes inside the existing executor, It also can be a cause for a version update. It depends based on your requirement. If the new update should not be impacted on the old events that are remains to be executed, you have to change the aggregator version. Then you can filter the old event from the new events. If the changes that you made are for all events, there is a possibility to ignore the updating.

The following diagram shows some change has been done inside the command-executor 3 than the old one.

stacksaga diagram executor change

Changing Executors count

If the executor count is changed for an aggregator, it should be a cause for an aggregator version update.

Adding new executor(s)

If some new executor(s) are added to the entire transaction, it should be a cause for an aggregator version update. It can be any executor either command-executor or query-executor or any sub-executor.

The following diagram shows that a new command-executor has been added called command-executor-4.

stacksaga diagram executor adding

Removing existing executor(s)

If some executor is removed from the entire transaction flow, it should be a cause for an aggregator version update. Even though it can be done theoretically, there is something important that you should know when removing existing executor(s).

Removing existing executor is one of the most important. Because it should be done carefully. The reason is that all the executors are exposed to event-retrying. If there are some old events related to the executor that you are going to remove, the old events can not be run without that executor. Therefore, it is recommended to keep the executor as it is even though if it’s not used for the new version.

The following diagram shows that the executor called revert-before-executor-2.1 has been removed which was in the old version.

stacksaga diagram executor removing

Aggregator Version Casting

All the applications will be updated by changing their versions from time to time. When a new version is deployed, the old version will be replaced by the new version (if you are not going to use service-mesh). But in the event-based architecture, some events can have been waiting to be executed when the new version is being deployed or after deployed as well.

Then the old event should be mapped with the new version. That means by using the old serialized aggregator objects that are in the event-store should be able to create a new aggregator object.

So, if you don’t consider the old version’s events that might be in the event-store, when developing the new version, the events will be conflicted or crashed while mapping the old event to the new aggregator version. Because the old version’s events are going to be executed by using the new aggregator.

Event rebuilding by the engine for transaction retrying.

stacksaga diagram event rebuilding

Aggregator-Oriented casting

If some changes are done for the aggregator, it will cause for a version-update. Even though the version is updated, there is something to be considered carefully. After updating the aggregator’s version, it is not a problem for that particular version at all. But StackSaga follows the event sourcing architecture and event re-invoking mechanism, in the event-store can have some events from the old version of the aggregator to be re-invoked. Therefore, the SEC has to cast (map/convert/deserialize) the old aggregator event to the new version. It is called as Aggregator-Oriented casting.

Even thought the term is quite simple at first glance, It can be effected to get crashed all the old events if you are unable to manage the version mapping. The casting process is failed for the old version’s events, there is no chance to be re-invoked that retry process with the old versions.

Even though the casting process is done by the SEC, as the developer, you are responsible for managing the casting the new version with the old versions' events.

Based on the behavior of the aggregator state-change, the Aggregator-Oriented version casting can be divided to two.

Up-Casting
Down-Casting

Aggregator-Oriented Up-Casting

If the new aggregator has more new attributes than the old one, that kind of event should be cast with upcasting.

For instance, the old version’s aggregator has 3 fields, and the new version can have 4 fields or more than that, the old version’s events should be cast to the new aggregator object, and that term is called as upcasting.

The following image shows how it is done by the SEC.

Stacksaga aggregator version up-casting

Explanation: In the event-store can have old remaining events to be re-tried. But due to the fact that the aggregator has been updated to a new version with new attributes, The old events also should be converted and should be run through the new aggregator. But if it is an upcast mapping, The new version doesn’t bother to the casting process at all due that SEC uses jackson objectmapper to serialization and deserialization both. The new values will have the default values with help of jackson objectmapper and you have nothing to worry about upcasting with StackSaga at all. See the <<,technical documentation>>

All the events that go through the SEO process can be identified easily in the Executor’s methods with the help of event-version data that provides with the aggregator object. See the [technical documentation]

Aggregator-Oriented Down-Casting

Stacksaga aggregator version down-casting

According to the example, the event-store might have the old events consisting of 2 fields. And if the new version has a less number of fields than the old one, it is called as down-casting. Now those remaining events are going to be executed through the new version. And the framework tries to build the new aggregator object by using the old event’s binaries. This is the time the casting is been worked on. You have to modify and annotate the aggregator so as not to conflict while building the object by the framework. Here you can see the best practices related to the casting of aggregators.

Aggregator’s Event Casting

As per the architecture documentation, StackSags does support Aggregator Event UpCasting and also Aggregator Event DownCasting.
For the upcasting of the Aggregator Event, there is nothing to be done. The only thing is adding the new properties. But the down-casting process is a little more complicated.

To overcome Aggregator’s Event down-casting in StackSaga, it can be followed in two different approaches technically.

Ignore Unknown JSON Properties.
Collect Missing properties

Ignore Unknown JSON Properties Is not a recommended approach. Technically, it is possible to ignore the missing properties in Object Mapper by using DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES globally or using @JsonIgnoreProperties(ignoreUnknown = true). Annotation in class level. But it is not recommended in StackSaga. Because old events have more data that the new one when it provided down-casting changes in your aggregator. Then, while the old events are processed, that removed data will not be available for accessing.

Collect Missing properties

For collecting missing properties it can be used @JsonAnySetter and @JsonAnyGetter with a custom method. But you don’t need to create all the things from scratch. Because StackSaga provides the helper class called MissingJsonPropertyCollector for extending without writing any custom methods.

The root implementation (Custom Aggregator) of the Aggregator is by default extended from MissingJsonPropertyCollector. Therefore, the root Custom Aggregator class is ready to collect the missing properties. But if you want to empower other inner classes that are used inside the Root Aggregator class also with Missing properties collector capabilities, you can extend all the inner classes from MissingJsonPropertyCollector as well.

Here is an example for down-casting with using MissingJsonPropertyCollector implementation.

Old Version Of the Custom Aggregator.

@SagaAggregator(
        version = @SagaAggregatorVersion(major = 1, minor = 1, patch = 0),
        idPrefix = "po",
        name = "PlaceOrderAggregator",
        sagaSerializable = PlaceOrderAggregatorSample.class,
        mapper = PlaceOrderAggregatorJsonMapper.class
)
@Getter
@Setter
public class PlaceOrderAggregator extends Aggregator {

    public PlaceOrderAggregator() {
        super(PlaceOrderAggregator.class);
    }

    @JsonProperty("order_id")
    private String orderId;

    @JsonProperty("username")
    private String username;

    @JsonProperty("total")
    private Double total;

    @JsonProperty("is_active")
    private Integer isActive;

    @JsonProperty("comment")
    private String comment;

    @JsonProperty("item_details")
    private List<ItemDetail> itemDetails = new ArrayList<>();

    @Getter
    @Setter
    @AllArgsConstructor
    @NoArgsConstructor
    public static class ItemDetail implements Serializable {

        @JsonProperty("order_id")
        private String itemName;

        @JsonProperty("qty")
        private int qty;

        @JsonProperty("price")
        private double price;

        @JsonProperty("note")
        private String note;
    }
}

In this PlaceOrderAggregator class you can see some properties in the root class, and also there is another nested class called ItemDetail and it contains more properties regarding the items.

New Version Of the Custom Aggregator.

@SagaAggregator(
        version = @SagaAggregatorVersion(major = 1, minor = 1, patch = 1),
        idPrefix = "po",
        name = "PlaceOrderAggregator",
        sagaSerializable = PlaceOrderAggregatorSample.class,
        mapper = PlaceOrderAggregatorJsonMapper.class
)
@Getter
@Setter
public class PlaceOrderAggregator extends Aggregator {

    public PlaceOrderAggregator() {
        super(PlaceOrderAggregator.class);
    }

    @JsonProperty("order_id")
    private String orderId;

    @JsonProperty("username")
    private String username;

    @JsonProperty("total")
    private Double total;

    @JsonProperty("is_active")
    private Integer isActive;

    //@JsonProperty("comment") (1)
    //private String comment;

    @JsonProperty("item_details")
    private List<ItemDetail> itemDetails = new ArrayList<>();

    @Getter
    @Setter
    @AllArgsConstructor
    @NoArgsConstructor
    public static class ItemDetail extends MissingJsonPropertyCollector { (3)

        @JsonProperty("order_id")
        private String itemName;

        @JsonProperty("qty")
        private int qty;

        @JsonProperty("price")
        private double price;

        //@JsonProperty("note") (2)
        //private String note;
    }
}

Relatively the old version, some attributes have been removed from the root class and also from the ItemDetail nested class. That means that the old event data should be cast down when it is deserialized into the new aggregator class.

1	The `comment` property has been removed from the root class. But should not be executed from the `MissingJsonPropertyCollector`. Because the root class is already executed from the `MissingJsonPropertyCollector` through the `Aggregator` class.
2	The `note` property has been removed from the `ItemDetail` class.
3	To be collected that missing property (`note`), the `ItemDetail` has been extended from the `MissingJsonPropertyCollector` class. Then the deserialization is happened that missing property will be saved in to the `missingProperties` map in side of teh `MissingJsonPropertyCollector` that has been provided by the framework.

If the ItemDetail has not been extended from the MissingJsonPropertyCollector class, an exception will be thrown by the framework when the application is started by mapping the old version’s samples that you have given in the previous version through the SagaSerializable implementation. It will ensure that the application is in a casting trouble.

Getting The Collected Properties For specific Version.

@SagaExecutor(executeFor = "order-service", liveCheck = true, value = "OrderSaveExecutor")
@AllArgsConstructor
public class OrderSaveExecutor implements CommandExecutor<PlaceOrderAggregator> {

    @Override
    public ProcessStepManager<PlaceOrderAggregator> doProcess(
            ProcessStack processStack,
            PlaceOrderAggregator aggregator,
            ProcessStepManagerUtil<PlaceOrderAggregator> stepManager
    ) throws RetryableExecutorException, NonRetryableExecutorException {

        if (aggregator.getRealVersionAsString().equals("1.0.0")) { (1)
            String comment = aggregator.getMissingProperties().get("comment").toString(); (2)
            System.out.println("comment = " + comment);

            for (PlaceOrderAggregator.ItemDetail itemDetail : aggregator.getItemDetails()) { (3)
                String note = itemDetail.getMissingProperties().get("note").toString(); (3)
                System.out.println("note = " + note);
            }
        }
        ...

        return stepManager.next(UpdateStockExecutor.class);
    }

    @Override
    public void doRevert(
            ProcessStack processStack,
            NonRetryableExecutorException e,
            PlaceOrderAggregator aggregator,
            RevertHintStore revertHintStore
    ) throws RetryableExecutorException {
        ...
    }
}

You already know that you have to use the same aggregator as well as the same executors for invoking the old transactions as well. Although the missing properties should not be need for the new version(1.0.1), If the event is an old transaction from the version of 1.0.0, the missing properties can be required. Therefore, it is necessary to identify the exact version of the execution (Event). To identify the exact version of the current execution (Event), The framework provides the data version data along with the Room Aggregator Object By default.

1	Check the current execution is 1.0.0 or another version by using the version data that provides by the Aggregator.
2	If the version is `1.0.0`, you can get the missing properties from the `aggregator` object by calling `getMissingProperties()` method. That pert is based on the root aggregator object.
3	If the version is `1.0.0`, you can get the missing properties from the `itemDetail` object by calling `getMissingProperties()` method. That pert is based on the root `ItemDetail` object.

It is possible to get the missing properties and the version of the current execution (Event) in every executor like Command-Executor, Query-Executor and Revert-Executor.