Due to the lack of ACID transaction property in Microservices, managing data integrity for transactions across a number of Microservices API is a big challenge. This article shares an approach to tackle this problem.
In monolithic applications, a single database is shared by various application modules. So data integrity is easy to maintain through ACID transaction management.
The reengineering of monolithic applications in the way of Microservices decomposes them into many individual while independent services. Each of the services has its own database. This makes data integrity management tremendously difficult.
There are several approaches to address this problem. One of the approaches is to use the publish and subscribe model in message brokers like Kafka, RibbitMQ or ActiveMQ.
Here we shall discuss how this approach addresses distributed transaction management in Microservices based applications, its advantages and disadvantages.
Read Only Transaction
When API ‘X’ receives a Read request, it pushes synchronized read events into the message queue. Those API that subscribe the events will retrieve the messages from the queue, read the corresponding data from their own database, then push the result into the return queue.
Since this is a synchronized call, API ‘X’ will wait until all return messages are received, then it will return the overall result to its consumers. In case of time out before all return messages are received, it will return an error with partial results to the consumer.
The advantage of the approach is the scalability. By launching more instances of read API, you can maintain the system throughput under a high loading environment. The disadvantage is the complexity and cost of the solution.
CRUD Transaction
CURD (Create, Update, Read, Delete) Transaction is similar to the Read Only Transaction. When API ‘X’ receives a CRUD request, it pushes synchronized or asynchronized CRUD events into a message queue. Those API that subscribe the events will retrieve the messages and execute CRUD transactions in their own database.
For synchronized CRUD Transaction, these downstream API push the result back to API ‘X’ through the return queue. API ‘X’ will wait for all event status until timeout and return the status of success or failure to its consumers.
For asynchronized CRUD Transaction, API ‘X’ will not wait for the completion of the transactions, it will return a completion status to its consumers once it has pushed all the CRUD events into a message queue.
The advantage of this approach is a simple implementation with high scalability. However, its disadvantage is the possibility of data integrity issues due to services failure.
Eventual Consistency
Obviously, the described approach may lead to data integrity issues when any of the CRUD services fail. Hence we need an error recovery process to maintain the Eventual Consistency.
A basic recovery solution is a data reconciliation process running across different databases regularly. The reconciliation process will flag out all the records and fields with data integrity issues.
Another solution is log scanning and analysis. This solution requires a distributed log and monitoring mechanism which has been discussed in our last article. By scanning and analysing the logs, we could identify all the failed transactions.
With the information of failed transactions or records with data integrity issues, we can rectify the data manually or automatically.
The advantage of Eventual Consistency is its simplicity while its disadvantage is that there will be a time window of data inconsistency which is not acceptable by systems that require strong data consistency, such as money transfer.
In the coming articles, we shall discuss how to handle distributed transactions for systems that require strong data consistency. Stay tuned.
Thank you for reading this article. For the details of managing distributed transaction in Microservices, please contact us. If you want to receive more information about finance and technology, please follow our LinkedIn or subscribe to the “FinTech Insights”.
Axisoft is a Top-Notch Financial Technology Provider in Hong Kong, Singapore and China. Since 1998, we have been helping financial institutions implement global banking solutions