How to prevent inconsistency between microservices or Data integrity in distributed systems

In a microservice architecture you can get dependencies that impose restrictions on the services used.

For example, let’s consider a car rental service that uses the database per service principle:

• Geoareas microservice owns data about different geographic polygons — cities, regions. Schema looks like this: {id, geometry}
• Tariffs microservice, in turn, manages the data of the rental price. Schema looks something like this: {id, geoarea_id, price_per_minute}. It has a dependence on the Geoareas microservice.

At some point, we need to delete an object from the Geoareas microservice — call the endpoint:

DELETE /geoarea?id={id}.

This can lead to a number of problems, even including the inaccessibility of the service, since we still have tariffs referencing to a removed geoarea.

An unpleasant dependence appears: in order to remove a geoarea from Geoareas you need to make a request to Tariffs at the time of removal to check which tariff depends on it. (sorry for this)

Dangerous cohesion appears — with this approach, in the future, Geoareas will send requests to all microservices where geoareas are used.

By the way, databases have automatic integrity control using foreign keys, but we have a distributed microservice architecture and there are several solution options:

1. Webhooks

The option implies creating a unified webhook mechanism in the microservice. The microservice will execute these hooks before modifying the object. If at least one hook fails, then action on the object is impossible.

An example of such a webhook would be a request for a third party microservice with a predefined API.
For our example, the webhook in the Geoareas microservice would look like this:

WEBHOOK target=”pre-delete” action=”POST tariffs/check_delete?geoarea_id={id}”

In the Tariffs microservice, respectively, you need to implement endpoint /check_delete to check that the zone can be deleted and the Tariffs microservice “is not against”.

The uniformity of these hooks makes them easy to configure on the fly. When a new microservice that uses Geoareas appears, add a new check to the list of hooks.

Thus, Geoareas supports the mechanism of unified webhooks, but knows nothing about what kind of business logic is behind these hooks.

2. Reference Counting

In this option, we will keep an eye on who uses Geoareas.

Extending the API of our microservice with these new endpoints:

POST /hold?geoarea_id={id}&holder={holder}

POST /release?geoarea_id={id}&holder={holder}

Thus, if we want to create a new tariff in a zone, then before that we need to block this zone in the Geoareas microservice, for example:

POST /hold?geoarea_id=moscow&holder=tariffs_tariffmoscow1

Next to geoareas in the Geoareas microservice, we store a list of those who “hold” these zones.

When trying to delete a geoarea, we check the list of holders and do not allow deleting if the list is not empty.

If the tariff that holds the geoarea is deleted, then after deletion you need to release the lock, for example:

/release?geoarea_id=moscow&holder=tariffs_tariffmoscow1

Other languages: [in Russian]