How To Achieve Bulletproof Microservice Communication

April 24, 2025

Utilizing microservices in order to decouple a backend system into individually scalable and customizable applications is an effective method to build sustainable infrastructure. Pair this with Dockerized services and an orchestration platform such as Kubernetes, and the infrastructure is robust.

With all these benefits, why would anyone ever decide to go in another direction? One of the biggest challenges when developing microservices is data integrity and consistency. Microservices that utilize simple HTTP requests and isolated database transactions are bound to end up in an inconsistent state sooner or later, and I want to discuss how to avoid it.

The Setup

Consider an eCommerce business where you can place orders for various different items, and have them shipped to you around the world. This business might have services for keeping track of orders, managing inventory, and validating shipping requirements. These can be called the Orders, Inventory, and Shipping microservices, respectively. When a user places an order, an Order Entity is entered into the database as a record in a pending state. Afterwards, a message is sent to the Inventory service to check whether there is enough inventory of the particular product. The Inventory service then creates a Reservation Entity, and stores it in the database to keep track of the current status of the warehouse, after which it notifies the Orders service of its success. Then, the Orders service must reach out to the Shipping service to validate that the customer is within the allowed shipping range for the specific product. Only after which, can the Orders service notify the Inventory application to delete the Reservation Entity and corresponding inventory, and set the Order Entity status to approved. All of these steps together form a Saga, a long running, eventually consistent business process.

Queues

There are many different ways that computers can communicate with one another. One of the most popular is the HTTP protocol. With HTTP, failed requests due to networking or service outages are hard to overcome. Exponential Backoff can be used to improve upon standard usage, although it achieves little of what could be done by a message broker. A message broker typically utilizes queues in order to route requests between services. Some of the biggest improvements seen when switching over to using message queues for communication are decoupling of applications and increased durability. A message queue can be thought of like a buffer. It holds messages until the receiver is ready. In the event of a message consumption failure, it can also re-queue messages. Below is a comparison between inter-service communication between two services, the Orders service and Inventory service. Orders is attempting to send a message to Inventory, although Inventory is currently unavailable.

Outbox Pattern

An outbox is a holding station for any outgoing messages that a service needs to eventually send. This is best implemented as a database table, so that sending a message and performing necessary business logic can all occur atomically. To get an idea of how this is structured, we can use the Orders and Inventory services once again. To kick off the process, an HTTP request is routed to the Orders service to place an order. The service starts a database transaction, writes the Order Entity to the database, and then writes a message to the InventoryReservationOutbox table. The service then commits the transaction, so that both the message and the Entity are both written, or neither are. On startup, the Orders service polls the InventoryReservationOutbox table for outgoing messages every couple of seconds. Once this polling callback is triggered, all messages are queried from the outbox table, converted into JSON, and sent into the queue for the Inventory service to process at a later point. Once the message is in the queue, it is safe to be deleted from the InventoryReservationOutbox table. Most queues have configurable durability, meaning if they crash with messages inside them they are able to requeue them after coming back up again. Next, we have to take a look at something known as "at least once" delivery. This is typical of message brokers, meaning messages might be sent to consumers more than once.

Inbox Pattern

The inbox pattern, better known as the idempotent consumer pattern, utilizes a nearly identical approach to the outbox pattern, just in reverse. When using this pattern, incoming messages are written into a database table within a transaction which handles the work associated with the message itself. If the work (for example, creating an Entity and inserting into the database) succeeds, then the message is written to the inbox table. If the work fails, then we can reprocess the message at a later point. When a new message comes in, the inbox table is first checked to determine if an identical message has already been processed to avoid duplicate work. To detect duplicate messages, a unique key must be assigned to each new message. To give a concrete example, the Orders service includes the specific, auto-incrementing primary key for each specific Order Entity to the message it sends to the Inventory service. With this identifier, the Inventory service is able to detect duplicated messages. It receives the message, begins a transaction, reserves inventory, writes the incoming message to the outbox, and a response message to the InventoryReservation outbox table, to later be sent to the Order service to continue on with the Saga.

Compensating Actions

What happens if one of the steps of our Saga fails? For example, after reserving inventory, the Shipping service says that we do not ship this particular product to the requested destination. We currently have InventoryReservation and pending Order entities in our other services, and they need to be removed. To do so, messages can be sent to each respective service to remove the corresponding resources. These messages are better known as compensating actions. For each step in a Saga that persists data or changes state, there must be a compensating action. If any step in the Saga fails, then all previously successful steps must be compensated.

Failed Implementations

Let's talk about the journey to get to the solution described above. For the code examples, I'll be using NodeJS, and NestJS, a backend framework which wraps Express. When sitting down to implement this saga, my first thought was to implement a representation of a saga in a class. It would implement a ISagaOrchestrator interface and have a list of steps, which implemented an ISagaStep interface. Below are the example classes

export interface ISagaOrchestrator {
  begin(): Promise;
}

export interface ISagaStep {
  invoke(): Promise;
  compensate(): Promise;
}

export class OrderSagaOrchestrator implements ISagaOrchestrator {
  // List of steps in the saga
  constructor(private steps: ISagaStep[]) {}
  
  async begin(): Promise {
    const succeededSteps: ISagaStep[] = []
    try {
      for (const step of steps) {
        const response = await step.invoke()
        if (!response.successful) {
          await compensate()
          return
        }
        succeededSteps.push(step)
      }
    } catch (e) {
      console.error(e)
      await compensate()
    }
  }
  
  private async compensate() {
    await Promise.all(succeededSteps.map(async (step) => {
      return await step.compensate()
    }))
  }
}

export class ReserveInventoryStep implements ISagaStep {
  // Example step, constructed with necessary data to send to queue 
  constructor(
    private rabbitMQService: RMQService,
    private productId: number,
    private quantity: number
  ) {}
  
  /* Send reserve inventory message, and wait for response. 
  Do not advance Saga until we get a response */
  invoke(): Promise {
    await this.rabbitMQService.sendReserveInventoryMessage(productId, quantity) 
    return await this.rabbitMQService.listenForReserveInventoryResponse()
  }
  
  compensate(): Promise {
    await this.rabbitMQService.compensateReserveInventoryMessage(productId, quantity)
  }
}
... other steps

export interface ISagaOrchestrator<T> {
  invokeStep(step: T): Promise<void>
  compensateStep(step: T): Promise<void>
 }
 

export interface ISagaStep {
  invoke(): Promise<void>;
  compensate(): Promise<void>;
}

export enum OrderSagaStep {
  INVENTORY_RESERVE,
  SHIPPING_VALIDATION,
  INVENTORY_REMOVE
}

export class OrderSagaOrchestrator implements ISagaOrchestrator<OrderSagaStep> {
  // Map to keep track of steps in the Saga
  constructor(
    private steps: Map<OrderSagaStep, ISagaStep>, 
    private successfulSteps: ISagaStep[]
  ) {}
  
  // Invoke a singular step in the Saga, one at a time
  async invokeStep(step: OrderSagaStep): Promise<void> {
    const response = await steps.get(step).invoke()
    if (!response.successful) {
      await this.compensate()
    }
    successfulSteps.push(step)
  }
  
  // Compensate a singular step in the Saga, one at a time
  async compensateStep(step: OrderSagaStep): Promise<void> {
    const response = await steps.get(step).compensate()
    if (!response.successful) {
      throw new Error('Failed to compensate')
    }
  }
}

export class ReserveInventoryStep implements ISagaStep {
  constructor(
    private rabbitMQService: RMQService,
    private productId: number,
    private quantity: number
  ) {}
  
  invoke(): Promise<void> {
    await this.rabbitMQService.sendReserveInventoryMessage(productId, quantity) 
    // do not await response here!
  }
  
  compensate(): Promise<void> {
    await this.rabbitMQService.compensateReserveInventoryMessage(productId, quantity)
  }
}

export class OrdersController implements ISagaStep {
  constructor(private rabbitmqService: RMQService) {}
  
  // All currently running Sagas
  private activeSagas: Map<number, ISagaOrchestrator>
  
  /* Await all InventoryReserve message responses in a "global" place, 
  and route them to appropriate saga based on orderId */
  public reserveShippingResponseHandler() {
    const response = this.rabbitmqService.getInventoryResponse()
    const relatedSaga = this.runningSagas.get(response.orderId);
    
    if (!relatedSaga) {
      throw new InternalServerErrorException(
        `No running saga is related to the most recent message received!`,
      );
    }
    
    if (response.successful) {
      // The reserve inventory step succeeded, start the next step in the saga
      relatedSaga.invokeStep(OrderSagaStep.SHIPPING_VALIDATION);
    } else {
      toRollback.forEach((step) => relatedSaga.compensateStep(OrderSagaStep.INVENTORY_RESERVE));
    }
  }
}