Distributed Asynchronous Systems

Created

In the early days of monoliths, everything was packaged into a single unit because it was simpler to develop, deploy, and scale vertically by adding more power to a single server.

Then as systems grew in complexity and the needed for scalability became paramount, the monolithic approach started showing cracks in long development time, riskier deployment, and expensive vertical scaling. Microservices broke monoliths into smaller, independent services, each responsible for a specific functionality. This included isolated deployments, fault resilience, and horizontal scaling capabilities.

Challenges in microservices- how should these services communicate?

  • Synchronous communication (one service makes a direct call to another and had to wait for a response) like RESTful APIs had downsides like tight coupling (services needed to be aware of one another), latency (waiting for responses increased), and reliability (if one service failed, it could cascade and affect others).
  • Asynchronous communication: services send events or messages without expecting an immediate response. This reduced direct dependencies between services. A service emits an event when something notable happens, and any other service interested in that event reacts to it. Message queues and event streams handle the transportation and storage of these events.

Distributed asynchronous systems are essential for

  • scalability: as traffic grows, you could add more instances of the service and let the message queue hadle the distribution
  • resilience: if a server goes down, message/events for it are simply queued up, reducing system downtime.
  • flexibility: new services can be added and old ones can be modified without affecting the entire system. They simply needed to subscribe or publish to the relevant events.
  • real-time processing: with event streams, real-time data processing became achievable, opening doors to real-time analytics, monitoring, and more.

In regards to real-time applications:

  • Streaming and scalability: data (or events) can be processed in real-time, enabling immediate insights. Distributed asynchronous systems are well-suited for handling a large number of simultaneous connections and high data throughput

  • Resilience and fault-tolerance: in real-time systems, you don't want a single failure to bring down the whole system. Decoupled nature of event-driven architecture ensures that if one part fails, other can continue, and the failed process can be reprocessed later.

Distributed Asynchronous Systems are a paradigm where services are highly decoupled, single-purpose event processing components that asynchronously receive and process events.

  • characteristics: operate independently, communicate asynchronously
  • focus: enhance service autonomy and handle high throughput efficiently.

Components

Message queues are fundamental tools in asynchronous systems, serving as intermediaries to manage communication between services.

  • role: temporarily store until they are retrieved and processed by the receiving service
  • decoupling: sender and receiver services operate independently, unaware of each other's state
  • benefits:
    • load leveling: absorb variation in the rate of message arrivals
    • load balancing: distribute processing work across multiple resources or services.

A message is a discrete unit of communication intended for transmission between service and components. It contains information about a specific action to be taken or data to be processed. It can be structured as an object with properties detailing its intent and content.

const message = {
  type: 'ORDER_PLACED',
  data: {
    orderId: '12345',
    userId: '67890',
    item: 'Laptop',
    quantity: 1
  }
};

Producers are entities that produce and create message. They don't send messages directly to consumers, but delete that to the message queue. E.g., an application logging user activity events.

Consumers: entities that process or act upon the messages. They retrieve messages from the message queue when ready. E.g., a service analyzing user activity for trends

Brokers: intermediate layers, like message queues, that store messages until consumers retrieve them. They ensure reliable message delivery. E.g., RabbitMQ or Apache Kafka serving as the message storage and delivery mechanism.

Popular solutions and providers:

  • RabbitMQ is an open source message broker software that offers reliable, flexible queuing with a variety of plugins
  • Apache Kafka: distributed streaming platform often used for real-time analytics and monitoring, can handle large volumes of data
  • Amazon SQS: managed queuing service by AWS, ensuring secure and scalable message communication
  • Azure Service Bus: MS's cloud-based messaging system for connecting apps across on-premises and cloud environments
  • Google Cloud Pub/Sub: real-time messaging service from GCP, designed for highly-dynamic, globally-distributed apps.

Event Streams

Event streams provide a continuous flow of events over time, allowing systems to process data in real-time.

  • nature: unlike message queues that store messages temporarily, event streams capture a continuous series of events.
  • purpose: designed to handle a large volume of fast-moving data, offering insights and reactions in real-time.
  • scenarios: real-time analytics, monitoring systems, and applications that require immediate response to events.

An event represents a change in state or an occurrence within a system. In programming, its often used to notify interested parts of the system that something bad has happened, allowing them to react accordingly. In JS, events are prevalent in frontend frameworks where user interactions like clicks or form submissions trigger events.

Role of WebSockets & pub/sub:

  • WebSockets establish a persistent, two-way communication channel where it enables servers to push real-time updates to clients. Common in web applications for live data updates.
  • =Pub/Sub systems allow the publisher to emit events without knowledge of its subscribers. The subscribers listen to the events of interest. This decouples event producers from consumers, enhancing scalability and flexibility.

Popular solutions and providers:

  • Apache Kafka was initially developed at LinkedIn and it is a distributed streaming platform known from high-throughput, built-in partitioning, and replication.
  • Amazon Kinesis is AWS's managed service for processing real-time data streams. Offers capabilities like stream analytics, data lakes and log and event data collection.
  • Azure Stream Analytics: MS's event streaming service, integrated with other Azure services. Provides real-time analytics and complex event processing
  • Google Cloud Dataflow, a fully managed stream and batch data processing service from GCP for reliable, scalable, and fast data analytics
  • Apache Flink: open-source stream processing framework that offers high-throughput, low-latency, and exactly-once processing semantics.

Apache Kafka is a scalable, distributed streaming platform that excels in message queuing and event streaming use cases, with real-time capabilities. As a message queue:

  • producers send messages to topics, consumers pull message in the order they arrive.
  • There can be multiple consumers, each tracking its position.

As an event stream:

  • handle continuous flows of data in real time
  • retains data for replay, enabling event sourcing scenarios
  • pair with Kafka Streams for real-time data processing

Apache Kafka in Node.js:

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka-broker:9092']
});

const producer = kafka.producer();

const run = async () => {
  await producer.connect();
  await producer.send({
    topic: 'user-registration',
    messages: [
      { value: JSON.stringify({ type: 'USER_REGISTERED', data: { userId: '123', name: 'Alice' } }) }
    ],
  });

  await producer.disconnect();
};

run().catch(console.error);

A message is a generic term for a discrete unit of data that is sent from one place to another. Messages can represent commands, queries, replies, or even events. A message is more about the act of sending and receiving data.

An event is a special kind of message that signifies a change in state. An event typically follows the "publish-subscribe" model, where the sender (or publisher) doesn't direct the event to a specific recipient. Instead, any number of recipients (subscribers) can react to the event. An event is more about signaling that something has happened.

Message vs event in Kafka, which is a log/event streaming platform that retains events over time and the consumers can read events from any point in the this log.

  • when you send data to Kafka, you're producing a message to a topic
  • while you're working with events, the act of sending or receiving these events in Kafka is often referred to as producing or consuming messages.
  • while Kafka is used for event streaming, the mechanics of how it handles data involves sending and receiving messages to/from topics.

Event-Driven Architecture (EDA) is an architectural approach where the design of the system is centered around producing, detecting, consuming, and reacting to events.

  • core principle: systems react to event rather than driving them
  • decoupling: producers and consumers of events operate independently.
  • flexibility: easy to add, modify, or remove system components without major disruptions.
  • real-time responsiveness: systems can respond immediately to changes or events as they occur.

EDA topologies to structure event interactions have two options:

  • mediator topology: centralized unit (the mediator) for event orchestration.
    • services emit events to the mediator, which then triggers other services
    • provides centralized control, but can be a potential bottleneck.
    • less complexity because central hub for event routing simplifies interactions.
  • broker topology: events are directly communicated between producers and consumers without a central orchestrator; services handle their own event logic
    • highly decentralized (more flexibility and autonomy), scalable (systems can more easily handle increased event traffic), but demands more coordination between services.
    • high decoupling means that services can be added, removed, or updated with minimal impact on the overall system.

The right topology is chosen after considering:

  • system needs: does the system benefit from centralized control or decentralized flexibility
  • scalability requirements: will the centralized mediator become a bottleneck as traffic grows?
  • desired level of decoupling: does your system need a high level of decoupling where services operate more autonomously?

Linked from Software System Design