Event Sourcing Design Pattern

Malware News
1

This blog will discuss the Event Sourcing design pattern and how this can be used in designing large-scale distributed systems.

Photo credit: Dirk Hoekstra

Conventional Design

The conventional thought process in an application’s design is to translate the business requirements into database entities which you can Store, Retrieve and potentially Change later (CRUD operations).

This means that the application maintains only the final state of these entities at any point in time. This approach has a few problems.

Problems

1. Single Data model is used for both read and write operations. Typically you design these models from a write perspective and create read queries on top of them. This may work well for small applications. However, this will become problematic for larger applications where the underlying domain is prone to change. Soon the queries will become very expensive to change or maintain. It is almost impossible to have a generic model optimized for both reads and writes.

2. Unless an independent auditing mechanism records all the state transitions, history is lost. Systems designed using the CRUD approach always provide the current state of the entities and have no clue How? you arrived at that state. For example, in any user management application, if a customer changes the email address n times, the application will store and retrieve only the final email address.

CQRS

Command Query Responsibility Segregation (CQRS) is an architectural pattern which separates the “Read” and “Write” operations of an application into different data models.

Command — Modifies data (e.g. Insert, Update & Delete Operations)
Query — Never modifies data (e.g. Read Operation)

Event Sourcing

In a system based on event sourcing, the primary source of truth is the stream of events. All states are derived from this stream. This includes the “current” state of entities, which is often the basis on which the system’s business logic operates.

An event is something that has already happened. As such, it cannot be modified and is immutable.

Projection

Projections in Event Sourcing is a technique to derive the current state from an event stream. This is done asynchronously by applying the events on a particular “View” in the same chronological order they happened.

A view is stateless and has no side effects — it does not change the original data(events) in any way. Multiple and independent views can be created from the same events stream.

Example

Let us say there is a shopping cart application in an eCommerce company.
In the usual CRUD-based design, the shopping cart entity will have the following fields.

status, user-id, list of items, shipping address, and other metadata

At any point in time, a particular shopping cart will only store the “current state” of the shopping cart.

However, in an application designed using event sourcing principles, the focus will be on storing all the events which happened instead eg.

“shopping cart created”, “item 1 added”, “item 2 added”, “item 1 removed”, “shipping address added”, “shopping cart checked out”

These events are then used to create any needed fact or state by using the projection technique. One of them could very well be the structure in the CRUD-based design which can power the read queries. In other words, is a derivative of these events.

CQRS + Event Sourcing

These techniques are frequently paired together. In a system designed using Event Sourcing design, CQRS is most likely used.
Since the events, the source of truth (or the Writes portion of the system) are stored separately from Views (Query portion of the system)

Benefits

  1. The flexibility of creating transient views on the fly. Since all possible views are derived from events, you are no longer bound by the system's single “current state”. If you need to view data in a new shape or form, you can write a piece of code which can process the event stream and update a new structure. This is a compelling and future-proof concept because if you look closely, the business events almost always remain the same, but the information you extract from them changes with time.
  2. Scalability. The read-and-write portions of the application can be independently managed and scaled. Even with a different tech stack altogether. Writes generally happen on the events store and are read from one of the materialized views.
  3. Business communications. Product and business teams describe the business requirement as a state machine comprising business events. They do not think in lines of Database entities and their current state. Using event sourcing as the core design approach, you are modelling the applications exactly how they define it. This leads to more effective communications and better products overall.
  4. Reporting. Since the complete historical records of events are stored, you can ask any question. From the shopping cart example above, the business team needs to create a report of all customers who removed the item(s) before checking out the cart and then buying later. They may use this data to send personalized feeds and offers. In the usual CRUD-based design, this would require a long and specific development effort. Still, the reports will apply only from when the feature was put into production.
    Event sourcing design, however, will require much less effort. You need a new code to process the events stream and update the underlying reporting structure. In addition, the event stream can be replayed for any required time range.

Concerns

  1. Eventual consistency. The dependent systems (materialized views/cross-team subscribers) won’t be updated immediately when you use event sourcing. Instead, these will be only eventually consistent. Remember the CAP theorem. This design is focused on providing high availability and partition tolerance.
  2. Storage cost. Since events are additionally stored along with materialized views, the storage cost can be higher. However, this can be managed using a hot and cold storage strategy. Events can be purged on time in consultation with product teams.

Summary

In summary, Event sourcing is an alternative design pattern, you can consider while designing your applications instead of the usual CRUD-based design. Event Sourcing and CQRS patterns are frequently paired together.
Event sourcing can be used to design large-scale distributed business software systems in Banking, Ecommerce, Travel domains etc. It allows you to communicate with business in their language and easily change and adapt the design.

Further Read

Please look at this blog which discusses about using Event Sourcing design in one of the real world examples of large scale distributed system.

Event Sourcing Design Pattern was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Article Link: Event Sourcing Design Pattern. This blog will discuss the Event… | by Keshav Gupta | Walmart Global Tech Blog | Medium