I used to think adding a ‘Read Replica’ was a magic button for scaling applications. I was wrong. While splitting read and write traffic is a standard system design pattern, implementing it introduces a world of pain – from stale reads to the dreaded Split-Brain problem. Here is how database replication actually works, and how to survive the transition.
When people talk about “scaling databases” or “adding read replicas”, they are almost always thinking about one specific architecture:
Single-leader (Primary-Replica) architecture with asynchronous replication
This is the architecture used by:
- MySQL + replicas
- PostgreSQL + streaming replication
- Google Cloud SQL
- PlanetScale, Neon, Supabase, etc.
There is exactly one node that accepts writes → called the Primary (or Leader/Master).
All other nodes are Read Replicas → they apply changes from the primary as fast as they can, but always with some delay (replication lag).
This is the default and dominant model in 99% of applications today.
Alternative architectures exist (multi-primary, leaderless, CRDTs, etc.), but they are rare and come with their own very different trade-offs.
The second axis that actually matters in practice is:
Who manages the replicas and failover for you?
1. Self-hosted / Self-managed
You run MySQL or PostgreSQL yourself (on EC2, Kubernetes, bare metal, etc.).
You are 100% responsible for:
- Setting up replication
- Promoting a new primary when the old one dies
- Routing traffic correctly
- Handling replication lag
- Monitoring, backups, point-in-time recovery, etc.
2. Fully-managed cloud services
RDS, Aurora, PlanetScale, Neon, Supabase, CockroachDB, Spanner, YugabyteDB, etc.
The provider gives you a single connection string (or two: one for writes, one for reads) and magically keeps it pointing to healthy nodes, handles failover in seconds, and often hides (or eliminates) replication lag headaches.
This second axis is the one that determines how much pain you will actually feel in production.
Now, suppose you have a naive application with a small user base, and you decide to spin up a single self-managed instance.
And on the application end, you have provided the DB configurations, and your queries are running on the provided instance.
Now that your user base has grown exponentially, you, as an architect, observed your query patterns and saw a read: write ratio of 100:1.
So you decided to make a pivotal decision for your application. And you decided to split your read and write traffic by adding replicas to your database.
Read-Write SplitHere, the trouble starts. You have two separate endpoints, suppose
primary: db.write.endpoint
read replica: db.read.endpoint
Now the question arises, how will my application know what kind of DB operation it is, and who will decide which endpoint to call?
Isn’t it thrilling?
Identifying the kind of transaction is one of the difficult problems. There is no automatic detection. Industry wide here are few technique which is being used:
- SQL Parsing: Decide the query type based on the keywords
- Annotation Based: Frameworks like Springboot, supports it using Transactional annotation
- Explict connection: While reading or writing, make use of operation specific connection
- Service Layer Separation: Same database schema, same User model, just two services pointing to different database connections
To read more about it you can check out my article on transaction routing
Among the available approaches, the explicit connection strategy appears feasible for your team and is therefore implemented. For a period of time, the application operates smoothly.
However, challenges are inevitable in system design.
Eventually, one of the replicas becomes unavailable, affecting customers. While the change may appear minimal, you must decide whether to replace the failed replica or tolerate downtime until the server is restored, implying a lower emphasis on availability.
If you see, here are your issues
- Your team didn’t get any notification about the replica failure.
- If the replica goes down, the team has to manually override db endpoint and may require redeployment or a rolling restart, depending on your architecture.
But damage is already done, so you started thinking about how to get notified about the failure early, or a way to manage it that doesn’t require manual override.
How failover can be managed automatically?
You started researching, and you came across terms like DB proxy, orchestrator, consensus, and a lot more. Let's see what fits where
DB Proxy is a smart routing tool that watches all the nodes and keeps pointing to healthy nodes, which means it provides failover support. It also does load balancing, caching, and performance optimisation using connection pooling. Commonly used DB proxies are Amazon RDS Proxy, ProxySQL, and PgBouncer.
But in the case of failure of the primary node, the proxy is not capable of promoting the replica as master or primary.
So we need someone who can detect the failure on leader and elect one of the replicas as Primary. That's the role of an orchestrator(like Patroni, Vitess) with a centralised coordination service(like zookeeper, etcd).
Think of it like air traffic control – the orchestrator monitors all planes (database nodes), decides which runway is active (primary), and updates the control tower (Zookeeper). All pilots (DB proxies) check with the control tower to know where to land.
In the same way, the orchestrator detects the failure, elects who will new primary, and tells the zookeeper. Now the zookeeper keeps the configuration like primary endpoint, replica endpoints, etc. And DB proxy reads the configurations from zookeeper and route the request.
Failover managementYou might be thinking if orchestrator does everything, then why does it need zookeeper?
To answer this, you first need to understand Split-Brain problem.
Split-Brain Problem
Imagine you have 3 database nodes: A (current primary), B and C (replicas).
A network partition happens:
- Node A can still talk to nodes B
- Node C is isolated from A and B, but can still talk to the outside world or some applications
Without protection, two bad things can happen:
- Node A thinks “I’m still primary” → keeps accepting writes
- Node C (or an orchestrator that only sees C) thinks “I can’t reach A → A must be dead → I promote myself/C as new primary”
Split BrainNow you have two primaries at the same time accepting writes → data corruption, lost updates, inventory goes negative, money disappears, etc.
This is split-brain. It is the single worst failure mode in distributed databases. Once it happens, you usually have to shut down one side manually and pray.
Zookeeper/etcd solves this problem for us. It uses the consensus algorithms like Raft, Paxos to enforces the strict majority quorum.
Wait – Consensus? Quorum?
If seeing those words makes you want to close the tab, don’t worry. We will deep dive into those topics in a separate article. For now, just understand that consensus algorithms are simply the way distributed nodes agree on a single source of truth.
Let’s look at how far we’ve come. Our system design has evolved significantly:
- Level 1: We started with a naive, self-managed single instance.
- Level 2: As traffic grew, we split Read and Write concerns.
- Level 3: We realized manual failover is a nightmare, so we introduced a DB Proxy.
- Level 4: To solve the “Who creates the new Primary?” problem, we added an Orchestrator and Zookeeper to prevent Split-Brain.
However, we have ignored the elephant in the room.
We know how to route traffic, but we haven’t discussed the physics of data movement:
- How do replicas actually receive updates from the Primary?
- What happens when the replica is 5 seconds behind (Replication Lag)?
In Part 2, we will tackle Replication Lag and why it breaks your application logic.
Follow me so you don’t miss the next part!
From Single Instance to Split-Brain: A Database Scaling Journey was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.