How Systems Distribute Requests Across Servers: Load Balancing & Consistent Hashing

Whenever you open a website or use an app, your device communicates with another computer called a server.

For example, when you open a food delivery app and search for restaurants, the app sends a request to a server.

The request might be asking:

“Show me restaurants near my location.”

The server receives the request, processes it, retrieves the required information, and sends back a response.

The response may contain restaurant names, menus, ratings, prices, and delivery times. The app then displays this information on your screen.

This communication usually happens through an API.

An API defines how an application can send requests to a server and what kind of response it can expect in return.

The basic flow looks like this:

User → Application → API Request → Server → API Response → Application → User

Now imagine that your application has only one server.

When ten users send requests, the server may handle them easily.

But what happens when thousands, or even millions, of users send requests at the same time?

One server may become slow, overloaded, or completely unavailable.

To handle more traffic, we add multiple servers.

But that creates a new question:

Which request should go to which server?

That is where load balancing comes in.

What Is Load Balancing?

Imagine you run a restaurant.

At first, you have one person taking all the orders. That works when only five customers walk in. But what happens when 500 customers arrive at the same time?

One person cannot handle everything.

So you add more people behind the counter.

Now the question is simple:

Which customer should go to which counter?

That is the basic idea behind load balancing.

Load balancing means distributing incoming requests across multiple servers so that no single server becomes overloaded.

In simple words:

Instead of sending every request to one server, we spread requests across many servers.

For example:

• Server A handles some users
• Server B handles some users
• Server C handles some users

This keeps the system faster, more reliable, and easier to scale.

The Simple Way: Hashing

One way to decide which server should handle a request is by using hashing.

Let's say every user has a unique ID.

We pass that ID through a hash function, which converts it into a seemingly random number.

For example:

User ID: 1024
        │
        ▼
Hash Function
        │
        ▼
8,374,291

Now suppose we have 4 servers.

We can use the following formula:

server = hash(user_id) % 4

The remainder will always be either 0, 1, 2, or 3, so every request gets assigned to one of the four servers.

At this point, you might be wondering:

How do we know all the requests won't end up on just one server?

That's a great question.

The answer lies in the hash function.

A good hash function is designed to distribute values as uniformly as possible across its output range. It behaves almost like a random number generator.

Imagine we have one million different users.

Instead of producing something like this:

Server A → 900,000 requests
Server B → 50,000 requests
Server C → 30,000 requests
Server D → 20,000 requests

A good hash function is much more likely to produce something closer to this:

Server A → 249,800 requests
Server B → 250,400 requests
Server C → 249,900 requests
Server D → 249,900 requests

Notice that the requests are not perfectly equal, but they're close enough that no single server becomes a bottleneck.

This is why hashing is commonly used for load balancing.

So far, everything sounds great.

But there's a catch.

The Problem with Normal Hashing

What happens when we add a new server?

We now have five servers instead of four.

The formula changes from:

hash(user_id) % 4

to:

hash(user_id) % 5

This may look like a small change, but it changes the server assignment for most users.

For example, suppose a user's hash value is 17.

With four servers:

17 % 4 = 1

The request goes to Server B.

After adding a fifth server:

17 % 5 = 2

The same request now goes to Server C.

This does not happen to just one user. When moving from four servers to five, roughly 80% of the users may be assigned to a different server.

Why Is That a Problem?

Suppose a user is currently assigned to Server B:

User → Server B → Cached data found → Fast response

After adding a new server, the same user may be reassigned:

User → Server C → Cached data missing → Database request → Slower response

This creates several problems:

• More cache misses: The new server may not have the data already stored by the previous server.
• Higher database load: Missing data must be fetched again from the main database.
• More network traffic: Data may need to move between servers.
• Slower responses: The server must retrieve and rebuild information before responding.
• Lost sessions: If session data is stored locally, users may lose temporary information or be asked to log in again.
• Large data transfers: If the servers store actual data, much of it may need to be moved to match the new assignments.

So, normal hashing can distribute requests reasonably well.

The problem is what happens when the number of servers changes:

Add or remove one server → Most assignments change → Requests and data are reshuffled.

We need a method that keeps most assignments unchanged when servers are added, removed, or fail.

That is where consistent hashing comes in.

Consistent Hashing

Consistent hashing solves this problem.

Instead of arranging servers in a straight line, imagine a circle.

This circle is called a hash ring.

Both requests and servers are placed somewhere on this ring using a hash function.

When a request comes in, we place it on the ring and move clockwise until we find the nearest server.

That server handles the request.

That’s the main idea.

Why This Helps

Now imagine we add a new server.

In normal hashing, many requests may move to different servers.

But in consistent hashing, only the requests near the new server on the ring move to it.

The rest stay where they were.

This means adding or removing a server causes much less disruption.

That is the power of consistent hashing.

What Happens When a Server Fails?

If a server goes down, the requests that were going to that server move clockwise to the next available server.

Most other requests continue going to the same servers as before.

Only the requests assigned to the failed server need to move.

This makes the system more stable when servers are added, removed, or temporarily unavailable.

The Load Skew Problem

Consistent hashing solves the reshuffling problem, but there is still another issue.

Servers may not be evenly spread across the hash ring.

Imagine that Server A is responsible for a small section of the ring, while Server B is responsible for a much larger section.

Server B may receive far more requests than Server A.

This creates an uneven distribution of traffic, also known as load skew.

Virtual Nodes

To solve this problem, we use virtual nodes.

Instead of placing each real server on the ring only once, we place it at multiple positions.

For example:

• Server A appears in 10 places
• Server B appears in 10 places
• Server C appears in 10 places

These are not additional physical servers.

They are simply multiple points on the ring that all represent the same real server.

Because each server appears in several places, the requests are distributed more evenly across the system.

Virtual nodes also help when a server fails.

Instead of all its requests moving to one neighbouring server, they are spread across several different servers.

This prevents one server from suddenly receiving all the extra traffic.

Where Is Consistent Hashing Used?

Consistent hashing is used in systems that need to distribute requests or data across many machines.

Here are some practical examples:

Streaming Platforms

A streaming service may use many cache servers to store popular movies and videos.

Consistent hashing can decide which cache server should store or serve a particular video.

When a new cache server is added, only some videos need to be reassigned instead of moving the entire cache.

Social Media Platforms

Social media platforms handle millions of users, posts, images, and videos.

Consistent hashing can help decide which server stores or processes data for a particular user.

If the platform adds more servers, most users can continue using the same servers as before.

Online Shopping Websites

Large shopping websites use caching systems to quickly return product details, prices, and search results.

Consistent hashing can distribute this cached information across several servers.

If one cache server fails, only the information assigned to that server needs to move.

Content Delivery Networks

A content delivery network stores copies of images, videos, and website files on servers around the world.

Consistent hashing can help decide which server should store a particular file.

This allows the network to add or remove servers without redistributing every file.

Distributed Databases

Large applications may store data across several database servers instead of keeping everything on one machine.

Consistent hashing can decide where each piece of data should be stored.

When the system grows, only a small portion of the data needs to move to the new server.

Consistent hashing is useful whenever requests or data need to be distributed across multiple machines without reshuffling everything whenever the system changes.

To conclude everything:

Load balancing is about spreading work.

Consistent hashing is about keeping that distribution stable as the system changes.

Normal hashing may work well when the number of servers never changes.

But real systems are constantly changing.

Servers are added. Servers fail. Traffic grows. Systems scale.

Consistent hashing allows these changes to happen with much less disruption.

That is why it is such an important concept in system design.

---

I am Aneesh.

I write, record, edit, and build everything you see here.