Partitioning data and load
How can work be distributed and data ownership transferred safely?
Slide contents
1. Partitioning data and load
How can work be distributed and data ownership transferred safely?
2. Equal data does not mean equal load
Start with task-service requests.
3. Choose the key for operations
One ordering cannot optimize every query.
4. Replicas and shards serve different purposes
Copies protect data; shards divide it.
5. Routing must reach the current owner
The placement map has a version.
6. Ranges preserve key locality
Scans are convenient; skew remains possible.
7. Hashing changes the cost of range queries
Exact lookup is easy; lists need gathering.
8. Changing the divisor moves many keys
Simple hash mod N depends on cluster size.
9. A ring separates keys from machine count
A key selects the next position around the ring.
10. Many tokens do not create independent copies
Virtual positions distribute ranges.
11. One hot key survives every ring
More machines still leave one owner per key.
12. A hot range can be split
The boundary must divide active work.
13. Partition size sets movement cost
Smaller units are flexible but need metadata.
14. Tenant keys concentrate tenant load
Tenant A overloads one shard: 600 > 300.
15. Splitting a tenant changes the cost of its list
Distributed writes need an explicit read path.
16. Migration must not lose acknowledged writes
Change a range’s owner with no loss and no second writer.
One authorized writer
Acknowledged writes preserved
Retries preserve identity
17. A snapshot anchors migration
The old owner still accepts new writes.
18. The copy catches up to a barrier
Finishing the copy is insufficient.
19. A barrier prevents new migration gaps
Drain old writes and record the boundary.
20. Authority transfers with its generation
New routing must not revive the old writer.
21. An old client refreshes routing before retrying
The operation key survives ownership changes.
22. An uncertain cutover requires reading the decision
A map-update timeout does not mean abort.
23. Rebalancing measures both benefit and disruption
Copying competes with foreground requests.
24. Design a key and safe migration
Tenants X/Y/Z: 480/160/160 requests/s; four shards at 240/s each.
Estimate the hot shard
Find the missing write
Define the cutover condition
25. Key choice fixes future trade-offs
State the decision through queries and invariants.
26. After this lecture, you can
Choose a key for operations and workload.
Distinguish hot keys from hot ranges.
Justify a migration barrier and its checks.