The network is part of the application
How can we retry a request without creating a second task?
Slide contents
1. The network is part of the application
How can we retry a request without creating a second task?
2. The reply is missing, but the task exists
Synthetic case: disconnect after commit.
3. A timeout leaves two possible outcomes
No reply does not establish storage state.
4. RPC numbers and operation keys solve different problems
A retry may reach a new API process.
5. Every attempt for one intent carries one key
Generate the key before the first network call.
6. A key has a scope and a memory horizon
In the teaching namespace, records last until explicit cleanup.
7. One key must not change its payload
Retrying K with different content returns a conflict.
8. Two separate steps admit duplicates
Both handlers can observe K as absent.
9. One txn binds task, key, and outbox
etcd checks the condition and atomically applies one branch.
10. Concurrent retries share one result
Only the winner satisfies the absent-K condition.
11. Restarting the API does not erase the keyed result
A new process reconstructs the reply from etcd.
12. A creation acknowledgement does not mean completion
The outbox preserves the obligation to publish.
13. All attempts spend one deadline
The budget includes waiting, work, and retries.
14. Cancellation does not undo a committed transaction
Cancellation can arrive too late.
15. Retrying depends on cause and identity
A retry cannot repair invalid input.
16. Independent retries multiply load
Three layers with three attempts each allow 27 calls.
17. Jitter spreads repeated attempts
Attempt count and total deadline remain bounded.
18. A queue accumulates the rate difference
At 120 arrivals/s and 100 completions/s, backlog grows by 20/s.
19. Limit admission before expensive work
Design extension: bound the queue and active handlers.
20. A bounded queue needs a responding client
Rejection without retry limits moves the queue outward.
21. The first attempt saves three records
Teaching trace: K=blue, payload A, taskId=T7.
22. The retry returns T7 without a new outbox
The same K and A take the recovery branch.
23. Verification counts tasks, keys, and intents
A successful HTTP retry alone is insufficient.
One taskId for K
One task and outbox
Changed payload → conflict
Restart preserves mapping
24. Determine the allowed replies to three attempts
K=green; A differs from B; the first commit succeeded.
25. The lab verifies one durable result
Python API, three etcd members, later JetStream delivery.
26. Decisions for our service
Distinguish retransmission from a new operation.
Commit task, key, and outbox in one transaction.
Bound retries by one time budget.