Filetable· Part 2 of 3
Deploying in the middle of the day: two versions, one database, open tabs
For a few minutes, the old and the new version run side by side on the same database. And an open tab can stay on the old one for days.
6 min readLire en français
Filetable ships often, in the middle of the day, while customers are working. Nobody should lose an edit, see an error or have to reload because of a deploy.
Why Kubernetes
Filetable runs in Switzerland, on a managed Kubernetes cluster. Kubernetes, the open-source container orchestrator born at Google and now hosted by the CNCF, does one thing very well: you describe the desired state (which image, how many copies, which resources) and it gets there, then keeps it there. A container crashes, it restarts it. A machine disappears, it moves the work. A big batch of documents arrives, you add workers by changing one number.
For a deploy, it replaces the pods one by one: a new one is ready before an old one leaves, and traffic moves from one to the other without a cut. But Kubernetes knows nothing about compatibility between versions: during the deploy, two versions run at the same time, and it is up to the application to cope.
Rolling deploy
N and N+1 side by side: a few minutesThe challenge. For a few minutes, the old and the new version share a database the new one has already migrated. And yesterday's open tab still runs the old JavaScript: it talks to the new server for hours, sometimes days.
Three rules, both ways
Everything rests on a simple discipline: every version must work beside the previous one, on the same database. And both ways, since a rollback puts the old version in front of data written by the new one.
Every version works beside the previous one
- 01
N+1 reads everything N writes
What breaks it: A one-shot data migration: the rows N writes after it are left behind.
- 02
N reads everything N+1 writes
What breaks it: A new value in a closed list: the old version cannot read it.
- 03
The migrated schema accepts every query of N
What breaks it: A renamed column: every write from N still names the old one, and fails.
A change that breaks a rule therefore ships over several deploys. It is the classic expand / contract pattern: add first, then switch, and only at the end remove. Renaming a column, a single line of SQL, takes four:
Expand → contract
- BeforeIn the databaseamount_textThe code writesamount_textThe code readsamount_text
- Deploy 1expandadd, write bothIn the databaseamount_textamount_centsThe code writesamount_textamount_centsThe code readsamount_text
- Deploy 2datacopy over, then read the new oneIn the databaseamount_textamount_centsThe code writesamount_textamount_centsThe code readsamount_cents
- Deploy 3expandstop writing the old oneIn the databaseamount_textamount_centsThe code writesamount_centsThe code readsamount_cents
- Deploy 4contractdrop the old oneIn the databaseamount_centsThe code writesamount_centsThe code readsamount_cents
A pod that leaves cleanly
Cutting nobody off also means a careful exit for each pod. Kubernetes takes it out of traffic, but the gateway takes a moment to learn it: the pod must keep serving a little, then finish what it started, downloads included, before it stops.
A pod that stops
- 1Taken out of traffic
Kubernetes sends it nothing new, but the gateway takes a moment to learn it: it keeps serving.
- 2SIGTERM
No new connection. Requests in flight finish, downloads included, then the process exits.
- 3SIGKILL
Only for whatever would still run at the end of the grace period.
An idle connection, from both sides
The tab left on the old version
A server is replaced in minutes; an open tab is not. So on every exchange, the server and the tab tell each other which version they come from. Most of the time, the gap is harmless.
What the tab learns from the server
New version, still compatible
A quiet banner. The tab keeps working; reload whenever you like.
New version, incompatible
A blocking dialog. Reloads by itself, never while you are typing.
Then a quiet banner offers to reload, whenever you like. In the rare case where the new version breaks the old one, a dialog blocks the page: better to stop it than to let someone type what can no longer be saved.
This version of Filetable is no longer compatible
Filetable was updated in a way this page cannot follow, so changes made here can no longer be saved. Reload to continue with the new version.
Otherwise it reloads by itself shortly — never while you are typing.
tab · version Aserver · version B· compatible
What I take from it
- Zero downtime is a compatibility contract, not a Kubernetes setting.
- Add now, remove later. An addition costs one deploy; a rename, four.
- Think of open tabs: they are the application's oldest clients.
- Blocking beats losing. When a write can no longer go through, say so before anyone types.