Kubernetes Patterns — Behavioral Patterns

The conversation opens with an argument about who the book is for: the guest of part one felt some capabilities sound more like anti-patterns, and Misha agrees — for him Kubernetes is a big, complicated machine a platform should expose in measured doses, behind its own abstractions. The counter-argument: behavioural patterns are useful to anyone not running hundreds of clusters or serving tens of thousands of requests per second. One of the hosts splits the patterns into the mandatory ones — health checks and graceful shutdown, without which the platform's guarantees do not hold — and the situational ones, while the SRE guest adds that knowing the cluster objects pays off for a quick MVP: it saves time and money. The first pattern, Job, covers short-lived processes: do the work, finish and stop holding resources; the practical example is migrations run before a deployment, while a bare pod suits only development, since you watch the result yourself. The key parameters are completions and parallelism: with five completions and parallelism 2 the pods run in pairs until all five are done.

With completions 1 and parallelism above one the authors suggest building a queue, but the coordination has to live inside the application — nobody in the room had done that, and one participant adds it will baffle whoever maintains it next; the indexed Job is illustrated by slicing a video file, each pod taking its own chunk. A CronJob is assembled out of Jobs, and the chapter travels from cron on one machine, a single point of failure, to a service with leader election for a task copying files once a day. Of the parameters they go through concurrency policy and startingDeadlineSeconds: if the task has not started on time, the data may already be stale. The caveats are plenty — the schedule is accurate only to a minute, and many CronJobs load the cluster themselves through scheduling and address allocation, so a single daemon with an internal ticker is sometimes simpler. One participant recalls a startup where the reports drifted: cron fired every ten minutes, the database grew, the task started taking longer, and none of the primitives a CronJob has were there.

The general moral about periodic tasks: they get forgotten, people leave, and a buried job keeps doing something. DaemonSet is explained through the plain parallel with Unix daemons: one pod per node, or per a chosen set of nodes, for infrastructure work — log and metric collectors, tracing agents, a proxy on every machine. By the participants' recollection, daemons used to have their own scheduler guaranteeing priority, and at some version it was replaced by the common one: the pinning to a node stayed, but node affinity arrived, and with it a single point of resource management. Misha confirms this is purely a platform matter: a developer cannot launch anything across the whole cluster alone. The pain they name is imbalance: an application on one node writes too many logs, the agent there suffers, and limits must be raised for everyone at once.

Some call the singleton service an anti-pattern, and Alexander says frankly he would not want such guarantees provided through Kubernetes: the task reduces to active/passive, where a ReplicaSet does not guarantee a single copy while a StatefulSet first makes sure the previous one has stopped. The objection from the floor: people keep dragging in schedulers that only work as a single instance, especially when an old monolith is being cut into microservices — and many think they are doing distributed locking when it is exactly this singleton. At one participant's company there is a whitelist: to live with a single replica you have to argue in writing why nothing else will do. The second option is leader election inside the application through an external lock service with leases — ZooKeeper, Consul and Redis come up, as does a controller around one Apache project that elects a leader through a ConfigMap. The lower bound of live pods is held by a pod disruption budget.

Stateless is the familiar part: the twelve-factor app and Heroku, cattle instead of pets, ReplicaSet as a building block under Deployment, and a service that picks pods by labels rather than by the deployment; a volume can be attached to a ReplicaSet, but it will be one volume for all the pods. Stateful is where the trouble starts: people wanted to put MongoDB and MySQL into the cluster, the authors list four requirements — storage, networking, identity and ordinality — then admit StatefulSet semantics are often not enough and send the reader to the operators chapter, which a user of somebody else's platform will not be allowed to deploy anyway. The participants note the problem statement here is unusually long, workarounds such as a separate folder per replica read like brainstorming, and the book lacks a note saying "this is not the target solution". A separate thread is why databases came back into the cluster: there are simply too many of them, while monitoring, discovery and the operator mechanism itself differ little from an ordinary deployment.

The service discovery chapter is commented on by Misha, and his attitude is unkind: it looks neat, but there is hardly anything usable in that form. Behind the default ClusterIP sits kube-proxy rewriting iptables — which is why ping does not work, and why in practice there is no balancing: a client opens a persistent connection to one pod, and out of ten pods six are working, or even four. A headless service returns all the pod addresses and allows client-side balancing, but clients cache DNS and keep knocking at a pod that is already gone; NodePort is not recommended in production because of the ports on the nodes and the lost real client IP; and Ingress with L7 balancing is unavoidable, while nginx, participants say, gives no zero downtime out of the box. The last pattern, self awareness, is about an application learning its pod and node name through the Downward API: mounted files follow label changes, environment variables do not, and it is safer than calling the API, which applications can knock over with simultaneous requests. One of the participants recalls a 1998 Jackie Chan film about a hero who has to work out who he is. In a week: configuration and the structural patterns.