Who Made Kubernetes So Complex? How Complexity Moved into Platforms and AI Agents

In November 2025, SIG Network and the Kubernetes Security Response Committee announced the retirement of ingress-nginx. Maintenance ended in March 2026. There would be no more releases, bug fixes, or security patches. Existing installations would keep running, which made the risk easy to miss. In its January statement, the Steering Committee cited internal Datadog research indicating that ingress-nginx was present in roughly half of cloud-native environments.

The scale matters less than the reason. For years, one or two people maintained the project in their spare time after work. Its flexibility, especially the ability to inject arbitrary NGINX configuration through snippet annotations, slowly changed from an advantage into technical debt that the retirement notice called insurmountable. A 2021 vulnerability had already shown that snippets could expose cluster secrets. In March 2025, the IngressNightmare chain escalated the same class of design risk to unauthenticated remote code execution.010101A classic pattern: a library critical to the entire internet was likewise maintained by a handful of volunteers. I retold the Log4Shell story on my channel, from the Log4j maintainer's account.Knizhny kub · the Log4Shell story (in Russian)

There is therefore no single defendant in the case of Kubernetes complexity. Upstream shipped APIs with imperfect boundaries. Vendors and users extended them for production. Organizations demanded centralized compliance, multi-tenancy, meshes, custom control planes, and new workload types. Platform teams promised that another abstraction would hide the previous one. Complexity did not disappear; it changed owners.

Kubernetes manages a distributed system in which processes fail, networks partition, configuration changes asynchronously, and actual state must converge on desired state. That complexity is essential. It existed in Borg, in home-grown schedulers, and in shell scripts. Removing Kubernetes does not remove service discovery, rollout, isolation, capacity planning, or recovery. It only changes where those problems are solved.

There is also accidental complexity. PodSecurityPolicy lived for years, was deprecated in 1.21, and disappeared in 1.25. NetworkPolicy deliberately omitted deny rules and cluster-wide defaults, so centralized governance later required a separate AdminNetworkPolicy API. Policy engines grew outside core; Kubernetes then brought part of that work back into the API server through CEL-based ValidatingAdmissionPolicy. These are not merely business requirements. They are architectural decisions, some of which had to be reversed.

Tim Hockin, one of the project's founders, describes a finite complexity budget. Every feature spends more than lines of code: it spends the project's ability to explain, test, and eventually change behavior. The problem is particularly sharp in an extensible system. CRDs are among Kubernetes' strongest mechanisms and also the easiest way to turn a cluster into a distribution of dozens of independent products with unrelated upgrade cycles.020202On how complexity accrues and how to cut it at the design level, the best book is John Ousterhout's “A Philosophy of Software Design” (he's also a Raft co-author). I covered it on my channel.Knizhny kub · A Philosophy of Software Design (in Russian)

Why is Kubernetes called boring, then?

The CNCF Annual Survey 2025 reports that 82% of container users run Kubernetes in production, while 34% name complexity as a blocker. Culture, training, and security rank higher. That is an important counterweight to the disaster narrative: APIs stabilized, managed services became routine, operational patterns became known, and Kubernetes stopped being an experiment.

Adoption surveys and operating cost answer different questions. “We are no longer afraid to adopt Kubernetes” does not mean “Kubernetes is cheap and simple to run.” Once a dedicated platform function owns the burden, product teams genuinely stop seeing much of it. The system becomes more boring for its users and thicker for its owners.

The original Ingress API was intentionally small: host, path, backend, and TLS. It had no common language for timeouts, retries, canaries, external authorization, header rewriting, or WAF behavior. Controllers filled the gap with annotations. By July 2026, ingress-nginx documented roughly 130 unique keys, while snippets allowed raw NGINX configuration. A locally simple API produced a global, non-portable domain language.

Gateway API does not answer with less YAML. It answers with an explicit model. GatewayClass belongs to the infrastructure provider, Gateway to a cluster operator, and Route to an application team. ReferenceGrant constrains cross-namespace references, status exposes what a controller actually accepted, and policy gets a formal attachment point instead of a string in metadata. As of this article, the current line is 1.6.x, with TCPRoute and UDPRoute graduated to GA.

The cost is explicit too. One route may now depend on several resources, while policy attachment must handle inheritance, conflicts, discoverability, and status fanout. GEP-713 itself acknowledges that the mechanism is more complex than its authors would like. Yet this is a different kind of complexity: it can be validated, observed, and divided between roles. An annotation looked simpler only until the first incident.

eBPF does not invalidate the argument

eBPF gave Cilium a powerful data plane and solved real hyperscale problems, but it is not the universal answer to Kubernetes networking. kube-proxy's nftables mode reached GA in Kubernetes 1.33 and removed the linear iptables lookup without replacing the entire CNI. Istio ambient removed per-pod sidecars through the Rust-based ztunnel rather than mandatory eBPF. Dramatic gains measured at a million containers should not be projected automatically onto a fifty-node cluster. A new data plane is justified when it solves a measured constraint, not because the previous layer looks unfashionable.030303Under most of these data planes and meshes sits Envoy — there's a great documentary on its history (including the date-handling bug that logged out millions of users). I covered it on my channel.Knizhny kub · the Inside Envoy documentary (in Russian)

Cluster separation is useful for blast radius, versions, regions, tenants, and upgrade requirements. The number of control planes alone guarantees nothing. In March 2023, Datadog lost networking on more than 60% of its instances across five regions and three cloud providers. A systemd security update, distributed through one shared policy, removed Cilium routes. The fleet was geographically and commercially diverse but still correlated through its OS image and update mechanism.

The same trap appears in hub-and-spoke GitOps, a single registry, a global identity provider, a common admission webhook, or one Cluster API template. A hundred clusters updated by one policy within an hour are one failure domain with a hundred control planes. Independence requires separation not just of runtime but of change delivery, trust, artifacts, and authority.040404Resilience is an architectural characteristic you design for, not a property of how many clusters you run. There's a good summary of fault, failure, and resilience from “Continuous Architecture in Practice” on my channel.Knizhny kub · resilience as an architecture characteristic (in Russian)

Cluster count also has a direct price. EKS charges $0.10 per hour for a control plane under standard support and $0.60 under extended support for an old version. Across fifty clusters, delayed upgrades become a budget line rather than a minor inconvenience. Multi-cluster should be a deliberate isolation model, not a reflex to “create one more.”

Platform engineering did not emerge because developers wanted another portal. Kubernetes standardized runtime but left product teams with too many decisions: charts, policy, networking, observability, secrets, rollout, and ownership. An internal developer platform assembled those decisions into a golden path and offered self-service instead of tickets. Team Topologies supplied the right goal—reduce cognitive load—and a useful constraint: a Thinnest Viable Platform may be curated agreements and templates rather than a portal.050505I opened my Code of Leadership podcast with exactly this book, Team Topologies — my guest was a colleague who spent years building our IDP. The episode is on my channel.Knizhny kub · Code of Leadership #1 on Team Topologies (in Russian)

Backstage demonstrated the product form: catalog, templates, documentation, and plugins. That form is natural at large scale. Avito's platform serves thousands of services and hides Kubernetes behind anapp.toml file and a CLI. Ozon values uniformity over local perfection. T-Bank has built its internal platform since late 2020 with a team of roughly fifty engineers. These examples do not prove that every company needs an IDP. They demonstrate the scale and sustained ownership required to operate a real internal PaaS.

Better for the developer, not necessarily for the system

DORA 2024 found a paradox. Internal-platform use was associated with 8% higher individual productivity and 10% higher team performance, while change throughput fell by 8% and change stability by 14%. The report offers a cautious J-curve interpretation: an immature platform creates transition costs before its benefits arrive. Less comfortable explanations are also possible—a new queue, an abstraction that is too thick, or a platform optimized for satisfaction rather than flow.

This is the boundary between a golden path and a golden cage. Sam Newman warns that a mandatory platform loses its incentive to be good because users cannot leave. Yet security, compliance, and audit do not work as optional suggestions. Permitting everything is not product thinking; banning every alternative is not product thinking either. A legitimate mandate protects verifiable invariants rather than one implementation of every delivery step.

• —identity, audit trail, encryption, and policy evidence are mandatory;
• —service templates, CI/CD, and the self-service interface are preferred but replaceable;
• —exceptions have an owner, an expiry date, and an explicit support cost;
• —the platform measures lead time, reliability, and safely completed intents—not the number of generated components.

AI workloads exposed another boundary in Kubernetes' original model. The device plugin framework arrived in 1.8 but treated a GPU as a whole integer for years. MIG, time-slicing, separate schedulers, and the GPU Operator filled the gap in different ways. Dynamic Resource Allocation reached GA in 1.34 after an API redesign, while fractional capacity continued separately. It is the same cycle again: a small primitive, expanding requirements, an ecosystem of workarounds, and eventually a more explicit model in core.

CNCF calls Kubernetes the de facto operating system for AI, while vendor reports simultaneously show low utilization of expensive GPUs. Those numbers cannot be merged. The first describes adoption among container users; the second describes telemetry from a particular optimization vendor and its sample. Together they pose a useful question: does the standard help manage the workload, or merely provide a familiar interface to a new inefficiency?

The industry wanted humans away from kubectl before LLMs

In 2017, Kelsey Hightower compared kubectl to the new SSH: direct human control of production was evidence that a higher-level interface was missing. K8sGPT, kubectl-ai, HolmesGPT, kagent, and MCP servers now compete for that role. Assistance, however, is not autonomy. IBM Research'sITBench contains 94 reproducible SRE, CISO, and FinOps scenarios. The strongest agents in the original study solved 13.8% of SRE tasks, 25.2% of security tasks, and none of the FinOps tasks. That is far from a general license to modify production.060606In a recent interview Hightower jokes that prompt engineering is just another unstable programming language: engineers fought for determinism for years and now commit 500 markdown files of prompts. I covered that interview on my channel.Knizhny kub · Kelsey Hightower, part 1 (in Russian)Knizhny kub · Kelsey Hightower, part 2 (in Russian)

A useful agent starts with read-only diagnosis, inherits the user's identity and RBAC, presents evidence, and can hand the investigation to a human. The widely used Kubernetes MCP Server in the containers organization talks directly to the API server and exposes--read-only and --disable-destructive modes. A flag is not governance, but it marks the correct product boundary: reading a cluster and changing it require different contracts and evals.

The read → recommend → approve → act → rollback ladder must apply to a specific capability, not to a model's general intelligence. Restarting a pod, performing rollout undo, and draining a node need different policies, sandboxes, blast radii, and stop conditions. Every level retains identity, a policy decision, a trace, human handoff, and a tested rollback. Otherwise the agent becomes the next snippet: a convenient escape hatch that later turns into an ungovernable API.

“Do we need Kubernetes?” is too coarse a question. The real choice is the thickness of the operating model. A ten-person team may need Docker and managed deployment. Dozens of teams may need a common Kubernetes API without an internal portal. A full IDP is justified when repeated demand, compliance, and coordination cost exceed the cost of another platform product.

A scorecard before the next platform layer

This is not a maturity benchmark or a universal developer-count threshold. It is a set of six questions that expose ownership cost before another mandatory API exists.

If there is no measured problem, durable owner, or removal plan, the safest choice is not to build the layer. If the problem is real but the audience is small, start with a TVP: documentation, a template, and one observable capability. A portal, control plane, or agent comes only after repeated demand has been demonstrated.070707Exactly this is Gregor Hohpe's talk “Build Abstractions Not Illusions”: over-abstraction drops what matters and turns into a leaky illusion. I retold it on my channel.Knizhny kub · Build Abstractions Not Illusions (in Russian)