Who Made Kubernetes So Complex? How Complexity Moved into Platforms and AI Agents

In November 2025, SIG Network and the Kubernetes Security Response Committee announced the retirement of ingress-nginx. Maintenance ended in March 2026. There would be no more releases, bug fixes, or security patches. Existing installations would keep running, which made the risk easy to miss. In its January statement, the Steering Committee cited internal Datadog research indicating that ingress-nginx was present in roughly half of cloud-native environments. That is one observability vendor's sample, not a market census.

The scale matters less than the reason. For years, one or two people maintained the project in their spare time after work. Its flexibility, especially the ability to inject arbitrary NGINX configuration through snippet annotations, slowly changed from an advantage into technical debt that the retirement notice called insurmountable. A 2021 vulnerability had already shown that snippets could expose cluster secrets. In March 2025, the IngressNightmare chain escalated the same class of design risk to unauthenticated remote code execution.010101A classic pattern: a library critical to the entire internet was likewise maintained by a handful of volunteers. I retold the Log4Shell story on my channel, from the Log4j maintainer's account.Knizhny kub · the Log4Shell story (in Russian)01A classic pattern: a library critical to the entire internet was likewise maintained by a handful of volunteers. I retold the Log4Shell story on my channel, from the Log4j maintainer's account.Knizhny kub · the Log4Shell story (in Russian)

There is therefore no single defendant in the case of Kubernetes complexity. Upstream shipped APIs with imperfect boundaries. Vendors and users extended them for production. Organizations demanded centralized compliance, multi-tenancy, meshes, custom control planes, and new workload types. Platform teams promised that another abstraction would hide the previous one. Complexity did not disappear; it changed owners.

Kubernetes manages a distributed system in which processes fail, networks partition, configuration changes asynchronously, and actual state must converge on desired state. That complexity is essential. It existed in Borg, in home-grown schedulers, and in shell scripts. Removing Kubernetes does not remove service discovery, rollout, isolation, capacity planning, or recovery. It only changes where those problems are solved.

There is also accidental complexity. PodSecurityPolicy lived for years, was deprecated in 1.21, and disappeared in 1.25. NetworkPolicy deliberately omitted deny rules and cluster-wide defaults, so centralized governance later required a separate AdminNetworkPolicy API. Policy engines grew outside core; Kubernetes then brought part of that work back into the API server through ValidatingAdmissionPolicy, whose rules use the Common Expression Language (CEL). These are not merely business requirements. They are architectural decisions, some of which had to be reversed.

Tim Hockin, one of the project's founders, describes a finite complexity budget. Every feature spends more than lines of code: it spends the project's ability to explain, test, and eventually change behavior. The problem is particularly sharp in an extensible system. CRDs are among Kubernetes' strongest mechanisms and also the easiest way to turn a cluster into a distribution of dozens of independent products with unrelated upgrade cycles.020202On how complexity accrues and how to cut it at the design level, the best book is John Ousterhout's “A Philosophy of Software Design” (he's also a Raft co-author). I covered it on my channel.Knizhny kub · A Philosophy of Software Design (in Russian)02On how complexity accrues and how to cut it at the design level, the best book is John Ousterhout's “A Philosophy of Software Design” (he's also a Raft co-author). I covered it on my channel.Knizhny kub · A Philosophy of Software Design (in Russian)

Why is Kubernetes called boring, then?

The CNCF Annual Survey 2025 reports that 82% of container users run Kubernetes in production, while 34% name complexity as a blocker. Culture, training, and security rank higher. That is an important counterweight to the disaster narrative: APIs stabilized, managed services became routine, operational patterns became known, and Kubernetes stopped being an experiment.

Adoption surveys and operating cost answer different questions. “We are no longer afraid to adopt Kubernetes” does not mean “Kubernetes is cheap and simple to run.” Once a dedicated platform function owns the burden, product teams genuinely stop seeing much of it. The system became boring for its users—at the cost of greater operational complexity for the platform team.

The original Ingress API was intentionally small: host, path, backend, and TLS. It had no common language for timeouts, retries, canaries, external authorization, header rewriting, or web application firewall (WAF) behavior. Controllers filled the gap with annotations. By July 2026, ingress-nginx documented roughly 130 unique keys, while snippets allowed raw NGINX configuration. A locally simple API produced a global, non-portable domain language.

Gateway API does not answer with less YAML but with an explicit role model: the infrastructure provider is responsible for GatewayClass, the cluster operator for Gateway, and the application team for Route. ReferenceGrant constrains cross-namespace references, status exposes what a controller actually accepted, and policy gets a formal attachment point instead of a string in metadata. As of this article, the current line is 1.6.x, with TCPRoute and UDPRoute graduated to GA.

The cost is explicit too. One route may now depend on several resources, while policy attachment must handle inheritance, conflicts, discoverability, and status fanout. GEP-713 itself acknowledges that the mechanism is more complex than its authors would like. Yet this is a different kind of complexity: it can be validated, observed, and divided between roles. An annotation looked simpler only until the first incident.

eBPF does not invalidate the argument

eBPF—a Linux kernel mechanism for running verified programs—helped Cilium build a powerful data plane and solve real hyperscale problems, but it is not the universal answer to Kubernetes networking. kube-proxy's nftables mode reached GA in Kubernetes 1.33 and removed the linear iptables lookup without replacing the Container Network Interface (CNI) plugin. Istio ambient removed per-pod sidecars through the Rust-based ztunnel rather than mandatory eBPF. In one of its network benchmarks, Cilium reports nearly one million requests per second, but the result comes from a two-node test bed with a 100 Gbit/s link. It should not be projected automatically onto a fifty-node cluster. A new data plane is justified when it solves a measured constraint, not because the previous layer looks unfashionable.030303Under most of these data planes and meshes sits Envoy — there's a great documentary on its history (including the date-handling bug that logged out millions of users). I covered it on my channel.Knizhny kub · the Inside Envoy documentary (in Russian)03Under most of these data planes and meshes sits Envoy — there's a great documentary on its history (including the date-handling bug that logged out millions of users). I covered it on my channel.Knizhny kub · the Inside Envoy documentary (in Russian)

Cluster separation is useful for blast radius, versions, regions, tenants, and upgrade requirements. The number of control planes alone guarantees nothing. In March 2023, Datadog lost networking on more than 60% of its instances across five regions and three cloud providers. A systemd security update, distributed through one shared policy, removed Cilium routes. The fleet was geographically and commercially diverse but still correlated through its OS image and update mechanism.

The same trap appears in hub-and-spoke GitOps, a single registry, a global identity provider, a common admission webhook, or one Cluster API template. A hundred clusters updated by one policy within an hour are one failure domain with a hundred control planes. Independence requires separation not just of runtime but of change delivery, trust, artifacts, and authority.040404Resilience is an architectural characteristic you design for, not a property of how many clusters you run. There's a good summary of fault, failure, and resilience from “Continuous Architecture in Practice” on my channel.Knizhny kub · resilience as an architecture characteristic (in Russian)04Resilience is an architectural characteristic you design for, not a property of how many clusters you run. There's a good summary of fault, failure, and resilience from “Continuous Architecture in Practice” on my channel.Knizhny kub · resilience as an architecture characteristic (in Russian)

Cluster count also has a direct price. Amazon Elastic Kubernetes Service (EKS) charges $0.10 per hour for a control plane under standard support and $0.60 under extended support for an old version (list price as of July 2026). Across fifty clusters, delayed upgrades become a budget line rather than a minor inconvenience. Multi-cluster should be a deliberate isolation model, not a reflex to “create one more.”

Kubernetes standardized how containerized applications run, but product teams still had to answer practical questions themselves: how to build and deploy an application, configure policy, networking, observability, and secrets, release changes safely, and take responsibility for ongoing operations. Platform engineering moves the repeated part of that work into a shared internal product: the platform team maintains templates, automation, and checks, while product teams remain responsible for their applications. An internal developer platform assembles those capabilities into a golden path and offers self-service instead of tickets. Team Topologies supplied the right goal—reduce cognitive load—and a useful constraint: a Thinnest Viable Platform may do without a dedicated web portal; sometimes a maintained set of conventions and templates is enough.050505I opened my Code of Leadership podcast with exactly this book, Team Topologies — my guest was a colleague who spent years building our IDP. The episode is on my channel.Knizhny kub · Code of Leadership #1 on Team Topologies (in Russian)05I opened my Code of Leadership podcast with exactly this book, Team Topologies — my guest was a colleague who spent years building our IDP. The episode is on my channel.Knizhny kub · Code of Leadership #1 on Team Topologies (in Russian)

Backstage demonstrated the product form: catalog, templates, documentation, and plugins. That form is natural at large scale. Avito's platform serves thousands of services and hides Kubernetes behind an app.toml file and a CLI. Ozon values uniformity over local perfection. T-Bank has built its internal platform since late 2020 with a team of roughly fifty people. These examples do not prove that every company needs an IDP. They demonstrate the scale and sustained ownership required to operate a real internal PaaS.

Better for the developer, not necessarily for the system

DORA 2024 found a paradox. Internal-platform use was associated with 8% higher individual productivity and 10% higher team performance, while change throughput fell by 8% and change stability by 14%. The report offers a cautious J-curve interpretation: an immature platform creates transition costs before its benefits arrive. Less comfortable explanations are also possible—a new queue, an abstraction that is too thick, or a platform optimized for satisfaction rather than flow.

This is the boundary between a golden path and a golden cage. Sam Newman warns that a mandatory platform loses its incentive to be good because users cannot leave. Yet security, compliance, and audit do not work as optional suggestions. Permitting everything is not product thinking; banning every alternative is not product thinking either. A legitimate mandate protects verifiable invariants rather than one implementation of every delivery step.

• —identity, audit trail, encryption, and policy evidence are mandatory;
• —service templates, CI/CD, and the self-service interface are preferred but replaceable;
• —exceptions have an owner, an expiry date, and an explicit support cost;
• —the platform measures lead time, reliability, and safely completed intents—not the number of generated components.

AI workloads exposed another boundary in Kubernetes' original model. The device plugin framework arrived in 1.8 but treated a GPU as a whole integer for years. NVIDIA Multi-Instance GPU (MIG) partitions one GPU into isolated instances. Time-slicing, separate schedulers, and the GPU Operator filled the same gap in other ways. Dynamic Resource Allocation reached GA in 1.34 after an API redesign, while fractional capacity continued separately. It is the same cycle again: a small primitive, expanding requirements, an ecosystem of workarounds, and eventually a more explicit model in core.

CNCF calls Kubernetes the de facto operating system for AI, while scheduling-optimizer vendors report low utilization of expensive GPUs. Vendor data are observations from their own telemetry, not an industry-wide measurement; this article cites no pooled GPU-utilization study with a disclosed methodology. The sources cannot be combined into a single estimate: the CNCF survey describes Kubernetes adoption among container users, while vendor publications describe telemetry from individual optimizers and their own samples. Even so, the contrast raises a useful question: does Kubernetes help manage the workload, or merely provide a familiar interface to a new inefficiency?

The industry wanted humans away from kubectl before LLMs

In 2017, Kelsey Hightower compared kubectl to the new SSH: direct human control of production was evidence that a higher-level interface was missing. K8sGPT, kubectl-ai, HolmesGPT, kagent, and MCP servers now compete for that role. Assistance, however, is not autonomy. The original ITBench study, from IBM Research and UIUC, contained 94 scenarios. The best reported results were 13.8% for SRE diagnosis and 25.2% for compliance assessment; across the two FinOps scenarios, inefficiency diagnosis reached 33%, while mitigation remained at 0%. In May 2026, IBM and Artificial Analysis extended this work with ITBench-AA: 59 SRE tasks built from offline Kubernetes incident snapshots—40 public and 19 private held-out—with three runs per task. The best score was 47% at launch; as of August 11, 2026, the top score on the leaderboard was 56.2% under a metric that gives a run zero if it misses any root-cause entity and otherwise penalizes extra entities. This figure is not directly comparable with the original 13.8% SRE diagnosis result: the task set, execution environment, and metric all changed. Neither figure measures safe incident remediation in production or grants an agent a general license to modify it.060606In a recent interview Hightower jokes that prompt engineering is just another unstable programming language: engineers fought for determinism for years and now commit 500 markdown files of prompts. I covered that interview on my channel.Knizhny kub · Kelsey Hightower, part 1 (in Russian)Knizhny kub · Kelsey Hightower, part 2 (in Russian)06In a recent interview Hightower jokes that prompt engineering is just another unstable programming language: engineers fought for determinism for years and now commit 500 markdown files of prompts. I covered that interview on my channel.Knizhny kub · Kelsey Hightower, part 1 (in Russian)Knizhny kub · Kelsey Hightower, part 2 (in Russian)

A useful agent starts with read-only diagnosis, inherits the user's identity and RBAC, presents evidence, and can hand the investigation to a human. The widely used Kubernetes MCP Server in the containers organization talks directly to the API server and exposes --read-only and --disable-destructive modes. A flag is not governance, but it marks the correct product boundary: reading a cluster and changing it require different contracts and evals.

The read → recommend → approve → act → rollback ladder must apply to a specific capability, not to a model's general intelligence. Restarting a pod, performing rollout undo, and draining a node need different policies, sandboxes, blast radii, and stop conditions. Every level retains identity, a policy decision, a trace, human handoff, and a tested rollback. Otherwise the agent becomes the next snippet: a convenient escape hatch that later turns into an ungovernable API.

“Do we need Kubernetes?” is too coarse a question. The real choice is the thickness of the operating model. A ten-person team may need Docker and managed deployment. Dozens of teams may need a common Kubernetes API without an internal portal. A full IDP is justified when repeated demand, compliance, and coordination cost exceed the cost of another platform product.

A scorecard before the next platform layer

This is not a maturity benchmark or a universal developer-count threshold. It is a set of six questions that expose ownership cost before another mandatory API exists.

If there is no measured problem, durable owner, or removal plan, the safest choice is not to build the layer. If the problem is real but the audience is small, start with a TVP: documentation, a template, and one observable capability. A portal, control plane, or agent comes only after repeated demand has been demonstrated.070707Exactly this is Gregor Hohpe's talk “Build Abstractions Not Illusions”: over-abstraction drops what matters and turns into a leaky illusion. I retold it on my channel.YOW! · Gregor Hohpe · Build Abstractions Not IllusionsKnizhny kub · talk summary (in Russian)07Exactly this is Gregor Hohpe's talk “Build Abstractions Not Illusions”: over-abstraction drops what matters and turns into a leaky illusion. I retold it on my channel.YOW! · Gregor Hohpe · Build Abstractions Not IllusionsKnizhny kub · talk summary (in Russian)

Which brings the title question back. Kubernetes was not made complex by core architects or by YAML authors: the complexity moved to where nobody budgets for it — controller annotations, the platform team, the agent prompt. In practice the question is not “who made it complex.” It is who will carry the pager for this layer in three years, and what they will measure to show it paid off. Until that has a name and a metric, it is too early to add the next layer.