Habitat: the evolution of OpenAI’s storage layer (Category #AI4SDLC)
In the second quarter of 2026, two engineers used Codex and GPT‑5.5 to rewrite Habitat from Python to Rust. In an article published on September 11, OpenAI said that Rust already handles 95% of this service's requests. Habitat is the layer between ChatGPT, Codex, and storage systems: routing, access rights, encryption, and data-placement rules. It is an interesting case of an AI-assisted migration of critical infrastructure, and a large migration at that — according to OpenAI: almost 40 regions, more than 500 PB, and 70 million internal requests per second. The company claims a 6-fold improvement in CPU efficiency and a 15-fold improvement in memory efficiency, although the article gives no comparison methodology.
The most interesting part, however, is how the team reached the point where rewriting became the right next task.
1️⃣ In 2023, Habitat began as a Python library over Cosmos DB. It hid storage details from product teams. Convenient: add the library and work with data without reasoning through routing and authorization every time.
2️⃣ By mid-2025, that simplicity had become expensive. New logic had to be rolled out through dozens of services. Add shadow-traffic validation, and there were several more days of coordination. Fix a bug, and another round followed. Then one team rolled back its service for an unrelated reason and restored the old bug. The result was the very failure they had tried to prevent.
3️⃣ The shared library stopped being a useful control boundary — Habitat became a service, allowing storage changes, observability, and access policies to be controlled centrally. — At that point, the team deliberately kept Python. First they needed to stabilize the platform and API and unblock product development. They chose to pay the efficiency cost later, counting in part on the progress of coding models. The technical debt was a conscious decision about work ordering. — The API was intentionally made less powerful. Simple operations on objects and relationships, a predictable amount of work, no arbitrary SQL queries, and no unbounded graph traversal. An object and its relationships live in one partition; neighboring objects may be in another region. An elegant graph in the data model does not promise cheap traversal. — Complex queries use separate Rockset instances fed through CDC. Teams scale those instances themselves. Yes, this created more work for clients. But the cost of a complex query became their explicit responsibility, while analytics was isolated from operational data access.
4️⃣ The next challenge was tail latency inside the service itself The database had already responded, but a busy asyncio loop had not yet processed the result. The team began measuring task-scheduling latency. Even periodic parsing of a large feature-flags configuration caused slowdowns; reducing the configuration and staggering updates helped. A LIFO connection pool created a particularly unpleasant loop: a slow server returned its connection later, that connection was reused first, and the overloaded process received still more work (this was a case of metastable failures). Switching to FIFO broke the feedback loop.
5️⃣ They reached Rust only after the platform was working and its constraints were understood. At peak, the Python version handled more than 20 million requests per second. Then it was time to reduce the resource cost of the architecture. “Two engineers rewrote the service” is therefore the ending of a long story. Before that, the team had to define responsibility boundaries, constrain the cost of operations, and understand the system's behavior under load.
Overall, AI helped rewrite the implementation, but the case itself is much more interesting.
#AI4SDLC #Architecture #PlatformEngineering #Engineering #Rust #AI