Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

As companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.

Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.

In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a multi-tenant database layer for many OpenAI products, but we went through a circular journey from using one multi-tenant Azure Cosmos DB container to provisioning single-tenant backing containers to reverting to a multi-tenant container by default. We’re still very early in our multi-tenancy story on Habitat: we support a large set of use cases, but we’ve been able to defer investment in this area by leveraging Azure Cosmos DB’s built-in primitives wherever possible.

Before walking through our Azure Cosmos DB journey, some basic context:

• An Azure Cosmos DB account defines a set of containers and a replication policy

• Each container is provisioned for a certain amount of throughput in request units per second (RU/s)

• Data within each container is stored in a typical shard-by-specified-logical-partition-key manner with documented limits on storage and throughput per logical partition key

• Azure Cosmos DB handles sharding logical partition keys across fewer backing physical partitions, splitting them into multiple physical partitions when storage exceeds a threshold or when the container requires more physical partitions to sustain the provisioned throughput

We originally allowed any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. When we first set out, we also used only one default Azure Cosmos DB container in each account. This worked well enough, but proved problematic for a few reasons:

Due to early API choices, we exposed logical partition key selection directly to clients. As a result, we have frequent collisions on commonly used partition keys, such as user_id or workspace_id, across different product use cases. Because these use cases shared Azure Cosmos DB limits on the same logical partition keys, the design constrained overall throughput for a given user.

One common problem we faced in scaling Postgres before we moved to Habitat and Azure Cosmos DB was the ability for many requests from a very hot organization or hot key—one tenant generating a disproportionately high load—to take out the whole database. Significant collective effort was spent to cache those keys and build leases into cache misses so that the resulting cache miss load does not stomp on the underlying database.

While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced. In practice, this means logical partitions are unable to reach 10,000 RU/s. Instead, they share the physical partition’s limits with every other logical partition key colocated on that physical partition.

This allowed noisy neighbors to impact many unrelated logical partition keys, with workloads operating on data for niche product features disrupting core login and signup flows.

The throughput limit for a given physical partition is also variable, dependent on the total throughput assigned to the Azure Cosmos DB container. The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.

As data volume grows, physical partitions split, increasing the total partition count while total provisioned throughput remains unchanged. The result is that the effective throughput limit per partition shrinks as data size grows.

For our most storage-heavy, blob-like use cases, this creates a failure mode where storage growth alone reduces throughput capacity for active workloads even while request rates and usage remain stable.

Separate containers help partially solve these partition key limits and noisy neighbors. By using separate containers, each can be assigned its own independent physical partitions and only the varying workloads on that container itself could be noisy neighbors or overlap in partition key space. On the downside, there is non-zero overhead in provisioning new containers—each container requires individualized capacity planning—so we want to create containers only at a service or use case boundary that makes sense.

We began to use separate Azure Cosmos DB containers for the most critical new use cases requiring strong isolation. Combined with a script for managing weak partitions by synchronizing the total throughput provisioned for each container with its number of physical partitions, we mostly avoided all the problems we were facing.

However, since we started with a default container, many legacy critical use cases were unfortunately stuck there without a data migration. To protect critical services that still relied on data in the shared default container, we often advised teams to create separate containers for new services or major product features. As we scaled, pressure on the default container grew and production incidents also grew. We began to provision containers for any new service or large product feature just to protect the default container from more load.

This saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages. However, the overhead of provisioning new containers eventually caught up to us:

In the old shared default container, any particular new use case going from 0 QPS to 1 million QPS was unlikely to significantly shift the total aggregate usage. On an isolated container however, a new use case going from 0 to 1 million QPS might represent 1,000x more load than the baseline attributed to only internal dogfooding.

With dedicated containers, teams now needed to remember to pre-provision containers to the expected level of throughput for their launches. Product teams would often ship their features to production and forget to pre-scale and we’d sit in SEV rooms on launch day attempting to scale up containers last minute.

This is a core workload mixing benefit of multi-tenant systems that we lose in favor of better isolation with this container strategy.

Over time, monitoring usage of dedicated containers also required intimate product knowledge of when traffic would shift or grow.

We can provide a basic control plane script to scale up the provisioned throughput for each container based on growth trends, but without the stable aggregate growth, scaling these correctly enabled essentially every product team at OpenAI to be involved in capacity planning.

This failed to scale organizationally with a wide variance in how much diligence each service’s owning team paid to their Azure Cosmos DB provisioning.

As the organization grew, those operational problems became difficult to scale. No lone, central team doing planning resulted in many teams planning poorly. As a result, we ended up shifting back towards a multi-tenant solution and worked closely with the Azure team to engineer fixes for our original multi-tenancy issues with a shared-tenancy design:

Throttling the hottest keys first prevents noisy neighbors from consuming all of the available throughput for a given physical partition This reduces the noisy neighbor effect, where one hot key can have a site-wide blast radius.

Request tiering QoS similarly prevents offline workloads, such as bulk inserts or data extracts from utilizing all of the throughput for a given physical partition. By assigning the requests for these workloads a lower priority, they are throttled first, preventing throttling on higher priority, interactive requests that serve end users.

Azure Cosmos DB already had a serverless mode that would’ve fixed our weak partitions problem with the provisioned mode. However, there was a major downside: it scaled based on storage usage only and limited RU usage to 5K RU/s per physical partition. This would solve our weak partitions problem, but prevent us from scaling up RU-heavy use cases.

A new improved serverless mode Azure Cosmos DB team built for us allowed every physical partition to scale up to 10K RU/s and introduced RU-usage-based splits in addition to storage-usage-based splits allowed us to stop monitoring and scaling up containers manually.

With these changes, we’ve been able to shift back to using one shared container which has significantly simplified the need to do any upfront or continued capacity planning for product teams while preserving isolation between workloads. We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development.

When companies grow to OpenAI’s scale, managed services become less “managed.” Many workloads become outliers in the cloud provider’s infrastructure. As OpenAI became one of the largest users of Azure Cosmos DB, operating it successfully required a different model: shared tests, shared dashboards, direct engineering channels, and joint ownership of failure modes that only appear at our scale. Today, our largest Azure Cosmos DB account stores 10s of petabytes of data and serves more than 20M QPS. To our knowledge, we are the only company which has deployed Azure Cosmos DB at this scale.

In late 2024, a regional outage made it clear why this collaboration was necessary. At the time, a power outage struck the primary Azure region for one of our largest Azure Cosmos DB accounts. In the resulting failover exercise, we realized that our largest accounts would take several hours to fail over to a secondary region, something neither our team nor the Azure Cosmos DB team had tested at our scale.

About a month later, while failing those accounts back to the original region, we triggered another serious outage. A connection stampede to the original region overwhelmed the original region’s Azure Cosmos DB gateway. A quirk in how the gateway-mode Azure Cosmos DB client handled failovers caused all clients to immediately establish connections and send requests to the original region, even though only a subset of physical partitions had moved back to that region.

In each outage, both OpenAI and Azure ran through their incident management processes, with a number of root causes identified and fixes deployed. But these outages were symptoms of a larger systemic issue between the way our team and Azure Cosmos DB team operated and tested. At first, leveraging managed services allowed us to move fast and focus on business goals. At scale, managed services become less managed and require deep integrations with the service provider at all levels from design to operations.

Today, the Habitat and Azure engineering teams jointly monitor every layer of Habitat from our service layers down into Azure Cosmos DB’s physical storage layers. We operate and test changes against a production-sized but non-production Azure Cosmos DB account. We test and iterate on SDKs together and determine best practices for how we scale and size our Azure Cosmos DB accounts. We have direct engineering access to each other through Teams and have even exposed ways to directly page each other without going through support tiers.

A substantial amount of joint work has occurred over the last year to improve Azure Cosmos DB control-plane and gateway scalability, adding load protection and caching of various Azure Cosmos DB metadata in their gateway to protect the Azure Cosmos DB metadata partition, reducing pressure from our fleet of Python habitat-service processes through connection pooling and higher connection reuse, colocating workloads where possible, and isolating critical workloads from offline workloads via different VPCs.

This level of co-engineering has been critical to our Azure Cosmos DB stability in the last year. Any system large or critical enough necessitates this kind of teamwork: breaking down ownership walls and operating across organizational boundaries.

As mentioned in Part I, our core philosophy is: make it work, make it fast, make it cheap. Reliability first, then performance, then efficiency. Habitat had become critical infrastructure for OpenAI, so the first priority was making it reliable. As the system stabilized in 2025 and OpenAI expanded globally, latency became the next constraint for users and product teams.

Compute and storage capacity were often located in different regions, making cross-region requests unavoidable.

Latency varied widely depending on where the data was stored and where the request was served. To make Habitat requests performant, product developers relied on additional client-side caching, request batching, minimizing database reads and writes, and moving work off the critical path wherever possible. While this led to some interesting product designs that were more resilient to database issues, it also made products more complex and slowed down product iteration: a failure to achieve Habitat’s goal to encourage fast product development and not needing to understand the database.

This has been a big focus for us in 2026, and we’ve developed a layered strategy for optimizing read performance, which represents over 90% of Habitat’s requests:

We have a number of Azure Cosmos DB accounts in different Azure regions around the world for users with data residency requirements. Even for users without hard data residency requirements, colocating data near users also improves their experience through lower latency.

In practice, we’re not able to achieve this broadly outside of data residency use cases because placement is constrained by physical cloud capacity. Due to the size of our data sets, many regions cannot accommodate the massive storage footprint required for their proportional share of user traffic.

In regions with limited datacenter capacity, power, or cooling, we sometimes face a direct tradeoff: reserve capacity for colocated storage and lower Habitat latency, or use that capacity for additional serving compute and route storage requests farther away.

As a result, placement decisions are often a product tradeoff between: A) optimizing latency for different products and user subsets, and B) raw storage and compute capacity.

For the most latency-critical workloads, like authentication metadata that is needed on every request, we reduce RTT between compute and Habitat by using highly replicated Azure Cosmos DB accounts with many secondary regions. Some accounts replicate to all 40 Azure regions where we serve traffic; others replicate to at least one region per continent.

We’ve had to build a transparent data movement and routing layer into Habitat earlier than desired so that we can move datasets between accounts with different replication topologies.

However, this approach is also ultimately capacity constrained and therefore limited to a few hand-selected datasets.

Habitat workloads are overwhelmingly read-heavy, with applications constantly fetching settings, state, and other properties; a disproportionate amount of traffic is attributable to a relatively small working set of frequently accessed types. This makes the system particularly well suited for distributed caching. While caching improves performance in single-region infrastructure, the improvements compound when working across regions, where it can be used to “hide” cross-region latency.

We’ve integrated a distributed lookaside caching layer into Habitat. It stores a copy of recently accessed objects and edges in Valkey servers placed in our infrastructure close to where user requests are processed. Typically, these sorts of caches use a TTL that trade off between hit rate and staleness. Short TTLs reduce staleness but blunt the positive impact of caching, because items expire out of the cache quickly and must be retrieved more frequently. Longer TTLs improve hit rates, but serve stale data for extended periods, leading to poor user experiences and unpredictable product behavior.

Another approach is to execute explicit invalidation operations when the application is performing write operations on the same underlying data. This dramatically improves the freshness of cached data, but can be hard to get right. It also remains subject to a number of edge cases, like partial failures, which leave the cache stale. Multi-region deployments compound these problems, with less reliable cross-region networks, and the cache racing with the replication of the underlying storage. Complementing this with TTLs bounds the maximum staleness, but still suffers from the same underlying TTL trade-off discussed above.

We’ve taken a more advanced approach to invalidation, following in the footsteps of systems like Meta’s TAO. Habitat’s integrated cache relies on change data capture (CDC) through Azure Cosmos DB’s change feed to propagate invalidations globally within seconds, achieving high hit rates while keeping staleness low. A service in our infrastructure consumes this change feed and executes invalidations to the local caches when the underlying data is updated. This causes a miss the next time the data is read, and the cache is filled with the latest version of the data.

We’ve also significantly reduced latency by removing overhead in the Habitat service itself through rewriting the code in Rust, which is more performant than the original Python.

Together, these changes have reduced our overall global read p50 from roughly 120 ms to roughly 10 ms, a 12x improvement since we began focusing on performance earlier this year.

While caching has dramatically reduced median latency, the slowest requests still require fetching uncached data from underlying storage, often across distant regions. As more capacity comes online and we improve storage efficiency, we plan to address a more fundamental challenge: placing data closer to where it will be accessed from the start.

Reliable and scalable CDC—propagating database changes to downstream systems—is a critical component in more than just Habitat’s caching strategy. Millions of captured changes per second, across thousands of containers and millions of container partitions feed through more than a hundred Kafka topics to also power eventually consistent secondary indexes, data-movement pipelines, offline analytics, and other real-time product features. As our backing infrastructure has grown globally, it’s become increasingly complex to listen to row changes. If a client wants to register a callback to run whenever rows of their type are changed, they must, among other things, first determine which Cosmos accounts and containers contain their data, which CDC service deployment is running in respective regions listening to those containers, and properly scale the CDC service worker fleet, Kafka, and their consumer. Our CDC platform aims to abstract all of this away from product developers and provide instead a simple surface where clients register a function they’d like called on their changed rows and we handle all of the plumbing and scaling.

An internal control plane continuously ensures all required containers are being captured, selects which Cosmos regions to read from, and efficiently distributes work among thousands of workers across multiple geographic regions. Using replica regions and lower-priority feed reads protect latency-sensitive Habitat traffic from background capture workloads. Worker crashes, container issues, lease-store errors, invalid configuration, and destination failures occur continually across a fleet this large. We hardened CDC worker lifecycle and lease handling, isolated failures between streams, and built safeguards for configuration changes and rollouts. We utilize dead letter queues to prevent head-of-line blocking from permanent delivery failures. We tolerate regional outages by switching readers to alternate Cosmos regions. At our scale, we’ve even had to make changes to Azure’s CDC SDK to make its checkpointing implementation more efficient by batching checkpoints in-memory and only periodically flushing them.

While Habitat’s internal usage of Azure Cosmos DB is complex, its value proposition has always been to abstract away infrastructure details from product developers to improve product development velocity, and our CDC platform is no different.

A small team was able to support a rapidly growing storage platform by keeping the storage API deliberately narrow, continuing to scale the Python service, and prioritizing the most urgent engineering challenges. We focused first on reliability for mission-critical product traffic, then on read performance for global users, and finally on operational efficiency at scale. This approach enabled rapid growth and new use cases while improving reliability, performance, and product development velocity.

In order to scale Habitat and conversation storage for the next billion users, we need to address the difficulty of replicating our data across our infrastructure, on an even larger scale. To preempt this, we’ve been building a custom in-house exabyte-scale online storage system, optimized for bulky Habitat data types. This in-development system combines the RocksDB Cloud core from Rockset with Raft-based replication. It implements the same document model as Azure Cosmos DB, but uses disaggregated storage and physical replication to reduce the amount of flash capacity needed, in exchange for a lower peak qps.

If you’re interested in working on these types of data infrastructure efforts at frontier scale, check out our open roles⁠.