From 15:04 to 21:48 UTC on 2025-09-22, a subset of Oso customers experienced delayed writes across all regions. For affected customers, this manifested as stale reads of up to 258 minutes. This was caused by a large series of expensive writes from a single tenant against a shared Kafka topic. Because writes to this shared resource are processed serially, writes from other tenants may have been enqueued until those expensive writes were processed by all regions.
This incident was triggered by an Oso customer accidentally triggering a clear and resync of their full production dataset to Oso Cloud. As this was using individual batch APIs rather than the currently recommended Oso Sync method for large updates, these operations were all enqueued in the normal write queue, rather than as background update operations.
<aside> ℹ️
In order to maintain a highly-available globally-distributed service, all writes are serialized to a Kafka topic which is then consumed by all nodes in each of our supported cloud regions. In practice this usually takes under 160ms. When a large number of expensive writes are loaded into the system, this latency increases.
</aside>
To improve workload isolation across our user-base, several months ago we deployed an update to Oso Cloud that supported dedicated customer Kafka topics, with all new Oso customer tenants receiving one. In order to minimize risk for existing customers, only new tenants were placed on these dedicated topics while we gained operational confidence and built support tooling. We have been gradually migrating existing customers to the new topics over the past few months. However, a bulk migration has not yet been done.
A customer who was still on the legacy shared topic performed a large number of expensive individual write requests to Oso Cloud, which caused the remaining customers on this legacy topic to have their writes queued for much longer than usual, resulting in delayed writes across all regions.
When the on-call team was made aware of the issue, they identified the customer responsible for the large number of expensive writes and worked with them to identify and cease the operations that erroneously introduced these expensive write requests (if the operations could not be identified, they were prepared to enforce hard write limits on that environment to stabilize the platform). In order to preserve data integrity, the team waited for multiple regions to catch up and stay caught up. Once they validated sustained low write delays in these regions, the team initiated a manual operation to bring the remaining delayed regions in sync with the recovered ones.
All times UTC
us-west-2) completes processing the bulk backlog and starts catching up.us-west-2 fully recovers. Other regions follow suit over the next 30 minutes, with additional ones still recovering.