https://blog.turboawesome.win/tags/distributed-systems/ Recent content in Distributed-Systems on Bits, Trades & Systems Hugo en-us Wed, 16 Apr 2025 13:12:00 +0000 https://blog.turboawesome.win/2025/04/distributed-consistency-models-what-your-service-actually-guarantees/ Wed, 16 Apr 2025 13:12:00 +0000 https://blog.turboawesome.win/2025/04/distributed-consistency-models-what-your-service-actually-guarantees/ Linearisability, serializability, eventual consistency, causal consistency — these terms are used loosely and understood imprecisely. Knowing what your data store actually guarantees determines whether your distributed system is correct. https://blog.turboawesome.win/2024/05/observability-at-scale-what-good-looks-like-when-you-have-too-much-data/ Wed, 29 May 2024 09:47:00 +0000 https://blog.turboawesome.win/2024/05/observability-at-scale-what-good-looks-like-when-you-have-too-much-data/ Observability problems at large scale are different from small-scale ones. Too little signal is replaced by too much signal, and the engineering challenge inverts. https://blog.turboawesome.win/2024/03/cache-design-as-a-reliability-practice-not-an-optimisation/ Wed, 27 Mar 2024 11:47:00 +0000 https://blog.turboawesome.win/2024/03/cache-design-as-a-reliability-practice-not-an-optimisation/ Most engineers add caches to make things faster. At scale, the more important reason to design caches carefully is reliability — a cache failure should not cascade into a system failure. The patterns that prevent that are different from the patterns that optimise for speed. https://blog.turboawesome.win/2024/02/engineering-at-enterprise-scale-what-changes-when-the-system-is-actually-big/ Wed, 14 Feb 2024 10:22:00 +0000 https://blog.turboawesome.win/2024/02/engineering-at-enterprise-scale-what-changes-when-the-system-is-actually-big/ After a decade in financial technology — trading firms, institutions, a startup, a European fintech — joining a large US technology company. The technical and organisational delta. https://blog.turboawesome.win/2022/05/opentelemetry-in-go-distributed-tracing-that-doesnt-get-in-the-way/ Tue, 31 May 2022 10:14:00 +0000 https://blog.turboawesome.win/2022/05/opentelemetry-in-go-distributed-tracing-that-doesnt-get-in-the-way/ OpenTelemetry standardised distributed tracing across languages and vendors. Here's what idiomatic OTel integration looks like in Go — and the parts that aren't obvious from the documentation. https://blog.turboawesome.win/2021/01/grpc-in-production-lessons-after-two-years/ Wed, 13 Jan 2021 14:55:00 +0000 https://blog.turboawesome.win/2021/01/grpc-in-production-lessons-after-two-years/ gRPC is a better default than REST for internal service communication. It's also a set of new problems you didn't have before. Two years of production experience. https://blog.turboawesome.win/2020/02/context-propagation-in-go-deadlines-cancellation-and-tracing/ Wed, 19 Feb 2020 11:14:00 +0000 https://blog.turboawesome.win/2020/02/context-propagation-in-go-deadlines-cancellation-and-tracing/ Go's context package is one of the most important idioms in the language. Used well it enables clean cancellation and distributed tracing. Used poorly it becomes a magic bag of values that creates invisible dependencies. https://blog.turboawesome.win/2018/10/schema-evolution-in-avro-the-hard-lessons-from-production/ Thu, 04 Oct 2018 11:29:00 +0000 https://blog.turboawesome.win/2018/10/schema-evolution-in-avro-the-hard-lessons-from-production/ Avro's schema evolution rules sound simple. In production with multiple services and a regulated data retention requirement, the edges are sharp. Here are the cases that burned us and the practices that prevented future ones. https://blog.turboawesome.win/2018/07/event-sourcing-in-financial-systems-real-benefits-real-costs/ Wed, 11 Jul 2018 11:03:00 +0000 https://blog.turboawesome.win/2018/07/event-sourcing-in-financial-systems-real-benefits-real-costs/ Event sourcing is a natural fit for financial systems that require audit trails and point-in-time reconstruction. The costs are real too — projections, eventual consistency, and the event schema evolution problem. https://blog.turboawesome.win/2018/06/backpressure-in-practice-keeping-fast-producers-from-killing-slow-consumers/ Thu, 14 Jun 2018 10:33:00 +0000 https://blog.turboawesome.win/2018/06/backpressure-in-practice-keeping-fast-producers-from-killing-slow-consumers/ Every system has components that produce faster than consumers can handle under some conditions. Backpressure is the mechanism by which fast producers are slowed rather than dropping data or consuming unbounded memory. Here's what the options look like in practice. https://blog.turboawesome.win/2018/01/distributed-transactions-are-a-lie-and-what-to-do-instead/ Wed, 17 Jan 2018 10:55:00 +0000 https://blog.turboawesome.win/2018/01/distributed-transactions-are-a-lie-and-what-to-do-instead/ Two-phase commit promises ACID semantics across distributed systems. In practice it's slow, fragile, and blocks under failure. The patterns that actually work — sagas, idempotency, and compensating transactions — are more complex but more reliable. https://blog.turboawesome.win/2017/10/building-mifid-ii-trade-reporting-infrastructure-an-engineers-view/ Tue, 03 Oct 2017 11:45:00 +0000 https://blog.turboawesome.win/2017/10/building-mifid-ii-trade-reporting-infrastructure-an-engineers-view/ MiFID II required every trade to be reported within 15 minutes of execution. Building the infrastructure to meet that requirement across a large, heterogeneous estate taught us about the gap between regulatory requirements and production reality. https://blog.turboawesome.win/2017/01/kafka-in-finance-what-exactly-once-actually-costs-you/ Tue, 10 Jan 2017 11:22:00 +0000 https://blog.turboawesome.win/2017/01/kafka-in-finance-what-exactly-once-actually-costs-you/ Kafka's exactly-once semantics arrived in 0.11 with significant caveats. Using it in a regulated financial context forced a clear-eyed view of what the guarantee actually covers and what it doesn't.