https://blog.turboawesome.win/categories/systems/ Recent content in Systems on Bits, Trades & Systems Hugo en-us Wed, 07 May 2025 11:00:00 +0000 https://blog.turboawesome.win/2025/05/evaluating-llm-integrated-systems-what-works-and-what-doesnt/ Wed, 07 May 2025 11:00:00 +0000 https://blog.turboawesome.win/2025/05/evaluating-llm-integrated-systems-what-works-and-what-doesnt/ LLM outputs are probabilistic and context-dependent. The testing and evaluation approaches from deterministic software don't transfer directly. What does work: eval datasets, LLM-as-judge, regression suites, and the practices that separate teams with confidence from teams flying blind. https://blog.turboawesome.win/2025/04/distributed-consistency-models-what-your-service-actually-guarantees/ Wed, 16 Apr 2025 13:12:00 +0000 https://blog.turboawesome.win/2025/04/distributed-consistency-models-what-your-service-actually-guarantees/ Linearisability, serializability, eventual consistency, causal consistency — these terms are used loosely and understood imprecisely. Knowing what your data store actually guarantees determines whether your distributed system is correct. https://blog.turboawesome.win/2024/10/tail-based-trace-sampling-why-head-sampling-is-usually-wrong/ Wed, 09 Oct 2024 13:00:00 +0000 https://blog.turboawesome.win/2024/10/tail-based-trace-sampling-why-head-sampling-is-usually-wrong/ Head-based sampling decides whether to trace a request at the start. Tail-based sampling decides after the request completes. For finding latency outliers and errors, tail-based sampling is almost always what you want — and almost never what gets implemented. https://blog.turboawesome.win/2024/09/rag-systems-in-production-what-the-tutorials-dont-cover/ Wed, 11 Sep 2024 10:44:00 +0000 https://blog.turboawesome.win/2024/09/rag-systems-in-production-what-the-tutorials-dont-cover/ Retrieval-Augmented Generation works well in demos and breaks in interesting ways in production. The gap between 'it answers questions' and 'it reliably answers questions correctly' is where most of the engineering lives. https://blog.turboawesome.win/2024/07/llm-integration-patterns-for-backend-engineers/ Wed, 10 Jul 2024 09:38:00 +0000 https://blog.turboawesome.win/2024/07/llm-integration-patterns-for-backend-engineers/ Integrating LLMs into backend systems requires engineering discipline that the AI ecosystem tutorials often skip. Structured output, function calling, retry strategy, and testing patterns from production. https://blog.turboawesome.win/2024/05/observability-at-scale-what-good-looks-like-when-you-have-too-much-data/ Wed, 29 May 2024 09:47:00 +0000 https://blog.turboawesome.win/2024/05/observability-at-scale-what-good-looks-like-when-you-have-too-much-data/ Observability problems at large scale are different from small-scale ones. Too little signal is replaced by too much signal, and the engineering challenge inverts. https://blog.turboawesome.win/2024/05/evaluating-llm-applications-why-it-looks-good-is-not-enough/ Tue, 14 May 2024 14:22:00 +0000 https://blog.turboawesome.win/2024/05/evaluating-llm-applications-why-it-looks-good-is-not-enough/ LLM applications fail in ways that traditional software testing doesn't catch. Building evaluation frameworks that give you real signal about quality — before and after deployment — is the engineering challenge that separates serious AI products from demos. https://blog.turboawesome.win/2024/03/cache-design-as-a-reliability-practice-not-an-optimisation/ Wed, 27 Mar 2024 11:47:00 +0000 https://blog.turboawesome.win/2024/03/cache-design-as-a-reliability-practice-not-an-optimisation/ Most engineers add caches to make things faster. At scale, the more important reason to design caches carefully is reliability — a cache failure should not cascade into a system failure. The patterns that prevent that are different from the patterns that optimise for speed. https://blog.turboawesome.win/2024/02/engineering-at-enterprise-scale-what-changes-when-the-system-is-actually-big/ Wed, 14 Feb 2024 10:22:00 +0000 https://blog.turboawesome.win/2024/02/engineering-at-enterprise-scale-what-changes-when-the-system-is-actually-big/ After a decade in financial technology — trading firms, institutions, a startup, a European fintech — joining a large US technology company. The technical and organisational delta. https://blog.turboawesome.win/2023/10/gos-race-detector-in-ci-catching-data-races-before-they-catch-you/ Wed, 04 Oct 2023 09:35:00 +0000 https://blog.turboawesome.win/2023/10/gos-race-detector-in-ci-catching-data-races-before-they-catch-you/ Data races are among the hardest bugs to find and reproduce. Go's built-in race detector finds them automatically — if you run it. Here's how to integrate it into CI effectively and what to do when it fires. https://blog.turboawesome.win/2023/07/gos-net/http-building-production-http-servers-without-a-framework/ Wed, 05 Jul 2023 10:44:00 +0000 https://blog.turboawesome.win/2023/07/gos-net/http-building-production-http-servers-without-a-framework/ Go's standard net/http package is capable enough for most production workloads without a framework. What 'capable enough' means in practice, and the patterns for middleware, graceful shutdown, and observability. https://blog.turboawesome.win/2023/05/clickhouse-for-application-analytics-fast-aggregations-without-spark/ Wed, 17 May 2023 14:08:00 +0000 https://blog.turboawesome.win/2023/05/clickhouse-for-application-analytics-fast-aggregations-without-spark/ When you need sub-second aggregations over billions of rows and don't want to run a Spark cluster, ClickHouse is often the answer. Notes from a year in production. https://blog.turboawesome.win/2023/04/database-access-patterns-in-go-sqlx-pgx-and-when-to-use-an-orm/ Wed, 05 Apr 2023 11:22:00 +0000 https://blog.turboawesome.win/2023/04/database-access-patterns-in-go-sqlx-pgx-and-when-to-use-an-orm/ The Go database access landscape — database/sql, sqlx, pgx, GORM, sqlc — each makes different tradeoffs. Here's what works for different use cases and why the common advice to avoid ORMs is sometimes right and sometimes not. https://blog.turboawesome.win/2023/02/feature-flags-at-scale-beyond-the-on/off-switch/ Tue, 28 Feb 2023 11:02:00 +0000 https://blog.turboawesome.win/2023/02/feature-flags-at-scale-beyond-the-on/off-switch/ Feature flags sound simple — toggle code on or off. At scale they're a deployment primitive, an experimentation platform, a kill-switch infrastructure, and a source of technical debt. Here's what a mature feature flag system looks like and how to avoid the traps. https://blog.turboawesome.win/2022/11/structured-logging-in-go-slog-zerolog-and-what-actually-matters/ Wed, 23 Nov 2022 10:58:00 +0000 https://blog.turboawesome.win/2022/11/structured-logging-in-go-slog-zerolog-and-what-actually-matters/ The Go ecosystem has converged on structured logging, but the choice of library and the conventions you adopt matter more than the specific API. Here's what structured logging should look like in a production Go service and why. https://blog.turboawesome.win/2022/08/memory-layout-in-go-structs-alignment-and-cache-performance/ Wed, 17 Aug 2022 10:03:00 +0000 https://blog.turboawesome.win/2022/08/memory-layout-in-go-structs-alignment-and-cache-performance/ Go's struct layout follows alignment rules that affect both memory consumption and cache performance. Understanding them matters more than most Go developers realise. https://blog.turboawesome.win/2022/05/opentelemetry-in-go-distributed-tracing-that-doesnt-get-in-the-way/ Tue, 31 May 2022 10:14:00 +0000 https://blog.turboawesome.win/2022/05/opentelemetry-in-go-distributed-tracing-that-doesnt-get-in-the-way/ OpenTelemetry standardised distributed tracing across languages and vendors. Here's what idiomatic OTel integration looks like in Go — and the parts that aren't obvious from the documentation. https://blog.turboawesome.win/2022/05/kafka-at-startup-scale/ Wed, 18 May 2022 14:00:00 +0000 https://blog.turboawesome.win/2022/05/kafka-at-startup-scale/ Running Kafka at a startup is different from running it at enterprise scale. The operational complexity is real, the defaults are wrong for small clusters, and the failure modes are different from what the documentation implies. https://blog.turboawesome.win/2021/11/building-reliable-pipelines-with-go-retry-circuit-breaker-and-backoff/ Wed, 17 Nov 2021 11:41:00 +0000 https://blog.turboawesome.win/2021/11/building-reliable-pipelines-with-go-retry-circuit-breaker-and-backoff/ Every service that calls another service will eventually face failures. The patterns that contain those failures — retry with backoff, circuit breakers, bulkheads — are not optional for production Go services. Here's how to implement them correctly. https://blog.turboawesome.win/2021/01/grpc-in-production-lessons-after-two-years/ Wed, 13 Jan 2021 14:55:00 +0000 https://blog.turboawesome.win/2021/01/grpc-in-production-lessons-after-two-years/ gRPC is a better default than REST for internal service communication. It's also a set of new problems you didn't have before. Two years of production experience.