https://blog.turboawesome.win/tags/observability/ Recent content in Observability on Bits, Trades & Systems Hugo en-us Tue, 07 Oct 2025 09:15:00 +0000 https://blog.turboawesome.win/2025/10/migrating-a-production-service-from-zap-to-slog-notes-from-the-trenches/ Tue, 07 Oct 2025 09:15:00 +0000 https://blog.turboawesome.win/2025/10/migrating-a-production-service-from-zap-to-slog-notes-from-the-trenches/ log/slog landed in the standard library in Go 1.21, but most production services are still on zap or zerolog. Here is what an actual migration looks like: where slog wins, where zap still wins, and how to move incrementally without a big-bang rewrite. https://blog.turboawesome.win/2025/05/evaluating-llm-integrated-systems-what-works-and-what-doesnt/ Wed, 07 May 2025 11:00:00 +0000 https://blog.turboawesome.win/2025/05/evaluating-llm-integrated-systems-what-works-and-what-doesnt/ LLM outputs are probabilistic and context-dependent. The testing and evaluation approaches from deterministic software don't transfer directly. What does work: eval datasets, LLM-as-judge, regression suites, and the practices that separate teams with confidence from teams flying blind. https://blog.turboawesome.win/2024/10/tail-based-trace-sampling-why-head-sampling-is-usually-wrong/ Wed, 09 Oct 2024 13:00:00 +0000 https://blog.turboawesome.win/2024/10/tail-based-trace-sampling-why-head-sampling-is-usually-wrong/ Head-based sampling decides whether to trace a request at the start. Tail-based sampling decides after the request completes. For finding latency outliers and errors, tail-based sampling is almost always what you want — and almost never what gets implemented. https://blog.turboawesome.win/2024/05/observability-at-scale-what-good-looks-like-when-you-have-too-much-data/ Wed, 29 May 2024 09:47:00 +0000 https://blog.turboawesome.win/2024/05/observability-at-scale-what-good-looks-like-when-you-have-too-much-data/ Observability problems at large scale are different from small-scale ones. Too little signal is replaced by too much signal, and the engineering challenge inverts. https://blog.turboawesome.win/2022/11/structured-logging-in-go-slog-zerolog-and-what-actually-matters/ Wed, 23 Nov 2022 10:58:00 +0000 https://blog.turboawesome.win/2022/11/structured-logging-in-go-slog-zerolog-and-what-actually-matters/ The Go ecosystem has converged on structured logging, but the choice of library and the conventions you adopt matter more than the specific API. Here's what structured logging should look like in a production Go service and why. https://blog.turboawesome.win/2022/05/opentelemetry-in-go-distributed-tracing-that-doesnt-get-in-the-way/ Tue, 31 May 2022 10:14:00 +0000 https://blog.turboawesome.win/2022/05/opentelemetry-in-go-distributed-tracing-that-doesnt-get-in-the-way/ OpenTelemetry standardised distributed tracing across languages and vendors. Here's what idiomatic OTel integration looks like in Go — and the parts that aren't obvious from the documentation.