https://blog.turboawesome.win/tags/kafka/ Recent content in Kafka on Bits, Trades & Systems Hugo en-us Wed, 18 May 2022 14:00:00 +0000 https://blog.turboawesome.win/2022/05/kafka-at-startup-scale/ Wed, 18 May 2022 14:00:00 +0000 https://blog.turboawesome.win/2022/05/kafka-at-startup-scale/ Running Kafka at a startup is different from running it at enterprise scale. The operational complexity is real, the defaults are wrong for small clusters, and the failure modes are different from what the documentation implies. https://blog.turboawesome.win/2018/10/schema-evolution-in-avro-the-hard-lessons-from-production/ Thu, 04 Oct 2018 11:29:00 +0000 https://blog.turboawesome.win/2018/10/schema-evolution-in-avro-the-hard-lessons-from-production/ Avro's schema evolution rules sound simple. In production with multiple services and a regulated data retention requirement, the edges are sharp. Here are the cases that burned us and the practices that prevented future ones. https://blog.turboawesome.win/2018/06/backpressure-in-practice-keeping-fast-producers-from-killing-slow-consumers/ Thu, 14 Jun 2018 10:33:00 +0000 https://blog.turboawesome.win/2018/06/backpressure-in-practice-keeping-fast-producers-from-killing-slow-consumers/ Every system has components that produce faster than consumers can handle under some conditions. Backpressure is the mechanism by which fast producers are slowed rather than dropping data or consuming unbounded memory. Here's what the options look like in practice. https://blog.turboawesome.win/2017/10/building-mifid-ii-trade-reporting-infrastructure-an-engineers-view/ Tue, 03 Oct 2017 11:45:00 +0000 https://blog.turboawesome.win/2017/10/building-mifid-ii-trade-reporting-infrastructure-an-engineers-view/ MiFID II required every trade to be reported within 15 minutes of execution. Building the infrastructure to meet that requirement across a large, heterogeneous estate taught us about the gap between regulatory requirements and production reality. https://blog.turboawesome.win/2017/09/stream-processing-with-kafka-streams-vs-flink-a-real-comparison/ Wed, 27 Sep 2017 14:02:00 +0000 https://blog.turboawesome.win/2017/09/stream-processing-with-kafka-streams-vs-flink-a-real-comparison/ We evaluated Kafka Streams and Apache Flink for a real-time trade enrichment and aggregation pipeline. The technical comparison produced a clear result; the operational comparison was more nuanced. https://blog.turboawesome.win/2017/01/kafka-in-finance-what-exactly-once-actually-costs-you/ Tue, 10 Jan 2017 11:22:00 +0000 https://blog.turboawesome.win/2017/01/kafka-in-finance-what-exactly-once-actually-costs-you/ Kafka's exactly-once semantics arrived in 0.11 with significant caveats. Using it in a regulated financial context forced a clear-eyed view of what the guarantee actually covers and what it doesn't. https://blog.turboawesome.win/2015/01/chronicle-queue-vs-kafka-choosing-a-persistent-journal-at-nanosecond-scale/ Wed, 21 Jan 2015 10:44:00 +0000 https://blog.turboawesome.win/2015/01/chronicle-queue-vs-kafka-choosing-a-persistent-journal-at-nanosecond-scale/ Both Chronicle Queue and Kafka provide persistent, ordered message logs. Their performance profiles, operational models, and use cases are almost completely different.