4 ms·

What makes Apache Kafka fast (2020)

• the_gipsy 6y ago
  Can't read wihout signing up.
  • jwow_mat 6y ago
    You can use an incognito window.
    • the_gipsy 6y ago
      Or they could host it on any site that's not user hostile.
• zihotki 6y ago
  TLDR: They use many good industry practices: - Append-only log with sequential IO - Record batching on client and broker side (works well with sequential IO) - Batch compression (Json and other text formats could be compressed very well) - Unflushed buffered writes - Zero-copy approach for working with data In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides. The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that. Edit: formatting
• kovac 6y ago
  > "For those unfamiliar with the term, ‘real-time’ does not mean ‘fast’, it means ‘predictable’." So, if a system is predictable up to an upper bound of one year, is that still real-time? My understanding of real-time was pretty much as quickly as something happens with hard constraints on latency. I think "fast" is implied when we say real-time.