Consumers request blocks of messages starting at a given offset. If the consumers are keeping up then memory pages of message can be served directly from the RA

Consumers request blocks of messages starting at a given offset. If the consumers are keeping up then memory pages of message can be served directly from the RAM buffer they were produced into, since the broker is constantly appending new messages a memory-mapped file. Producers can request acknowledgements, which they get after the buffer has been sync'd to disk. In practice a broker uses very little CPU even with compression and compaction happening, and is bound by the disk or network speed--our brokers on AWS run about 80MByte/sec with room on top for bursts to ~110. Interestingly and in addition, blocks of messages can pass through Kafka without being decompressed because of this design. Everything happens from one port on the brokers, who are backed with Zookeeper for cluster management; consumers may also connect to Zookeeper or similar to track the latest offset that they have processed.