I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events. Wh

I think the focus on the $/GB is wrong. The main goal of Kafka is providing certain processing guarantees and connecting consumers with producers of events. What Kafka is not for is dealing with enormous number of events or enormous amount of data. Not that it is particularly slow at it (actually, quite fast) but I see a lot of people needlessly try to push more data and events than they need and then complain they have problems. One easy trick to deal with enormous numbers of events and amounts of data is batching. Just batch some number of events into a single event so that a consumer can pick it up in one go. Removes a lot of cost of transferring the event. Too much data to transfer? Don't be stupid and try to transfer those huge documents through the Kafka topic -- there is no need for this. Just upload them to S3 and pass through a reference. Or even better, take 10k events, extract data, zip it up, push to S3, and THEN pass a single event that describes 10k large events.