3 ms·

You are probably right. But then you come across the 1% case and Kafka is the only thing that you can throw at the problem that works. Not sure of exact numbe

by buro9 9y ago

You are probably right. But then you come across the 1% case and Kafka is the only thing that you can throw at the problem that works. Not sure of exact numbers, but I think we're doing 4-5m writes per second across one Kafka cluster, and around 1m writes per second against another. Most people don't have big data, but some people do.

cookiecaper 9y ago
Another caveat on this. Some people are forced into "big data" solutions not because their data is inherently "big data", but because their applications output data with wanton disregard for its arrangement or structure, making it so only systems designed to manage massive streams can readily handle it. Many databases that I regularly touch would shrink an order of magnitude or more if someone went through, redid the layout, and scripted a process to perform the migration.
e12e 9y ago
> Not sure of exact numbers, but I think we're doing 4-5m writes per second across one Kafka cluster, and around 1m writes per second against another. I'm not saying that's the case with your system, but my immediate thought when I see those numbers is: how much of the value from those 4m/s writes could you get with a system that did something like 100 w/s? Either through sampling/statistical methods, reduce "write amplification" or simply looking hard at what is written and how often :-)