3 ms·
You don't need to scale for performance, but you do for reliability. SQL databases still tend to rely on having one server that never goes down, or at best havi
by lmm 4y ago
You don't need to scale for performance, but you do for reliability. SQL databases still tend to rely on having one server that never goes down, or at best having a replica and a semi-manual "promotion" procedure.
Kafka does what a datastore should and nothing more. It's what databases should have been. High performance is a consequence of that good design, but the good design is valuable in itself.
I would always start with Kafka. There are so many ways to shoot yourself in the foot with an SQL database that will bite you later, sometimes in a way that may corrupt your data permanently (e.g. read-modify-write logic), and you'll almost certainly end up having to change your data architecture to the one you'd've started with if you were using Kafka. Or you can just throw it in Kafka from day 1 and a lot of bad ideas just become impossible because the system gives you no way to do them.
I was also working on phone systems in the 2000's that ran at "web scale". We used MySQL, but we essentially built Kafka on top of MySQL - tables for every intermediate step in the process, and a way for the next step in the processing pipeline to mark which rows it had started/finished consuming.