3 ms·
I think append only type logs like Kafka are more important from a semantic point of view than from an IO point of view. The importance of having an immutable,
by jillesvangurp 4y ago
I think append only type logs like Kafka are more important from a semantic point of view than from an IO point of view. The importance of having an immutable, append only data store is that it is immutable. That gives you some nice properties with consistency in especially distributed setups and synchronizing state between different nodes.
The IO write speed is a bit of a secondary concern. And since the data is immutable, you don't actually overwrite it all the time. And of course, SSDs have been around for quite long and spinning disk was already on the way out over a decade ago on high end servers. So, things like Kafka and event sourcing mostly became popular after that started happening; not before. This was never really about SSDs vs. spinning disks. Instead it is about the guarantees that come with having immutable data in a distributed setup. You see that in the Elasticsearch world as well (lucene uses append only data structures). Elasticsearch emerged around 2010. Almost from the beginning the consensus was that ssd was vastly preferable to spinning disks for scaling. Not because of the writes but for reads. Of course lots of people still used spinning disk around the time as well.
So, sequential reads and writes are two things. Elasticsearch writes sequentially; but reading is random access. Which is why SSDs are nice to have for Elasticsearch clusters even though it uses append only storage.
- yuhong 4y ago[dead]