3 ms·
It's not only the "VM" experiments but generally the very basic approach to persistence that's kept Redis from being an actual database and has most users (hope
by thamer 5y ago
It's not only the "VM" experiments but generally the very basic approach to persistence that's kept Redis from being an actual database and has most users (hopefully) only using it as a cache.
Around the same period as this blog post antirez had also announced that he had written a B-Tree that could be potentially used to store Redis data. The code is still up: https://github.com/antirez/otree https://github.com/antirez/otree – needless to say serious storage engines like LevelDB or RocksDB are more than a single 1,000-line C file, while this looked more like a weekend project.
The persistence model that has stayed in Redis is also a pretty basic approach: fork + dump all memory to a single file. Write every command to a file and compact it by doing the same fork + scan/flush. None of this is efficient, and with a tens of GB of cache you get long pauses during fork() as the page table gets copied to the child process (yes just the table, not the memory pages themselves).
All this to say that his advice on disk I/O is to be taken in the context of his own experience in this domain.
- jasonwatkinspdx 5y agoYeah, I completely agree. Redis makes a great cache with richer operations, but less efficiency than memcached. Using it as a system of record in any context will come back to haunt you in the worst way. I first ran into the more minimal append only log persistence pattern with the jvm project Prevayler. It's not a totally unreasonable idea but has to be implemented carefully. The fork as copy on write mechanism initially seems clever, but in practice is extremely brittle. There's a reason classic databases have a lot of sophisticated logic around checkpointing and truncating the write ahead log. It's not a problem that can be oversimplified. I wouldn't be totally down on minimal btree implementations, but 1000 LOC is pushing it. Even LMDB as minimalist as it is, is around 10k as I recall.