4 ms·
Redis: new disk storage to replace VM
- snissn 16y agosounds like he's trying to implement something similar to a 'dbm' (tokyo/kyoto as current/modern implementations) which are k/v caches that seem to intelligently write to disk. Presumably the keyset can still be in ram (so that [star]foo[star]bar[star]1 searches on keys can work?), but the dbm model is a fairly efficient key/val implementation, and tokyo/kyoto is fast, and fairly smart about writing to disk, although I haven't explicitly testing their limitations as you approach ram limits in production. Not sure what tradeoffs are in mind, but atleast a feature/perf comparison to kyoto compared with diskstore as an internal back for redis would be interesting [1](doesnt seem i can escape an *)
- pashields 16y agoI don't think the keyset will be in ram: "Redis will never use more RAM, even if we have 2 MB of max memory and 1 billion of keys. This works since now we don't need to take keys in memory." I haven't looked at the current code to see if there is a way to favor keeping the keys in memory, but it would seem that wildcard searches here can/will be disk-bound.
- snissn 16y agotokyo supports a b-tree index on the keys written to disk which would optimize blah* queries but not [wildcard]foo[wildcard], and then writes become log(N)
- ShabbyDoo 16y agoSo, as a non-Redis user, am I correct in my understanding that an admin may define a maximum delay before write-behind persistence must be attempted? For applications which can afford to lose a few seconds of data in the case of a failure, this seems like a great way to improve latency. More generally, why are there no MySQL (or whatever) engines which offer similar capabilities? Wouldn't it be possible for DB clients to "commit" transactions to memory and then have them flushed to disk asynchronously with all other ACI(d) properties maintained? There are many applications which can survive a few seconds of data loss but need transactional properties to avoid data corruption.
- siculars 16y agoThis is kinda what mongodb[0] does now and is a question of durability[1]. Current rdbms systems that lay claim to consistency must flush to disk to ensure ACID[2] compliance. Redis does not claim such compliance and I am of the opinion that it should not. [0]http://www.mongodb.org/ http://www.mongodb.org/ [1]http://en.wikipedia.org/wiki/Durability_(database_systems) http://en.wikipedia.org/wiki/Durability_(database_systems) [2]http://en.wikipedia.org/wiki/ACID http://en.wikipedia.org/wiki/ACID
- justinsb 16y agoAll modern relational databases implement exactly what you've described... transactions are written to a transaction log, which is flushed to disk every few seconds (or whenever you want to guarantee that a txn is durable). Changes to the actual data need not be persisted in a timely manner, because in the event of a crash the data is recovered from the transaction log.
- dorianj 16y agoOn MySQL / InnoDB, this is innodb_flush_log_at_trx_commit and how the buffer log is flushed can have a tremendous impact on the latency of writes.
- ShabbyDoo 16y agoSo, no physical disk write need occur before a client can continue with processing? If so, cool.
- electrum 16y agohttp://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.htm... ``If the value of innodb_flush_log_at_trx_commit is 0, the log buffer is written out to the log file once per second and the flush to disk operation is performed on the log file, but nothing is done at a transaction commit.''
- 16y ago
- dchest 16y agoSee also: http://news.ycombinator.com/item?id=2053594 http://news.ycombinator.com/item?id=2053594
- nas 16y agoAre people aware of Python object databases like ZODB and Durus? I'm not very familiar with Redis. However, the model used by ZODB and Durus (on disk durable storage, in memory client caches) can be extremely efficient depending on workloads.
- IgorPartola 16y agoI am curious as to what people who use Redis in production think of these types of changes. Is this alarming or hopeful? Seems like a rather large shift in trade-offs and a whole new set of tuning parameters to play with.
- jashkenas 16y agoAlarming, I'd imagine: https://groups.google.com/forum/#!topic/redis-db/ZTSm-1w-6AQ https://groups.google.com/forum/#!topic/redis-db/ZTSm-1w-6AQ To quote from the end of the post: so, to sum this up -- after a while, you are stuck with an in-memory database that you cannot backup, cannot replicate to a standby machine, and that will eventually consume all memory and crash (if it does not crash earlier). conclusion: redis with vm enabled is pretty much unusable, and we would really not recommend it to anybody else for production use at the moment. (at least not as a database, it might work better as a cache.)
- madlep 16y agoActually, that is talking about the existing virtual-memory (VM) implementation, which swaps data in and out to disk, and doesn't work so great. The change being talked about here is all about replacing that exact flakey VM with a more solid disk backed approach
- jashkenas 16y agoI'm sorry, but that's what I meant. I'd be more alarmed than enticed to discover that the current implementation of datasets-larger-than-RAM for my chosen database was considered "flakey", and was going to be swapped out for a green-field approach in the next release. For reference, this is the blog post that introduced the VM idea: http://antirez.com/post/redis-virtual-memory-story.html http://antirez.com/post/redis-virtual-memory-story.html
- samstokes 16y agoIf Redis was "my chosen database" then I should have done my homework better. Redis has always been billed as an in-RAM database, which hard-implies that your dataset must fit in RAM, and thus that Redis is unlikely to be your only database. VM - i.e. even the possibility of having a dataset larger than RAM - is a recent addition, and one flagged as experimental. It had some issues which meant (for us and for the author of the post you linked to above) Redis was still not feasible for datasets larger than RAM. This new diskstore model, and the associated rethink, is very encouraging news, as it means that scenario might one day be possible after all.