6 ms·
Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB? Thus far, I've been pressing SQLite into use for a lot of monotyped data that w
by qwertyuiop924 10y ago
Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB? Thus far, I've been pressing SQLite into use for a lot of monotyped data that would have been better served by a NoSQL DB, but I couldn't find one suitable.
Apparently, in-memory DBs in now. They're fast and all, that's nice, but I'm still running on a machine with 8 Gigs of ram, and when the dataset starts to exceed that, I want an option other than "grab some more DDR3." I remember the fate of TinyMUD: on-disk operation must at least be an option for any DB I'd use for datasets of a non-fixed size.
- brightball 10y agoMnesia built into Elixir/Erlang potentially?
- misframer 10y agoThere are several key-value stores you can choose from. RocksDB, LevelDB, lmdb, Berkeley DB, Tokyo Cabinet, sophia, and others. What about those? You can always build abstractions on top.
- lopatin 10y agoBoltDB if you'd like to embed it into a Go program.
- chaotic-good 10y agoBoltDB can be corrupted on crash (kill -9 or hard reset) at least this was the case when I gave it a try. Any database can but in that case that was easy to achieve.
- mbertschler 10y agodid you file a bug for this? I thought it was pretty resilient in such cases
- chaotic-good 10y agoI've found several submitted issues that described this problem. All this issues are closed now so maybe I was wrong about Bolt.
- qwertyuiop924 10y agoThanks for the suggestions. Some of those might work. The only problem with k/v stores is the serialization/deserialization cost, as most of them can only store strings.
- ddorian43 10y agolmdb has some special cases where you can store capnproto/flat-buffers objects and read without copying/deserializing!
- qwertyuiop924 10y agoReally? that could be useful...
- ddorian43 10y agoLMDB can hand you back a pointer to your object in the memory map, and we can easily use that pointer with Cap'n Proto without any calls to malloc(). source: https://news.ycombinator.com/item?id=12394385 https://news.ycombinator.com/item?id=12394385
- erichocean 10y ago> The only problem with k/v stores is the serialization/deserialization cost, as most of them can only store strings. I store Cap'n Proto objects directly into LMDB, and can read them without any memory allocation or deserialization cost. Freaking rocks.
- coleifer 10y agohttp://engine.so/ http://engine.so/ -- quite a nice list of small, fast, in-process databases.
- maerF0x0 10y agomake a gigantic swap file?
- deleted 10y ago[deleted]
- chaotic-good 10y agoYou can try LMBD - https://symas.com/products/lightning-memory-mapped-database/ https://symas.com/products/lightning-memory-mapped-database/ In my opinion there is exactly two production ready embedded databases - SQLite and LMDB. At least this two databases have decent crash recovery mechanisms (I've tested both). I haven't tested all of them, maybe ForestDB and BerkleyDB are good as well.
- qwertyuiop924 10y agoThe only question is this: Is the dataset required to fit into RAM? Because if it is, than I haven't fixed the problem, now have I...
- erichocean 10y ago> The only question is this: Is the dataset required to fit into RAM? I run an LMDB dataset many times the size of RAM, works great. Most data isn't recent, so the virtual memory cache works extremely well.
- qwertyuiop924 10y agoOkay. Good.
- bruno2223 10y agoI use LevelDB, it is a key-value database, from Google. I am using it with Nodejs, on t2.micro, handling 10+ million requests / day Never had an issue. in production for one year already. Really good :-)
- qwertyuiop924 10y agoLevel could work, but it depends on how expensive serialization is.
- tracker1 10y agoThere's also a lot of higher level abstractions over leveldb at this point.
- alixaxel 10y agoHave you heard about TieDot?
- erichocean 10y ago> Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB? That's LMDB, as others have noted. I have used it in production for years, never once lost data. Insanely fast, low footprint, crash-proof. (P.s. I have seen SQLite get corrupted regularly on mobile devices, which is why I no longer use it there.)
- qwertyuiop924 10y agoBizarre. SQLite is known for being hard to corrupt. How did it happen?
- erichocean 10y agoNo idea, but with about 50,000 regular users hitting the local SQLite DB on their mobile devices hard, I'd get a corrupted DB report a few times a week. It's possible my SQLite configuration wasn't as crash-proof as the default install. I moved onto another project before I could investigate further.
- not_kurt_godel 10y agoSeems possible you were just seeing the rate at which people's storage hardware gets corrupted rather than an issue with SQLite.