4 ms·

I was surprised to see LevelDB (https://code.google.com/p/leveldb/ https://code.google.com/p/leveldb/) was missing from the list of storage solutions you tried,

by atdt 13y ago

I was surprised to see LevelDB (https://code.google.com/p/leveldb/ https://code.google.com/p/leveldb/) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?

progx 13y ago
The problem is, that so many projects for so many things exists. It is hard to find the matching project, thats why many people invent the wheele again.
tinco 13y ago
Why do you think it seems optimal for their use case? A quick glance over LevelDB's features gives me the impression that its bulk-write performance would not be sufficient.
blippie 13y ago
I'm not sure about the optimal use-case match. Sparkey is for "mostly static" datasets where on disk structures are generated by a batch process and pushed to servers providing read only access to the data to consumers. leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.
- illumen 13y ago
  Also Sparkey seems to work well with bittorrent/rsync distribution. I recall spotify use bittorrent to distribute files to their servers.
sorbits 13y ago
According to GitHub, last commit (which is also initial checkin) is two years old.
- woogley 13y ago
  The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183453bdeaff1eb0da8ccf5adacb796e7 https://code.google.com/p/leveldb/source/detail?r=748539c183...
  - bouk 13y ago
    He means the last commit on sparkey
    - krka 13y ago
      Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.
  - iampims 13y ago
    There is also the basho fork of LevelDB (https://github.com/basho/leveldb https://github.com/basho/leveldb) which adds a few nice things. If you're interested in the internals of LevelDB, I strongly recommend watching this talk http://www.youtube.com/watch?v=vo88IdglU_8 http://www.youtube.com/watch?v=vo88IdglU_8 (slides here https://speakerdeck.com/basho/optimizing-leveldb-for-performance-and-scale-ricon-east-2013 https://speakerdeck.com/basho/optimizing-leveldb-for-perform...)
hyc_symas 13y ago
LevelDB has very few good use cases and this certainly isn't one of them. http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/ http://symas.com/mdb/hyperdex/ http://symas.com/mdb/hyperdex/

by atdt 13y ago

progx 13y ago

The problem is, that so many projects for so many things exists. It is hard to find the matching project, thats why many people invent the wheele again.

tinco 13y ago

Why do you think it seems optimal for their use case? A quick glance over LevelDB's features gives me the impression that its bulk-write performance would not be sufficient.

blippie 13y ago

I'm not sure about the optimal use-case match. Sparkey is for "mostly static" datasets where on disk structures are generated by a batch process and pushed to servers providing read only access to the data to consumers. leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.

illumen 13y ago
Also Sparkey seems to work well with bittorrent/rsync distribution. I recall spotify use bittorrent to distribute files to their servers.

sorbits 13y ago

According to GitHub, last commit (which is also initial checkin) is two years old.

woogley 13y ago
The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183453bdeaff1eb0da8ccf5adacb796e7 https://code.google.com/p/leveldb/source/detail?r=748539c183...
- bouk 13y ago
  He means the last commit on sparkey
  - krka 13y ago
    Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.
- iampims 13y ago
  There is also the basho fork of LevelDB (https://github.com/basho/leveldb https://github.com/basho/leveldb) which adds a few nice things. If you're interested in the internals of LevelDB, I strongly recommend watching this talk http://www.youtube.com/watch?v=vo88IdglU_8 http://www.youtube.com/watch?v=vo88IdglU_8 (slides here https://speakerdeck.com/basho/optimizing-leveldb-for-performance-and-scale-ricon-east-2013 https://speakerdeck.com/basho/optimizing-leveldb-for-perform...)

hyc_symas 13y ago

LevelDB has very few good use cases and this certainly isn't one of them. http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/ http://symas.com/mdb/hyperdex/ http://symas.com/mdb/hyperdex/