4 ms·
Show HN: On-disk B+ tree for Python 3
- antman 9y agoCannot delete items yet
- nicolaslem 9y agoYes, it's a work in progress, see the `remove` branch.
- IvyMike 9y agoThat is a very dangerous name for a branch.
- liopleurodon 9y agolol, truth
- coleifer 9y agoWhy is this not mentioned very clearly in the README? Seems like willful misrepresentation. You might also mention that, if replacing large values that use overflow pages, the file has the potential to grow without bounds as it looks like overflow pages are not collected?
- dimatura 9y agoNice, I'm a big user of various data stores for scientific work. How does this compare to LMDB (http://www.lmdb.tech/doc/ http://www.lmdb.tech/doc/), which uses B-trees (and has a Python interface)?
- nicolaslem 9y agoI haven't done any benchmark yet but I expect my implementation to be at least an order of magnitude slower. I've found my implementation to be CPU intensive: creating Python objects from the raw pages is expensive. That's why bulk inserts and iterations are much faster than insert/get in a loop.
- halayli 9y agoWhat's wrong with LMDB?
- coleifer 9y agoThis is pretty cool. As I was reading the code I really wished for an explanation or simple ASCII diagram of the serialization formats for the various node/record types, as well as for the frames/wal format. Given that the poster is the author of the project, I hope you'll consider filling in these kinds of details, as they'd presumably be of interest to the people you're "Show"-ing this to.
- nicolaslem 9y agoThank you. I know what you mean, I'm not happy with how the serialization is done right now, it's too complicated. Maybe someone on HN knows a Python serialization library beyond pickle that would allow to describe how the data is laid out and take care of the rest. It looks like struct is not flexible enough for this usage.
- zerokernel 9y agohttps://pypi.python.org/pypi/BTrees https://pypi.python.org/pypi/BTrees
- laurencerowe 9y agoZODB has a mature B+Tree implementation for on disk use in the BTrees package. https://pypi.python.org/pypi/BTrees https://pypi.python.org/pypi/BTrees
- coleifer 9y agoThanks for sharing, I've been interested in finding pure python implementations of storage engines. Another one I came across is whoosh: https://bitbucket.org/mchaput/whoosh/src/a16ebacb47191afaf2d1fda8b1fcd84e6981582f/src/whoosh/filedb/?at=default https://bitbucket.org/mchaput/whoosh/src/a16ebacb47191afaf2d...
- shalabhc 9y agoCheck out zodb and durus.
- sprt 9y agoThe code looks very clean. Uses type annotations too. Wish there were more comments though.
- nicolaslem 9y agoThank you! I found that type annotations make code easier to read so I tend to write less comments. But I'll make an effort.
- j_s 9y agoAre there any production data stores recommended for low memory usage? What should I use to stream data to disk and back with minimal overhead, preferably with indexed lookups?
- bufferoverflow 9y agoAn SSD with the highest random-read/write IOPS you can afford. Something like this: https://www.computerworld.com/article/2987956/solid-state-drives/intel-samsung-launch-their-fastest-ssds-with-up-to-5gbps-speeds.html https://www.computerworld.com/article/2987956/solid-state-dr... Or FusionIO ioDrive2 Duo, which is above 900K IOPS for both reads and writes.
- evolighting 9y agowhat about using a DB? Is there any really simple database for the same purpose?
- erezsh 9y agoHow does this perform compared to sqlite3? And and can you use keys that aren't builtin values?
- nicolaslem 9y agoIt is possible to use your own key if it has a natural order and you write your own simple serializer: https://github.com/NicolasLM/bplustree/blob/master/bplustree/serializer.py https://github.com/NicolasLM/bplustree/blob/master/bplustree...