11 ms·
TIL–Python has a built-in persistent key-value store (2018)
- blibble 4y agothe gnu version of dbm has some nice surprises if you go and look at the binary database you'll see random other bits of memory in there, because it doesn't initialise most of its buffers
- nonethewiser 4y agoHuh. I am working on a little utility to deduplicate csvs using sqlite. Seems like this would likely be better. I used sqlite in the projects name. This makes me think of an article I read recently that suggested not using descriptive names for projects. For reasons such as this.
- ivvve 4y agoThis is probably dumb intervention but forgive me because it made me giddy to find: the pandas drop_duplicates() method is pretty great, if you're in a pinch. Again -- forgive me if you know about it!
- ALittleLight 4y agoYou may want to look at pandas too (unless the issue is your CSVs are too large to fit in memory).
- karteum 4y agoif the lines are strictly identical, you may just use cat foo.csv | sort | uniq
- j0hnyl 4y agoI wonder if it's still faster than sqlite in 2022.
- calpaterson 4y agoAccording to his benchmark run today on Python 3.9 on linux, dbm is still considerably faster for random access. That said, what database dbm is using varies a lot by platform. I think on linux it's usually BerkleyDB. I use dbm a lot to cache pickles of object that were slow to create. eg pandas dataframes
- curiousgal 4y agoSorry if it's a dumb question but sinc dbm writes stuff to disk, how is that different than loading the pickles?
- dfox 4y agoYou get one file for the whole store, not one per key. Also, it tends to better handle having lots of keys significantly better than filesystems handle lots of files in one directory.
- curiousgal 4y agoThank you!
- qalmakka 4y agoI think on Linux is often GNU DBM, Berkeley DB is a different, better implementation by Oracle that requires a third party module.
- chaxor 4y ago...it's... Better? But... It's Oracle... https://youtu.be/-zRN7XLCRhc https://youtu.be/-zRN7XLCRhc
- cbdumas 4y agoThe script linked at the bottom of this page uses sqlite without an index on the table. In my testing if you add an index sqlite quickly outperforms dbm in terms of read performance
- deleted 4y ago[deleted]
- im3w1l 4y agoAs far as I can tell this doesn't support transactions, which imho is table stakes for a persistence solution in 2022.
- eesmith 4y agoFWIW, it was added in 1992. https://github.com/python/cpython/commit/dd9ed839d6958f737045306488995a1097df93b8 https://github.com/python/cpython/commit/dd9ed839d6958f73704... with gdbm added in 1994. https://github.com/python/cpython/commit/4b4c664d2e93279c8d749da027000453f9e2cd46 https://github.com/python/cpython/commit/4b4c664d2e93279c8d7... .
- im3w1l 4y agoThe article makes it sound like it's something to consider today (well 2018), that's why I think it's fair to see how well it satisfies modern requirements.
- ok123456 4y agoIf you just need a simple key value store that stays on disk, it's fine.
- jonnycomputer 4y agorequirements for what is the question. "modern requirements" doens't mean that much.
- im3w1l 4y agoWithout transactions the data can become corrupt in case power fails / the process is killed / the container is restarted.
- pjc50 4y agoDBM is the persistence solution of 1979. https://en.wikipedia.org/wiki/DBM_(computing) https://en.wikipedia.org/wiki/DBM_(computing)
- akubera 4y agoThere's also the shelve[0] module which allows storing any pickleable object in a persistent key-value store, not just string/bytes. I've found it's very handy for caching while developing scripts which query remote resources, and not have to worry about serialization. [0] https://docs.python.org/3.10/library/shelve.html https://docs.python.org/3.10/library/shelve.html Obligatory pickle note: one should be aware of pickle security implications and should not open a "Shelf" provided by untrusted sources, or rather should treat opening a shelf (or any pickle deserialization operation for that matter) as running an arbitrary Python script (which cannot be read).
- cristoperb 4y agoI once wrote a locking wrapper around the shelve module so I could use it as a thread- and multiprocess-safe key-value cache (including a wrapper around the requests modules `get()` to transparently cache/validate http resources according to headers): https://github.com/cristoper/shelfcache https://github.com/cristoper/shelfcache It works despite some cross-platform issues (flock and macos's version of gdbm interacting to create a deadlock), but if I were to do again I would just use sqlite (which Python's standard library has an interface for).
- TillE 4y ago> but if I were to do again I would just use sqlite Yeah, I tried to use shelve for some very simple stuff because it seemed like a great fit, but ultimately found that I had a much better time with tortoise-orm on top of sqlite. If you need any kind of real feature, just use sqlite.
- formerly_proven 4y agoThere's even an object database built around pickle, ZODB. It even has a networked implementation with replication / failover. Used to be part of Zope, originally written in the 1998 time frame or so - GvR actually committed a bunch of stuff to it.
- nas 4y ago
- giraffe_lady 4y agoThis is a wrapper around dbm, the unix library? Though having a portable interface to that tool is nothing small, and most languages don't have one in their standard library. Though I think all languages with a stdlib as big as python's probably do. It's not inherently a python thing though, unless I'm badly misunderstanding.
- collyw 4y agoI look at it the other way, this is the least effort disk persistence that I have seen in python. A fair amount of dev stuff is reading from disk and putting it into some kind of object / dictionary format. I am not really bothered about what it uses under the hood (ok it is interesting enough that I have looked up DMB). To me the he main appeal of this that you can use what appears to be a normal python dictionary and have it save to disk with minimal effort. This is a lot less friction that using a database.
- giraffe_lady 4y agoOh yeah for sure. I don't mean to convey any disdain for dbm itself or for having it rolled into the standard library. Reading and writing custom files for "convenience persistence" or configuration is one of those really common things that you don't usually need to actually do if you know about tools like this. I was just pointing out that the title and even the article seem to associate this system with python per se, rather than understanding it as python's interface to a common and pre-existing system.
- Snawoot 4y agoThere is no point to use sqlite3 in default journaling mode. I bet results may be even better than dbm if you use PRAGMA journal_mode=WAL;
- nonethewiser 4y agomore info https://sqlite.org/wal.html https://sqlite.org/wal.html
- kuschku 4y agoThat only works if all processes accessing it share a memory namespace (aka, are not in containers/jails/VMs and on the same physical machine).
- joppy 4y agoWAL works for different processes accessing the same SQLite database. It's meant to improve concurrency, in fact.
- kuschku 4y agoYes, but the processes still need to share a memory namespace. In WAL mode, SQLite creates a shared memory file which gets mapped into any process trying to access it (that’s why WAL mode creates additional .shm files). But that only works if all given processes run on the same physical machine, and not in containers/jails/VM.
- formerly_proven 4y agoI dunno what SQLite is doing specifically but you can certainly mmap a file across containers and it behaves like you’d think. SELinux and other hardening options might interfere though, because shared mmaps should imho be viewed as a very likely vector for cross-process exploitation.
- fuckstick 4y agoThere isn’t really a thing called a memory namespace, in Linux or the BSDs at least. In Linux there are cgroups that deal with accounting and you can share VM space among cloned processes (which we then typically refer to as threads). Neither of these affect the ability to share memory among processes. The only namespace that matters here is the filesystem/mount namespace. There is no reason you can’t access the same shared SQLite database on a common volume between containers on Linux for instance.
- wodenokoto 4y agoAccording to documentation it is not so much a db included as an interface to one: > dbm is a generic interface to variants of the DBM database — dbm.gnu or dbm.ndbm. If none of these modules is installed, the slow-but-simple implementation in module dbm.dumb will be used. There is a third party interface to the Oracle Berkeley DB https://docs.python.org/3/library/dbm.html https://docs.python.org/3/library/dbm.html
- chrismorgan 4y agoWhen it includes a fallback implementation so that you can rely on it working, I think it’s fair to call it “included”.
- qalmakka 4y agoThe main issue here is that, given it relies on whatever implementation of DBM you have installed, its performance will vary considerably from OS to OS. I don't know how fast or slow the "dumb" implementation is, but I can bet it is way slower than gdbm. I can see someone using this module, considering it "fast enough" on GNU/Linux and then finding out that it's painfully slow on Windows and shipping binaries of another implementation is a massive PITA.
- cozzyd 4y agoRight but the on-disk formats between versions won't be compatible...
- tootie 4y agoYeah, dbm is a posix database utility that's been around forever. There's a few implementations available and there are bindings for most languages. https://en.wikipedia.org/wiki/DBM_(computing) https://en.wikipedia.org/wiki/DBM_(computing)
- tyingq 4y agoEach of which has it's own set of limitations and watch-outs. I would not use the python dbm interface for anything serious. Ndbm, for example, has a per item value limit that varies between ~1-4k.
- Zababa 4y ago> Why is sqlite3 so slow? Well, the benchmark is probably not representative of the typical workload for sqlite (lots of individual insertions and selections). If we perform the same operations using executemany and one select of all the keys at once, we get: I think this is because each time you call execute (which is probably sqlite3_exec under the hood), your statement is prepared again, and then deleted, while with executemany it's prepared once and then used with the data. According to the SQLite3 documentation of sqlite3_exec: > The sqlite3_exec() interface is a convenience wrapper around sqlite3_prepare_v2(), sqlite3_step(), and sqlite3_finalize(), that allows an application to run multiple statements of SQL without having to use a lot of C code. I knew that when you execute a request a lot, prepared statements are faster, but it seems that it's not exactly the case and that all statements are prepared, the performance improvements come from preparing (and deleting) only once. The documentation page about the Prepared Statement Object has a good explanation of the lifecycle of a prepared statement, which also seems to be the lifecycle of all statements. [1]: https://www.sqlite.org/c3ref/exec.html https://www.sqlite.org/c3ref/exec.html [2]: https://www.sqlite.org/c3ref/stmt.html https://www.sqlite.org/c3ref/stmt.html
- quietbritishjim 4y agoThe Python sqlite3 module caches prepared statements, I think it's the last 100 by default but can be overridden. To be effective, this cache requires you to use proper parameters (rather than string interpolation) but the benchmark code for this article does that. So, in effect, it already uses stored procedures. As others have pointed out, it's the lack of a suitable index that's the real problem here.
- cbdumas 4y agoSo I read the script used to compare 'dbm' and 'sqlite3', and in sqlite it creates a table with no index. Hard to take that comparison seriously. I wrote a little benchmark script of my own just now and sqlite3 beats DBM handily if you add an index on the key.
- simonw 4y agoYeah I tried that myself - I took that benchmark script and changed the CREATE TABLE lines to look like this (adding the primary key): CREATE TABLE store(key TEXT PRIMARY KEY, value TEXT) Here are the results before I made that change: sqlite Took 0.032 seconds, 3.19541 microseconds / record dict_open Took 0.002 seconds, 0.20261 microseconds / record dbm_open Took 0.043 seconds, 4.26550 microseconds / record sqlite3_mem_open Took 2.240 seconds, 224.02620 microseconds / record sqlite3_file_open Took 7.119 seconds, 711.87410 microseconds / record And here's what I got after adding the primary keys: sqlite Took 0.040 seconds, 3.97618 microseconds / record dict_open Took 0.002 seconds, 0.19641 microseconds / record dbm_open Took 0.042 seconds, 4.18961 microseconds / record sqlite3_mem_open Took 0.116 seconds, 11.58359 microseconds / record sqlite3_file_open Took 5.571 seconds, 557.13968 microseconds / record My code is here: https://gist.github.com/simonw/019ddf08150178d49f4967cc383563ea https://gist.github.com/simonw/019ddf08150178d49f4967cc38356...
- divbzero 4y agoJust to spell out the results: Adding the primary key appears to improve SQLite performance but still falls short of DBM.
- cbdumas 4y agoYeah now that I dig in a little further it looks like it's not as clear cut as I thought. DBM read performance is better for me with larger test cases than I was initially using, though I am getting some weird performance hiccups when I use very large test cases. It looks like I'm too late to edit my top level response to reflect that.
- banana_giraffe 4y ago
- pmarreck 4y agoFor the record, so does Erlang/Elixir, right in their stdlib (OTP), as either ETS, DETS, or Mnesia https://en.wikipedia.org/wiki/Mnesia https://en.wikipedia.org/wiki/Mnesia https://www.erlang.org/doc/man/mnesia.html https://www.erlang.org/doc/man/mnesia.html
- jonnycomputer 4y agoI did not know this either. Wow.
- MrBuddyCasino 4y agoPHP also has support several Berkeley-style databases: https://www.php.net/manual/en/dba.requirements.php https://www.php.net/manual/en/dba.requirements.php
- isoprophlex 4y agoThere exists the 'diskcache' library too, which gives you a persistent dictionary that stores any kind of pickle-able python object. There's also a nice persistent memoization decorator, with optional TTL... I've always found 'diskcache' to be very ergonomic.
- BiteCode_dev 4y agoIf you can pip install something, I would also recommend diskcache: - it backed up by sqlite, so the data is much more secure - you can access it from several processes - it's more portable
- collyw 4y agoI thought Perl's Tie::DB_file module was unique for that sort of thing. That's probably quite useful.
- thraxil 4y agoBack in the 90's I used to use MLDBM as the database for simple CGI apps that didn't call for a full-blown RDBMS. There are some obvious limitations, but at the time it was a pretty powerful setup for simple applications.
- collyw 4y agoIt's amazing how few people seemed to understand that NoSQL existed long before Mongo's marketing team cashed in on the trend. I was using various NoSQl solutions before they were trendy.
- gerty 4y agoIt's not built-in, of course, but if you can live with the quirky API, LMDB as an embedded key-value store will be hard to beat in terms of speed. It can also do "one-writer, many-readers" and transactions.
- syncsynchalt 4y agoThis is libdbm, a relatively ancient db library that you could find on most unixes. In the 90s you might target dbm for your portable Unix application because you could reasonably expect it to be implemented on your platform. It lost a lot of share to gdbm and sleepycat's BerkeleyDB, both of which I consider its successor. Of course all of this is now sqlite3.
- ridiculous_fish 4y agoIn AppleScript you can mark a variable as a property, and then its value is automatically remembered across runs. For example a script can easily remember how many times it has run: property runCount : 0 set runCount to runCount + 1 So natural to use! https://developer.apple.com/library/archive/documentation/AppleScript/Conceptual/AppleScriptLangGuide/conceptual/ASLR_variables.html#//apple_ref/doc/uid/TP40000983-CH223-SW2 https://developer.apple.com/library/archive/documentation/Ap...
- nathancahill 4y agoI feel like AppleScript could have really shown if things had played out differently in Mac OS automation. I learned to code with AppleScript on OS 9 and it really got me in to programming. MacScripter.net was my old hangout. Good times.
- nijave 4y agoWonder how this compares to using the filesystem and keys where keys are either filenames or a file path (and the value is stored in the file)?
- kortex 4y agoI suspect it'll be way, way faster. There's a fair amount of overhead in filesystem calls. (simplification) With any time you are mapping some file to some data structure (dbm, sqlite, mmap), the file descriptor generally stays open while you are working with it, meaning reads/writes/seeks are just about as fast as IO can go.
- amelius 4y agoThis is going to fall flat when you try to use multithreading. The GIL will rear its head.
- Someone 4y agoFTA: Why is sqlite3 so slow? My money is that’s mostly because it actually stores the data. I don’t think dbm guarantees any of the data written makes it to disk until you sync or close the database. This writes, guesstimating, on the order of 100 kilobytes, so chances are the dbm data never hits the disk.
- a-dub 4y agodbm is an ancient unix thing. better in class are gdbm and berkeleydb... or these days of course, sqlite.
- deleted 4y ago[deleted]