4 ms·

import json json = json.dumps(database) with open("database.json","w") as f: f.write(json) :) - 0 dependencies - easily inspectable and editable with an

import json json = json.dumps(database) with open("database.json","w") as f: f.write(json) :) - 0 dependencies - easily inspectable and editable with any text editor or cli via jq - backup and diff - language agnostic

chousuke 6y ago
This "simple" approach will also lead to all kinds of problems quite quickly once you grow past one concurrent user. If you really must use JSON, at least use SQLite in place of open().
- bullen 6y ago
  A actually use one file per value! I have to partition the ext4 filesystem with type small othervise I run out if inodes before diskspace! Here is what it looks like in action: http://root.rupy.se/link/type/task/847068548006606746 http://root.rupy.se/link/type/task/847068548006606746 The front end: http://talk.binarytask.com http://talk.binarytask.com
uyt 6y ago
I tried this approach for scraping a REST API returning json (on a few tens of thousands of ids). Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script. I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty. I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.
scottlamb 6y ago
- occasionally accidentally truncates the database If you're going to do this, at least write to a temporary file, fsync the file, rename into place, and fsync the directory. But I recommend SQLite any time someone is tempted to write these lines.

chousuke 6y ago

This "simple" approach will also lead to all kinds of problems quite quickly once you grow past one concurrent user. If you really must use JSON, at least use SQLite in place of open().

bullen 6y ago
A actually use one file per value! I have to partition the ext4 filesystem with type small othervise I run out if inodes before diskspace! Here is what it looks like in action: http://root.rupy.se/link/type/task/847068548006606746 http://root.rupy.se/link/type/task/847068548006606746 The front end: http://talk.binarytask.com http://talk.binarytask.com

uyt 6y ago

I tried this approach for scraping a REST API returning json (on a few tens of thousands of ids). Since it was a long running job with many failures (throttling and connectivity issues), I had to constantly kill and restart the script. I needed to know what ids to skip over on restart but for some reason listing a folder with a few tens of thousands of files is very slow. So restarting takes forever. I also ran into issues with nonatomic file writes where even though a json file was written, it was incomplete or empty. I think if I had just inserted them as json strings into sqlite it would've been more robust? I am okay with losing writes (since I will just redownload them), but it was the incomplete writes and long time to reload the set of seen ids on restart that drove me crazy.

scottlamb 6y ago

- occasionally accidentally truncates the database If you're going to do this, at least write to a temporary file, fsync the file, rename into place, and fsync the directory. But I recommend SQLite any time someone is tempted to write these lines.