6 ms·

Show HN: JSONlite – A simple, serverless, zero-configuration JSON document store

74 points by nodesocket 10y ago

PythonicAlpha 10y ago
I currently don't see the similarity to SQLite, that is implied in the readme. SQLite stores all its data (every table and all entries in tables) in one single file. JSONlite does store every JSON entry in one particular file, as much I saw. So, JSONlite uses the file system (one directory) as data storage -- in contrast to SQLite!
CJefferson 10y ago
If you are going to invoke the sacred name of 'sqlite', then I expect to see rock-solid code and quality testing. Instead, there are two tiny tests, and I found a couple of issues in 30 seconds of looking at the code. EDIT: I should prove my point. 1) Race condition on calculating uuid (obviously won't be too serious for a good uuid implementation) 2) No check that file is written successfully. 3) No check when json is invalid, just silently swallowed, or filesystem full. 4) Makes one file a json file, will scale terribly past a few thousand json files.
- nodesocket 10y ago
  Create some issues and pull requests and help out. :-)
  - rad_gruchalski 10y ago
    How about some due diligence first? I know, it's so easy to say "just create a pull request". To which the answer is: "I'm not going to use your software if you can't make effort to test it and document properly".
  - throwanem 10y ago
    Show me why your project is worth some of my finite time and effort, and I might. This looks like a toy project to me, and while there's nothing wrong with a toy project, I'm not seeing why it's more fun for me to play with this toy than some other of the myriad from which I have to choose.
- nodesocket 10y ago
  Good point on 2 and 3. Should just involve checking the previous commands return code before echoing the uuid. I'll get that patched up and ship a new version in a bit. Not sure what you mean exactly by 1 and 4 though.
  - jakewins 10y ago
    Many operating systems do not handle tens of thousands of file handles open simultaneously very well. For instance, if you run this database on Linux, the RAM overhead for each document is 1K just for the file handle, see the comment at the bottom here http://lxr.free-electrons.com/source/fs/file_table.c http://lxr.free-electrons.com/source/fs/file_table.c
  - CJefferson 10y ago
    With 1, you are trusting the directory returned by uuid will be unique --it should be, but it might not be, in particular if it is based on time and I run in two threads. With 4, I find once a directory has about 100,000 files in it, things get bad, from the simple (ls * won't work) to the nastier (git starts using huge amounts of space, you'll hit github's size limit even though all your files are quite small).
    - avip 10y ago
      In all fairness, I find this (and only this) criticism unsubstantiated/nitpicky. Many systems are relying on the statistical improbability of 32-hexdigits uuid collision (name-dropping one - Disque).
    - sz4kerto 10y ago
      I would not let anyone close to the code I'm working on if he wanted to verify whether a properly generated UUID clashes with some previously generated UUID. You could just as do stuff like "int x=1; assert(x == 1);" and so on. And it would be just as futile.
      - CJefferson 10y ago
        In this case, you are relying on what installed program or function is called when you invoke uuid in the shell. It seems reasonable to me to check if it is having sensibly, if the reliability of your program counts on it. I have no idea what dodgy uuid programs you might find installed.
- nodesocket 10y ago
  Heads up, just fixed the bug where providing invalid json to set would write an empty file with a uuid filename to the data directory. Now providing invalid json to set errors with no file written to the data directory. New version is 1.1.0 (https://github.com/nodesocket/jsonlite/releases/tag/1.1.0 https://github.com/nodesocket/jsonlite/releases/tag/1.1.0) if you care.
  - CJefferson 10y ago
    Just wanted to say, glad to see these are being fixed. My initial message was a little harsh -- although github makes it hard to tell if you are making a new library for fun, or something you really want people using for important customer data!
sigmonsays 10y ago
This is 200 lines of bash using the file system as a database. The term document store implies a database which this is not. How is this different than just writing files to a directory? What are the use cases of this project?
- pmelendez 10y ago
  "A database is an organized collection of data". I think writing files in a directory with a convention falls in that category. [1]https://en.m.wikipedia.org/wiki/Database https://en.m.wikipedia.org/wiki/Database
  - hashkb 10y ago
    Missing the point on purpose?
  - camus2 10y ago
    The point is the project seems a bit low-effort. The most complicated thing in that bash script is argument processing. What is the added value of that project? does it come up with something new or interesting? a query language for data? an efficient data storage? no, it just store json files in directories. Sure the file system is already a database, but that project doesn't add anything to the file system.
mtolan 10y ago
I think your title might not be the best; * Serverless typically implies something accessible from more than the host you're on * It's not zero configuration, there's at least 1 configurable parameter (albeit with a sensible default) * I'm actually not even clear why this is a document store limited to JSON, other than you're piping it though a JSON python module. Having done some things like this in the past, as you continue, you'll probably want to create subdirectories based on the first N characters of the UUID, but I'm not sure you wouldn't get considerably more value of just putting your json blobs into an AWS Dynamo table.
- nodesocket 10y ago
  There is already an issue about supporting sharding https://github.com/nodesocket/jsonlite/issues/15 https://github.com/nodesocket/jsonlite/issues/15.
  - camus2 10y ago
    I thought it was "serverless"? what does "sharding" have to do with anything? I'm sorry but it seems like you throw around some vocabulary that doesn't describe your project properly, to make it look bigger than it really is. That's nice marketing but it does not feel like there is much effort put into it, when I looked at your bash script.
    - nodesocket 10y ago
      "The word shard means a small part of a whole." The idea is breakup uuid keys into sub directories based on the first couple of letters to prevent file system performance issues. Seems like the right usage of the word from a dictionary perspective.
      - sciurus 10y ago
        According to your logic, my home directory is also a sharded, serverless, zero-configuration document store. I should apply for some VC funding. ;-)
camus2 10y ago
Serverless means anything really, a cli app is now serverless. I didn't know I was writing serverless applications for 20 years then. Serverless was already a bad buzz word, it gets more and more meaningless by the hour. By the way sqlite just got support for JSON : https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html off-topic : I'd like to see more books on database implementation for beginners. Of all the crap load of CS books that come out year after year this is a matter with very little literature, as most papers on the subject are research papers. Seems like an excellent topic to teach distributed programming.
- dchest 10y ago
  No, serverless database is not a recent buzz word, but a term frequently used to describe DB that doesn't need a separate server running to which clients connect: compare MySQL, PostgreSQL against BDB, LMDB, SQLite.
  - evanweaver 10y ago
    In my communities we called the library-style dbs "embedded". I've seen serverless used in this way but it didn't seem common.
    - dchest 10y ago
      Not every serverless database is embedded, but most of them - those that are also called "database library" are. Some examples of DBs that are both embedded and call themselves serverless: - SQLite: https://sqlite.org/serverless.html https://sqlite.org/serverless.html - UnQlite: https://www.unqlite.org/features.html#serverless https://www.unqlite.org/features.html#serverless Serverless is used to highlight this specific property of them. The OP's database is NOT embedded, but it is serverless.
  - camus2 10y ago
    "serverless" means so many things it really means nothing, that's like "cloud computing" and that's the definition of a buzzword. Sqlite is an embedded database and has nothing to do with your favorite buzzword.
jwilk 10y ago
> JSONlite sandboxes the current working directory similar to SQLite. Huh, what does that mean?
dunkey123 10y ago
I don't like that it is just a composition of bash and the filesystem. I'd just do this: id=$(uuidgen) echo '{"hi":"mom"}' | jq . > id cat id
noonespecial 10y ago
Hint: If you really want to use your filesystem as a database (and don't mind the haters, there's a long and storied history of doing just this), make sure you break it up into many subdirs usually based on the first few characters of the uuid. Example: ./jasondir/aa/bb/cc/aabbccdd You won't like what happens when you put 100k files in one directory.
- TheGreatPotatoe 10y ago
  There was an old document management system called Keyfile that did just that.
  - tyingq 10y ago
    Apache's mod_cache does this as well.
  - manojlds 10y ago
    Git does it too.
- PretzelFisch 10y ago
  hasn't the file system improved to the point where this is less of a problem?
  - tyingq 10y ago
    Even if it's running on a filesystem that deals with it well, standard utilities like 'ls' don't deal with it well, as the default behavior is to sort. You end up having to look up obscure options, like 'ls -U' to disable sorting. You can also run into issues like "Argument list too long". ARG_MAX is larger on linux than it used to be, but it's pretty short on older kernels. I assume similar issues might exist on other operating systems.
    - e12e 10y ago
      I'm not sure that's so much of a problem, I just tried: mkdir uuid; cd uuid uuid -v 4 -n 1000000 |\ while read uuid; do touch $uuid; done And ran out of inodes, but: time ls uuid|wc -l 425621 real 0m1.796s user 0m1.552s sys 0m0.240s Sure, it's not exactly stellar performance for a linear scan of ~400k keys - but it's not terrible (for various values and expectations of terrible). This is in a hyper-v vm on a Surface 4 pro/i5. The fact that the "uuid" program can quickly generate uuids make me wonder if maybe one approach would be to generate uuids to (a) fifo(s), and then let db thread/processes read uuids from the other end?
drinchev 10y ago
This reminds me of NodeJS configstore[1]. I would be really happy to learn what are the use-cases for this tool, since using it via bash is actually an overkill. 1: https://github.com/yeoman/configstore https://github.com/yeoman/configstore
bvinc 10y ago
Wouldn't you be much better off storing JSON in a SQLite table?
- PretzelFisch 10y ago
  How would that be an improvement? you still wouldn't be able to readily query and build indexes? Or does SQLite handle that now?
  - camus2 10y ago
    https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html I really don't get why this made it to HN first page, especially when plenty of other show HN demonstrate way more efforts than writing a short bashscript yet never make it. I suspect it has to do with the number of buzzwords that were inserted into the title.
Waterluvian 10y ago
I think there is a lot of feedback, mostly constructive criticisms. I would encourage you to embrace a lot of what everyone is saying and not to take it personally. You've done far better than most developers: you authored something and shared it with the world. To this day I'm still too afraid to show HN or Reddit my work.
sscarduzio 10y ago
lol why is this so high in the front page? BTW this is NOT SERVERLESS therefore click bait in my book
vikingcaffiene 10y ago
First I want to say this is really cool. Thanks for taking the time to make it and I'll definitely give a test drive. That said, it's not zero configuration and your title is kinda misleading.
Tepix 10y ago
Cool! Can you add an append function?
NelsonMinar 10y ago
Why not just use SQLite? Its JSON1 extension is pretty good, and then you get all the rest of SQLite's goodness as part of the package. https://sqlite.org/json1.html https://sqlite.org/json1.html
agounaris 10y ago
The "serverless" starts being used and abused the same way "microservices" and "rest" are.
nerdponx 10y ago
There's already an R package called jsonlite: https://cran.r-project.org/package=jsonlite https://cran.r-project.org/package=jsonlite
laughfactory 10y ago
Also the name of a very popular R package. Maybe call it JSONdb? Or JSONstore?

74 points by nodesocket 10y ago

I currently don't see the similarity to SQLite, that is implied in the readme. SQLite stores all its data (every table and all entries in tables) in one single file. JSONlite does store every JSON entry in one particular file, as much I saw. So, JSONlite uses the file system (one directory) as data storage -- in contrast to SQLite!

CJefferson 10y ago

If you are going to invoke the sacred name of 'sqlite', then I expect to see rock-solid code and quality testing. Instead, there are two tiny tests, and I found a couple of issues in 30 seconds of looking at the code. EDIT: I should prove my point. 1) Race condition on calculating uuid (obviously won't be too serious for a good uuid implementation) 2) No check that file is written successfully. 3) No check when json is invalid, just silently swallowed, or filesystem full. 4) Makes one file a json file, will scale terribly past a few thousand json files.

nodesocket 10y ago
Create some issues and pull requests and help out. :-)
- rad_gruchalski 10y ago
  How about some due diligence first? I know, it's so easy to say "just create a pull request". To which the answer is: "I'm not going to use your software if you can't make effort to test it and document properly".
- throwanem 10y ago
  Show me why your project is worth some of my finite time and effort, and I might. This looks like a toy project to me, and while there's nothing wrong with a toy project, I'm not seeing why it's more fun for me to play with this toy than some other of the myriad from which I have to choose.
nodesocket 10y ago
Good point on 2 and 3. Should just involve checking the previous commands return code before echoing the uuid. I'll get that patched up and ship a new version in a bit. Not sure what you mean exactly by 1 and 4 though.
- jakewins 10y ago
  Many operating systems do not handle tens of thousands of file handles open simultaneously very well. For instance, if you run this database on Linux, the RAM overhead for each document is 1K just for the file handle, see the comment at the bottom here http://lxr.free-electrons.com/source/fs/file_table.c http://lxr.free-electrons.com/source/fs/file_table.c
- CJefferson 10y ago
  With 1, you are trusting the directory returned by uuid will be unique --it should be, but it might not be, in particular if it is based on time and I run in two threads. With 4, I find once a directory has about 100,000 files in it, things get bad, from the simple (ls * won't work) to the nastier (git starts using huge amounts of space, you'll hit github's size limit even though all your files are quite small).
  - avip 10y ago
    In all fairness, I find this (and only this) criticism unsubstantiated/nitpicky. Many systems are relying on the statistical improbability of 32-hexdigits uuid collision (name-dropping one - Disque).
  - sz4kerto 10y ago
    I would not let anyone close to the code I'm working on if he wanted to verify whether a properly generated UUID clashes with some previously generated UUID. You could just as do stuff like "int x=1; assert(x == 1);" and so on. And it would be just as futile.
    - CJefferson 10y ago
      In this case, you are relying on what installed program or function is called when you invoke uuid in the shell. It seems reasonable to me to check if it is having sensibly, if the reliability of your program counts on it. I have no idea what dodgy uuid programs you might find installed.
nodesocket 10y ago
Heads up, just fixed the bug where providing invalid json to set would write an empty file with a uuid filename to the data directory. Now providing invalid json to set errors with no file written to the data directory. New version is 1.1.0 (https://github.com/nodesocket/jsonlite/releases/tag/1.1.0 https://github.com/nodesocket/jsonlite/releases/tag/1.1.0) if you care.
- CJefferson 10y ago
  Just wanted to say, glad to see these are being fixed. My initial message was a little harsh -- although github makes it hard to tell if you are making a new library for fun, or something you really want people using for important customer data!

sigmonsays 10y ago

This is 200 lines of bash using the file system as a database. The term document store implies a database which this is not. How is this different than just writing files to a directory? What are the use cases of this project?

pmelendez 10y ago
"A database is an organized collection of data". I think writing files in a directory with a convention falls in that category. [1]https://en.m.wikipedia.org/wiki/Database https://en.m.wikipedia.org/wiki/Database
- hashkb 10y ago
  Missing the point on purpose?
- camus2 10y ago
  The point is the project seems a bit low-effort. The most complicated thing in that bash script is argument processing. What is the added value of that project? does it come up with something new or interesting? a query language for data? an efficient data storage? no, it just store json files in directories. Sure the file system is already a database, but that project doesn't add anything to the file system.

mtolan 10y ago

I think your title might not be the best; * Serverless typically implies something accessible from more than the host you're on * It's not zero configuration, there's at least 1 configurable parameter (albeit with a sensible default) * I'm actually not even clear why this is a document store limited to JSON, other than you're piping it though a JSON python module. Having done some things like this in the past, as you continue, you'll probably want to create subdirectories based on the first N characters of the UUID, but I'm not sure you wouldn't get considerably more value of just putting your json blobs into an AWS Dynamo table.

nodesocket 10y ago
There is already an issue about supporting sharding https://github.com/nodesocket/jsonlite/issues/15 https://github.com/nodesocket/jsonlite/issues/15.
- camus2 10y ago
  I thought it was "serverless"? what does "sharding" have to do with anything? I'm sorry but it seems like you throw around some vocabulary that doesn't describe your project properly, to make it look bigger than it really is. That's nice marketing but it does not feel like there is much effort put into it, when I looked at your bash script.
  - nodesocket 10y ago
    "The word shard means a small part of a whole." The idea is breakup uuid keys into sub directories based on the first couple of letters to prevent file system performance issues. Seems like the right usage of the word from a dictionary perspective.
    - sciurus 10y ago
      According to your logic, my home directory is also a sharded, serverless, zero-configuration document store. I should apply for some VC funding. ;-)

camus2 10y ago

Serverless means anything really, a cli app is now serverless. I didn't know I was writing serverless applications for 20 years then. Serverless was already a bad buzz word, it gets more and more meaningless by the hour. By the way sqlite just got support for JSON : https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html off-topic : I'd like to see more books on database implementation for beginners. Of all the crap load of CS books that come out year after year this is a matter with very little literature, as most papers on the subject are research papers. Seems like an excellent topic to teach distributed programming.

dchest 10y ago
No, serverless database is not a recent buzz word, but a term frequently used to describe DB that doesn't need a separate server running to which clients connect: compare MySQL, PostgreSQL against BDB, LMDB, SQLite.
- evanweaver 10y ago
  In my communities we called the library-style dbs "embedded". I've seen serverless used in this way but it didn't seem common.
  - dchest 10y ago
    Not every serverless database is embedded, but most of them - those that are also called "database library" are. Some examples of DBs that are both embedded and call themselves serverless: - SQLite: https://sqlite.org/serverless.html https://sqlite.org/serverless.html - UnQlite: https://www.unqlite.org/features.html#serverless https://www.unqlite.org/features.html#serverless Serverless is used to highlight this specific property of them. The OP's database is NOT embedded, but it is serverless.
- camus2 10y ago
  "serverless" means so many things it really means nothing, that's like "cloud computing" and that's the definition of a buzzword. Sqlite is an embedded database and has nothing to do with your favorite buzzword.

jwilk 10y ago

> JSONlite sandboxes the current working directory similar to SQLite. Huh, what does that mean?

dunkey123 10y ago

I don't like that it is just a composition of bash and the filesystem. I'd just do this: id=$(uuidgen) echo '{"hi":"mom"}' | jq . > id cat id

noonespecial 10y ago

Hint: If you really want to use your filesystem as a database (and don't mind the haters, there's a long and storied history of doing just this), make sure you break it up into many subdirs usually based on the first few characters of the uuid. Example: ./jasondir/aa/bb/cc/aabbccdd You won't like what happens when you put 100k files in one directory.

TheGreatPotatoe 10y ago
There was an old document management system called Keyfile that did just that.
- tyingq 10y ago
  Apache's mod_cache does this as well.
- manojlds 10y ago
  Git does it too.
PretzelFisch 10y ago
hasn't the file system improved to the point where this is less of a problem?
- tyingq 10y ago
  Even if it's running on a filesystem that deals with it well, standard utilities like 'ls' don't deal with it well, as the default behavior is to sort. You end up having to look up obscure options, like 'ls -U' to disable sorting. You can also run into issues like "Argument list too long". ARG_MAX is larger on linux than it used to be, but it's pretty short on older kernels. I assume similar issues might exist on other operating systems.
  - e12e 10y ago
    I'm not sure that's so much of a problem, I just tried: mkdir uuid; cd uuid uuid -v 4 -n 1000000 |\ while read uuid; do touch $uuid; done And ran out of inodes, but: time ls uuid|wc -l 425621 real 0m1.796s user 0m1.552s sys 0m0.240s Sure, it's not exactly stellar performance for a linear scan of ~400k keys - but it's not terrible (for various values and expectations of terrible). This is in a hyper-v vm on a Surface 4 pro/i5. The fact that the "uuid" program can quickly generate uuids make me wonder if maybe one approach would be to generate uuids to (a) fifo(s), and then let db thread/processes read uuids from the other end?

drinchev 10y ago

This reminds me of NodeJS configstore[1]. I would be really happy to learn what are the use-cases for this tool, since using it via bash is actually an overkill. 1: https://github.com/yeoman/configstore https://github.com/yeoman/configstore

bvinc 10y ago

Wouldn't you be much better off storing JSON in a SQLite table?

PretzelFisch 10y ago
How would that be an improvement? you still wouldn't be able to readily query and build indexes? Or does SQLite handle that now?
- camus2 10y ago
  https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html I really don't get why this made it to HN first page, especially when plenty of other show HN demonstrate way more efforts than writing a short bashscript yet never make it. I suspect it has to do with the number of buzzwords that were inserted into the title.

Waterluvian 10y ago

I think there is a lot of feedback, mostly constructive criticisms. I would encourage you to embrace a lot of what everyone is saying and not to take it personally. You've done far better than most developers: you authored something and shared it with the world. To this day I'm still too afraid to show HN or Reddit my work.

sscarduzio 10y ago

lol why is this so high in the front page? BTW this is NOT SERVERLESS therefore click bait in my book

vikingcaffiene 10y ago

First I want to say this is really cool. Thanks for taking the time to make it and I'll definitely give a test drive. That said, it's not zero configuration and your title is kinda misleading.

Tepix 10y ago

Cool! Can you add an append function?

NelsonMinar 10y ago

Why not just use SQLite? Its JSON1 extension is pretty good, and then you get all the rest of SQLite's goodness as part of the package. https://sqlite.org/json1.html https://sqlite.org/json1.html

agounaris 10y ago

The "serverless" starts being used and abused the same way "microservices" and "rest" are.

nerdponx 10y ago

There's already an R package called jsonlite: https://cran.r-project.org/package=jsonlite https://cran.r-project.org/package=jsonlite

laughfactory 10y ago

Also the name of a very popular R package. Maybe call it JSONdb? Or JSONstore?