3 ms·
Making it available via a static file host dramatically lowers the barrier. If you have an interesting dataset, you can even throw it on github pages and pay no
by sekao 5y ago
Making it available via a static file host dramatically lowers the barrier. If you have an interesting dataset, you can even throw it on github pages and pay nothing; that likely is not true for your mongo db server.
A typical request using indexes will be less than 10 separate 1KB GET requests, not 50. But yeah, more work needs to be done on performance.
Whether it makes sense to fully download the dataset depends on the project; maybe it does not. But it doesn't have to be a monolithic file. You can use SQLite's multiplex VFS to split the SQLite file into many smaller pieces (and still update the db later!).
- SergeAx 5y agoI think this can be carefully thought through and become something interesting. I don't fancy SQL, to be honest, despite having "structural" in its name it is too chaotic for XXI century. Rows and columns! Schema-enforced document databases, on the other hand, are neat and mostly people- and machine-readable at the same time. Another idea: downloading only indexes may greatly reduce number of requests needed to query the data.
- ricardobeat 5y agoThat’s what those tiny 1KB requests are doing mostly, downloading indexes. With http pipelining taken into account, it’s way faster than trying to preload the full index data. Despite the number of requests seeming excessive to us, the performance of this setup is already in the ballpark of your usual underpowered MySQL going through an app server.