3 ms·

What is the difference between this idea and exposing a read-only MongoDB (JSON in, JSON out) via HTTP endpoint? In the end, 50 ranged HTTP requests is not that

What is the difference between this idea and exposing a read-only MongoDB (JSON in, JSON out) via HTTP endpoint? In the end, 50 ranged HTTP requests is not that great for unstable connection, 1 request is way better, and you need a server anyway. Okay, offline first, what does that mean? Should I download the entire 600mb SQLite database? Should I do it every time it changes? Who will pay for the bandwidth? We can not employ standard HTTP proxy and caching mechanism here, it is not for 600mb files.

sekao 5y ago
Making it available via a static file host dramatically lowers the barrier. If you have an interesting dataset, you can even throw it on github pages and pay nothing; that likely is not true for your mongo db server. A typical request using indexes will be less than 10 separate 1KB GET requests, not 50. But yeah, more work needs to be done on performance. Whether it makes sense to fully download the dataset depends on the project; maybe it does not. But it doesn't have to be a monolithic file. You can use SQLite's multiplex VFS to split the SQLite file into many smaller pieces (and still update the db later!).
- SergeAx 5y ago
  I think this can be carefully thought through and become something interesting. I don't fancy SQL, to be honest, despite having "structural" in its name it is too chaotic for XXI century. Rows and columns! Schema-enforced document databases, on the other hand, are neat and mostly people- and machine-readable at the same time. Another idea: downloading only indexes may greatly reduce number of requests needed to query the data.
  - ricardobeat 5y ago
    That’s what those tiny 1KB requests are doing mostly, downloading indexes. With http pipelining taken into account, it’s way faster than trying to preload the full index data. Despite the number of requests seeming excessive to us, the performance of this setup is already in the ballpark of your usual underpowered MySQL going through an app server.

sekao 5y ago

Making it available via a static file host dramatically lowers the barrier. If you have an interesting dataset, you can even throw it on github pages and pay nothing; that likely is not true for your mongo db server. A typical request using indexes will be less than 10 separate 1KB GET requests, not 50. But yeah, more work needs to be done on performance. Whether it makes sense to fully download the dataset depends on the project; maybe it does not. But it doesn't have to be a monolithic file. You can use SQLite's multiplex VFS to split the SQLite file into many smaller pieces (and still update the db later!).

SergeAx 5y ago
I think this can be carefully thought through and become something interesting. I don't fancy SQL, to be honest, despite having "structural" in its name it is too chaotic for XXI century. Rows and columns! Schema-enforced document databases, on the other hand, are neat and mostly people- and machine-readable at the same time. Another idea: downloading only indexes may greatly reduce number of requests needed to query the data.
- ricardobeat 5y ago
  That’s what those tiny 1KB requests are doing mostly, downloading indexes. With http pipelining taken into account, it’s way faster than trying to preload the full index data. Despite the number of requests seeming excessive to us, the performance of this setup is already in the ballpark of your usual underpowered MySQL going through an app server.