3 ms·

Prior discussion (May 2, 2021): https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630

AceJohnny2 4y ago
It's funny, I was wondering whether this replicated Simon Willison's work on Datasette [1], but I guess I got things confused because Simon himself was the first to comment on that older post about how clever this was. [1] https://simonwillison.net/2018/Oct/4/datasette-ideas/ https://simonwillison.net/2018/Oct/4/datasette-ideas/
- crazysim 4y ago
  There's a web browser version of datasette called datasette-lite which runs on Python ported to WASM with Pyodide which can load SQLite databases. I grafted the enhanced lazyFile implementation from emscripten and then from this implementation to datasette-lite relatively recently as a curious test. Threw in a 18GB CSV from CA's unclaimed property records here https://www.sco.ca.gov/upd_download_property_records.html https://www.sco.ca.gov/upd_download_property_records.html into a FTS5 Sqlite Database which came out to about 28GB after processing: POC, non-merging Log/Draft PR for the hack: https://github.com/simonw/datasette-lite/pull/49 https://github.com/simonw/datasette-lite/pull/49 You can run queries through to datasette-lite if you URL hack into it and just get to the query dialog, browsing is kind of a dud at the moment since datasette runs a count(*) which downloads everything. Example: https://datasette-lite-lab.mindflakes.com/index.html?url=https://datasette-lite-lab.mindflakes.com/sdb/2022-10-02_93eff57de3573985_ca_unclaimed_property.sqlite#/2022-10-02_93eff57de3573985_ca_unclaimed_property?sql=SELECT+*+FROM+records+WHERE+records.owner_name+MATCH+%22Elon+Musk%22+ORDER+BY+CAST%28CURRENT_CASH_BALANCE+AS+FLOAT%29++DESC%3B https://datasette-lite-lab.mindflakes.com/index.html?url=htt... Still, not bad for a $0.42/mo hostable cached CDN'd read-only database. It's on Cloudflare R2, so there's no BW costs.
  - marvel_boy 4y ago
    Amazing. Runs smoothly. Elon could not like the real state example :). By the way, R2 service is in beta, right?
    - crazysim 4y ago
      GA'd https://blog.cloudflare.com/r2-ga/ https://blog.cloudflare.com/r2-ga/

AceJohnny2 4y ago

It's funny, I was wondering whether this replicated Simon Willison's work on Datasette [1], but I guess I got things confused because Simon himself was the first to comment on that older post about how clever this was. [1] https://simonwillison.net/2018/Oct/4/datasette-ideas/ https://simonwillison.net/2018/Oct/4/datasette-ideas/

crazysim 4y ago
There's a web browser version of datasette called datasette-lite which runs on Python ported to WASM with Pyodide which can load SQLite databases. I grafted the enhanced lazyFile implementation from emscripten and then from this implementation to datasette-lite relatively recently as a curious test. Threw in a 18GB CSV from CA's unclaimed property records here https://www.sco.ca.gov/upd_download_property_records.html https://www.sco.ca.gov/upd_download_property_records.html into a FTS5 Sqlite Database which came out to about 28GB after processing: POC, non-merging Log/Draft PR for the hack: https://github.com/simonw/datasette-lite/pull/49 https://github.com/simonw/datasette-lite/pull/49 You can run queries through to datasette-lite if you URL hack into it and just get to the query dialog, browsing is kind of a dud at the moment since datasette runs a count(*) which downloads everything. Example: https://datasette-lite-lab.mindflakes.com/index.html?url=https://datasette-lite-lab.mindflakes.com/sdb/2022-10-02_93eff57de3573985_ca_unclaimed_property.sqlite#/2022-10-02_93eff57de3573985_ca_unclaimed_property?sql=SELECT+*+FROM+records+WHERE+records.owner_name+MATCH+%22Elon+Musk%22+ORDER+BY+CAST%28CURRENT_CASH_BALANCE+AS+FLOAT%29++DESC%3B https://datasette-lite-lab.mindflakes.com/index.html?url=htt... Still, not bad for a $0.42/mo hostable cached CDN'd read-only database. It's on Cloudflare R2, so there's no BW costs.
- marvel_boy 4y ago
  Amazing. Runs smoothly. Elon could not like the real state example :). By the way, R2 service is in beta, right?
  - crazysim 4y ago
    GA'd https://blog.cloudflare.com/r2-ga/ https://blog.cloudflare.com/r2-ga/