4 ms·

There was a UK government GitHub repo that did something interesting with this kind of trick against S3 but I checked just now and the repo is a 404. Here are m

by simonw 11mo ago

There was a UK government GitHub repo that did something interesting with this kind of trick against S3 but I checked just now and the repo is a 404. Here are my notes about what it did: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ Looks like it's still on PyPI though: https://pypi.org/project/sqlite-s3vfs/ https://pypi.org/project/sqlite-s3vfs/ You can see inside it with my PyPI package explorer: https://tools.simonwillison.net/zip-wheel-explorer?package=sqlite-s3vfs#sqlite_s3vfs.py https://tools.simonwillison.net/zip-wheel-explorer?package=s...

simonw 11mo ago
I recovered it from https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/uktrade/sqlite-s3vfs https://archive.softwareheritage.org/browse/origin/directory... and pushed a fresh copy to GitHub here: https://github.com/simonw/sqlite-s3vfs https://github.com/simonw/sqlite-s3vfs This comment was helpful in figuring out how to get a full Git clone out of the heritage archive: https://news.ycombinator.com/item?id=37516523#37517378 https://news.ycombinator.com/item?id=37516523#37517378 Here's a TIL I wrote up of the process: https://til.simonwillison.net/github/software-archive-recovery https://til.simonwillison.net/github/software-archive-recove...
- AceJohnny2 11mo ago
  didn't you do something similar for Datasette, Simon?
  - simonw 11mo ago
    Nothing smart with HTTP range requests yet - I have https://lite.datasette.io https://lite.datasette.io which runs the full Python server app in the browser via WebAssembly and Pyodide but it still works by fetching the entire SQLite file at once.
    - AceJohnny2 11mo ago
      oh! I must've been confused with your TIL where you linked to an explainer of this technique https://simonwillison.net/2021/May/2/hosting-sqlite-databases-on-github-pages/ https://simonwillison.net/2021/May/2/hosting-sqlite-database... https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
- QuantumNomad_ 11mo ago
  I also have a locally cloned copy of that repo from when it was on GitHub. Same latest commit as your copy of it. From what I see in GitHub in your copy of the repo, it looks like you don’t have the tags. Do you have the tags locally? If you don’t have the tags, I can push a copy of the repo to GitHub too and you can get the tags from my copy.
  - simonw 11mo ago
    I don't have the tags! It would be awesome if you could push that.
    - QuantumNomad_ 11mo ago
      Uploaded here: https://github.com/Quantum-Nomad/sqlite-s3vfs https://github.com/Quantum-Nomad/sqlite-s3vfs
      - simonw 11mo ago
        Thanks for that, though actually it turns out I had them after all - I needed to run: git push --tags origin
        QuantumNomad_ 11mo ago
        All the better :)
- bspammer 11mo ago
  Doing all this in an hour is such a good example of how absurdly efficient you can be with LLMs.
  - socialcommenter 11mo ago
    From reading the TIL, it doesn't appear as if Simon used LLM for a large portion of what he did; only the initial suggestion to check the archive, and the web tool to make his process reproducible. Also, if you read the script from his chat with Claude code, the prompt really does the heavy lifting. Sure, the LLM fills in all the boilerplate and makes an easy-to-use, reproducible tool with loads of documentation, and credit for that. But is it not more accurate to say that Simon is absurdly efficient, LLM or sans LLM? :)
billywhizz 11mo ago
i played around with this a while back. you can see a demo here. it also lets you pull new WAL segments in and apply them to the current database. never got much time to go any further with it than this. https://just.billywhizz.io/sqlite/demo/#https://raw.githubusercontent.com/just-js/just.billywhizz.io/refs/heads/main/sqlite/demo/db/chinook.db https://just.billywhizz.io/sqlite/demo/#https://raw.githubus...

by simonw 11mo ago

simonw 11mo ago

I recovered it from https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/uktrade/sqlite-s3vfs https://archive.softwareheritage.org/browse/origin/directory... and pushed a fresh copy to GitHub here: https://github.com/simonw/sqlite-s3vfs https://github.com/simonw/sqlite-s3vfs This comment was helpful in figuring out how to get a full Git clone out of the heritage archive: https://news.ycombinator.com/item?id=37516523#37517378 https://news.ycombinator.com/item?id=37516523#37517378 Here's a TIL I wrote up of the process: https://til.simonwillison.net/github/software-archive-recovery https://til.simonwillison.net/github/software-archive-recove...

AceJohnny2 11mo ago
didn't you do something similar for Datasette, Simon?
- simonw 11mo ago
  Nothing smart with HTTP range requests yet - I have https://lite.datasette.io https://lite.datasette.io which runs the full Python server app in the browser via WebAssembly and Pyodide but it still works by fetching the entire SQLite file at once.
  - AceJohnny2 11mo ago
    oh! I must've been confused with your TIL where you linked to an explainer of this technique https://simonwillison.net/2021/May/2/hosting-sqlite-databases-on-github-pages/ https://simonwillison.net/2021/May/2/hosting-sqlite-database... https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
QuantumNomad_ 11mo ago
I also have a locally cloned copy of that repo from when it was on GitHub. Same latest commit as your copy of it. From what I see in GitHub in your copy of the repo, it looks like you don’t have the tags. Do you have the tags locally? If you don’t have the tags, I can push a copy of the repo to GitHub too and you can get the tags from my copy.
- simonw 11mo ago
  I don't have the tags! It would be awesome if you could push that.
  - QuantumNomad_ 11mo ago
    Uploaded here: https://github.com/Quantum-Nomad/sqlite-s3vfs https://github.com/Quantum-Nomad/sqlite-s3vfs
    - simonw 11mo ago
      Thanks for that, though actually it turns out I had them after all - I needed to run: git push --tags origin
      - QuantumNomad_ 11mo ago
        All the better :)
bspammer 11mo ago
Doing all this in an hour is such a good example of how absurdly efficient you can be with LLMs.
- socialcommenter 11mo ago
  From reading the TIL, it doesn't appear as if Simon used LLM for a large portion of what he did; only the initial suggestion to check the archive, and the web tool to make his process reproducible. Also, if you read the script from his chat with Claude code, the prompt really does the heavy lifting. Sure, the LLM fills in all the boilerplate and makes an easy-to-use, reproducible tool with loads of documentation, and credit for that. But is it not more accurate to say that Simon is absurdly efficient, LLM or sans LLM? :)

billywhizz 11mo ago

i played around with this a while back. you can see a demo here. it also lets you pull new WAL segments in and apply them to the current database. never got much time to go any further with it than this. https://just.billywhizz.io/sqlite/demo/#https://raw.githubusercontent.com/just-js/just.billywhizz.io/refs/heads/main/sqlite/demo/db/chinook.db https://just.billywhizz.io/sqlite/demo/#https://raw.githubus...