3 ms·

There are ways to do this, but GSA just doesn’t do it. Years ago they used to have a system that would combine all the code.jsons into a single db and provide

There are ways to do this, but GSA just doesn’t do it. Years ago they used to have a system that would combine all the code.jsons into a single db and provide a query interface. They stopped funding that system and redesigned this static site. But could have used GitHub actions or something to fetch and combine the code.jsons and do everything client side. That still wouldn’t have needed maintenance costs.

westurner 3y ago
Here's a way to scrape URLs to JSON/YAML and then build static HTML with Hugo in a GitHub Action: https://github.com/jackyzha0/hugo-obsidian https://github.com/jackyzha0/hugo-obsidian datasette is a webapp and CLI built on SQLite and Python. datasette-lite is the pyodide + WebAssembly build of datasette which can be served as static HTML, JS, and WASM SQlite. datasette: https://github.com/simonw/datasette https://github.com/simonw/datasette : > Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. > Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world. From "Deploying a live Datasette demo when the tests pass" (2022) https://til.simonwillison.net/github-actions/deploy-live-demo-when-tests-pass#user-content-deploying-to-vercel https://til.simonwillison.net/github-actions/deploy-live-dem... : datasette publish vercel fixtures.db [...] The `datasette publish` command supports Google Cloud Run, Heroku, Vercel, Fly, [Full Container or Serverless] https://docs.datasette.io/en/stable/publish.html https://docs.datasette.io/en/stable/publish.html datasette-lite: https://github.com/simonw/datasette-lite https://github.com/simonw/datasette-lite : > You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.* > [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you > To load a Parquet file, pass a URL to `?parquet=` > [...] https://lite.datasette.io/?parquet=https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet https://lite.datasette.io/?parquet=https://github.com/Terada...* There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet. datasette plugins are written in Python and/or JS w/ pluggy: https://docs.datasette.io/en/stable/plugins.html https://docs.datasette.io/en/stable/plugins.html https://datasette.io/plugins https://datasette.io/plugins datasette-scraper scrapes sitemaps.xml and crawls though it could surely be repurposed to instead scrape a list of code.json URLs within the datasette process, which is powered by asyncio and the asynchronous uvicorn ASGI HTTP web server. datasette-scraper/#architecture: https://github.com/cldellow/datasette-scraper/#architecture https://github.com/cldellow/datasette-scraper/#architecture (TIL datasette-scraper parses HTML with selectolax; and Selectolax with Modest or Lexbor is ~25x faster at HTML parsing than BeautifulSoup in the selectolax benchmark: https://github.com/rushter/selectolax#simple-benchmark https://github.com/rushter/selectolax#simple-benchmark ) (Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_built_with_Nutch https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct https://github.com/scrapinghub/extruct ) datasette-graphql adds a GraphQL HTTP API to a SQLite database: https://datasette.io/plugins/datasette-graphql https://datasette.io/plugins/datasette-graphql plugins?q=sqlite: https://datasette.io/plugins?q=sqlite https://datasette.io/plugins?q=sqlite datasette-sqlite-fts4: https://datasette.io/plugins/datasette-sqlite-fts4 https://datasette.io/plugins/datasette-sqlite-fts4 ; Full-Text Search with SQLite datasette-ripgrep: "deploy a regular expression search engine for your source code": https://github.com/simonw/datasette-ripgrep https://github.com/simonw/datasette-ripgrep Seeing as there's already a JSONLD @context (schema) for code.json, CSVW as JSONLD and/or YAMLLD would be an easy way merge Linked Data graphs of tabular data: https://github.com/semantalytics/awesome-semantic-web#csvw https://github.com/semantalytics/awesome-semantic-web#csvw A GitHub Action would run regularly, fetch each code.json, save each to a git repo, and then upsert each into a SQLite database to be published with e.g. datasette or datasette-lite.

westurner 3y ago

Here's a way to scrape URLs to JSON/YAML and then build static HTML with Hugo in a GitHub Action: https://github.com/jackyzha0/hugo-obsidian https://github.com/jackyzha0/hugo-obsidian datasette is a webapp and CLI built on SQLite and Python. datasette-lite is the pyodide + WebAssembly build of datasette which can be served as static HTML, JS, and WASM SQlite. datasette: https://github.com/simonw/datasette https://github.com/simonw/datasette : > Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. > Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world. From "Deploying a live Datasette demo when the tests pass" (2022) https://til.simonwillison.net/github-actions/deploy-live-demo-when-tests-pass#user-content-deploying-to-vercel https://til.simonwillison.net/github-actions/deploy-live-dem... : datasette publish vercel fixtures.db [...] The `datasette publish` command supports Google Cloud Run, Heroku, Vercel, Fly, [Full Container or Serverless] https://docs.datasette.io/en/stable/publish.html https://docs.datasette.io/en/stable/publish.html datasette-lite: https://github.com/simonw/datasette-lite https://github.com/simonw/datasette-lite : > You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.* > [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you > To load a Parquet file, pass a URL to `?parquet=` > [...] https://lite.datasette.io/?parquet=https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet https://lite.datasette.io/?parquet=https://github.com/Terada...* There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet. datasette plugins are written in Python and/or JS w/ pluggy: https://docs.datasette.io/en/stable/plugins.html https://docs.datasette.io/en/stable/plugins.html https://datasette.io/plugins https://datasette.io/plugins datasette-scraper scrapes sitemaps.xml and crawls though it could surely be repurposed to instead scrape a list of code.json URLs within the datasette process, which is powered by asyncio and the asynchronous uvicorn ASGI HTTP web server. datasette-scraper/#architecture: https://github.com/cldellow/datasette-scraper/#architecture https://github.com/cldellow/datasette-scraper/#architecture (TIL datasette-scraper parses HTML with selectolax; and Selectolax with Modest or Lexbor is ~25x faster at HTML parsing than BeautifulSoup in the selectolax benchmark: https://github.com/rushter/selectolax#simple-benchmark https://github.com/rushter/selectolax#simple-benchmark ) (Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_built_with_Nutch https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct https://github.com/scrapinghub/extruct ) datasette-graphql adds a GraphQL HTTP API to a SQLite database: https://datasette.io/plugins/datasette-graphql https://datasette.io/plugins/datasette-graphql plugins?q=sqlite: https://datasette.io/plugins?q=sqlite https://datasette.io/plugins?q=sqlite datasette-sqlite-fts4: https://datasette.io/plugins/datasette-sqlite-fts4 https://datasette.io/plugins/datasette-sqlite-fts4 ; Full-Text Search with SQLite datasette-ripgrep: "deploy a regular expression search engine for your source code": https://github.com/simonw/datasette-ripgrep https://github.com/simonw/datasette-ripgrep Seeing as there's already a JSONLD @context (schema) for code.json, CSVW as JSONLD and/or YAMLLD would be an easy way merge Linked Data graphs of tabular data: https://github.com/semantalytics/awesome-semantic-web#csvw https://github.com/semantalytics/awesome-semantic-web#csvw A GitHub Action would run regularly, fetch each code.json, save each to a git repo, and then upsert each into a SQLite database to be published with e.g. datasette or datasette-lite.