3 ms·
Here's a way to scrape URLs to JSON/YAML and then build static HTML with Hugo in a GitHub Action: https://github.com/jackyzha0/hugo-obsidian https://github.com/
by westurner 3y ago
Here's a way to scrape URLs to JSON/YAML and then build static HTML with Hugo in a GitHub Action: https://github.com/jackyzha0/hugo-obsidian https://github.com/jackyzha0/hugo-obsidian
datasette is a webapp and CLI built on SQLite and Python. datasette-lite is the pyodide + WebAssembly build of datasette which can be served as static HTML, JS, and WASM SQlite.
datasette:
https://github.com/simonw/datasette https://github.com/simonw/datasette :
> Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.
> Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world.
From "Deploying a live Datasette demo when the tests pass" (2022) https://til.simonwillison.net/github-actions/deploy-live-demo-when-tests-pass#user-content-deploying-to-vercel https://til.simonwillison.net/github-actions/deploy-live-dem... :
datasette publish vercel fixtures.db [...]
The `datasette publish` command supports
Google Cloud Run, Heroku, Vercel, Fly, [Full Container or Serverless]
https://docs.datasette.io/en/stable/publish.html https://docs.datasette.io/en/stable/publish.html
datasette-lite:
https://github.com/simonw/datasette-lite https://github.com/simonw/datasette-lite :
> You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.*
> [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you
> To load a Parquet file, pass a URL to `?parquet=`
> [...] https://lite.datasette.io/?parquet=https://github.com/Teradata/kylo/blob/master/samples/sample-data/parquet/userdata1.parquet https://lite.datasette.io/?parquet=https://github.com/Terada...*
There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet.
datasette plugins are written in Python and/or JS w/ pluggy:
https://docs.datasette.io/en/stable/plugins.html https://docs.datasette.io/en/stable/plugins.html https://datasette.io/plugins https://datasette.io/plugins
datasette-scraper scrapes sitemaps.xml and crawls though it could surely be repurposed to instead scrape a list of code.json URLs within the datasette process, which is powered by asyncio and the asynchronous uvicorn ASGI HTTP web server.
datasette-scraper/#architecture:
https://github.com/cldellow/datasette-scraper/#architecture https://github.com/cldellow/datasette-scraper/#architecture
(TIL datasette-scraper parses HTML with selectolax; and Selectolax with Modest or Lexbor is ~25x faster at HTML parsing than BeautifulSoup in the selectolax benchmark:
https://github.com/rushter/selectolax#simple-benchmark https://github.com/rushter/selectolax#simple-benchmark )
(Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_built_with_Nutch https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct https://github.com/scrapinghub/extruct )
datasette-graphql adds a GraphQL HTTP API to a SQLite database:
https://datasette.io/plugins/datasette-graphql https://datasette.io/plugins/datasette-graphql
plugins?q=sqlite:
https://datasette.io/plugins?q=sqlite https://datasette.io/plugins?q=sqlite
datasette-sqlite-fts4: https://datasette.io/plugins/datasette-sqlite-fts4 https://datasette.io/plugins/datasette-sqlite-fts4 ; Full-Text Search with SQLite
datasette-ripgrep: "deploy a regular expression search engine for your source code":
https://github.com/simonw/datasette-ripgrep https://github.com/simonw/datasette-ripgrep
Seeing as there's already a JSONLD @context (schema) for code.json, CSVW as JSONLD and/or YAMLLD would be an easy way merge Linked Data graphs of tabular data:
https://github.com/semantalytics/awesome-semantic-web#csvw https://github.com/semantalytics/awesome-semantic-web#csvw
A GitHub Action would run regularly, fetch each code.json, save each to a git repo, and then upsert each into a SQLite database to be published with e.g. datasette or datasette-lite.