3 ms·
Jina.ai offer a really neat (currently free) API for this - you add https://r.jina.ai/ https://r.jina.ai/ on the beginning of any API and it gives you back a Ma
by simonw 2y ago
Jina.ai offer a really neat (currently free) API for this - you add https://r.jina.ai/ https://r.jina.ai/ on the beginning of any API and it gives you back a Markdown version of the main content of that page, suitable for piping into an LLM.
Here's an example: https://r.jina.ai/https://simonwillison.net/2024/Sep/2/anatomy-of-a-textual-user-interface/ https://r.jina.ai/https://simonwillison.net/2024/Sep/2/anato... - for this page: https://simonwillison.net/2024/Sep/2/anatomy-of-a-textual-user-interface/ https://simonwillison.net/2024/Sep/2/anatomy-of-a-textual-us...
Their code is open source so you can run your own copy if you like: https://github.com/jina-ai/reader https://github.com/jina-ai/reader - it's written in TypeScript and uses Puppeteer and https://github.com/mozilla/readability https://github.com/mozilla/readability
I've been using Readability (minus the Markdown) bit myself to extract the title and main content from a page - I have a recipe for running it via Playwright using my shot-scraper tool here: https://shot-scraper.datasette.io/en/stable/javascript.html#example-extracting-page-content-with-readability-js https://shot-scraper.datasette.io/en/stable/javascript.html#...
shot-scraper javascript https://simonwillison.net/2024/Sep/2/anatomy-of-a-textual-user-interface/ "
async () => {
const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
return (new readability.Readability(document)).parse();
}"
- BeetleB 2y ago+1 for this. I too use Readability via Simon's shot-scraper tool.