3 ms·
Related to this: you can also use my shot-scraper tool to scrape web pages from the command line using JavaScript: % pip install shot-scraper % shot-sc
by simonw 5y ago
Related to this: you can also use my shot-scraper tool to scrape web pages from the command line using JavaScript:
% pip install shot-scraper
% shot-scraper install
% shot-scraper javascript https://datasette.io/ "({
title: document.title,
tagline: document.querySelector('.tagline').innerText
})"
{
"title": "Datasette: An open source multi-tool for exploring and publishing data",
"tagline": "An open source multi-tool for exploring and publishing data"
}
More here, including an example of using it to scrape data from Hacker News that's not available in the API: https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...
HN post about this from yesterday (which failed to get any traction): https://news.ycombinator.com/item?id=30667588 https://news.ycombinator.com/item?id=30667588
- marginalia_nu 5y agoThat's fantastic! I'll definitely investigate using this. I implemented my own MacGyver version of this basic functionality off selenium to grab screenshots for search.marginalia.nu/explore/random -- but that script is super sketchy and held together in with bubble gum and duct tape. Yours looks a lot better. By the way, is there a way to extract favicons as well?
- simonw 5y agoNo, I've not thought about favicons. That's a really interesting challenge. I wonder if there's a way to detect favicons just using JavaScript that runs against a page? Not sure if it's easy to detect /favicon.ico v.s. the various meta tags.
- simonw 5y agoWould be kind of fun to write JavaScript that runs against the page that first looks for the meta tags, then tries to fetch("/favicon.ico") and returns either the URL or a base64 encoded copy of the image (since the "shot-scraper javascript" command requires you to return JSON).
- marginalia_nu 5y agoThere's a lot of weird edge cases for favicons, most browsers fall back to just looking for /favicon.ico if you don't explicitly specify it in the meta tags, and if you do, there's sometimes different versions. Yeah, maybe it's a pipe dream :-/ But even without them, it looks really useful!
- Freeboots 5y agoI've found some fairly reliable repos in the past, or https://t1.gstatic.com/faviconV2?client=SOCIAL&type=FAVICON&fallback_opts=TYPE,SIZE,URL&url=http://news.ycombinator.com&size=64 https://t1.gstatic.com/faviconV2?client=SOCIAL&type=FAVICON&... still works (but for how long)