3 ms·

Thank you for shot-scraper! I've tested it in the past, but something severely missing from all screenshot tools, shot-scraper included, is a way to avoid scree

Thank you for shot-scraper! I've tested it in the past, but something severely missing from all screenshot tools, shot-scraper included, is a way to avoid screenshoting popups. For instance, newsletter or login popups, GDPR popups, etc. If shot-scraper has a reliable way of screenshoting websites while avoiding these popups, I would love to know. I'm on mobile so don't have access to my notes, but I'm pretty sure that a year ago when I tried there was no reliable way to screenshot e.g. the BBC news website without getting the popups. Again, thank you.

DANmode 3y ago
Screenshot the archive.org render?
grey8 3y ago
Just a thought, but what happens if you use orchestrate a browser instance with an installed ad blocker like uBlock Origin?
- walthamstow 3y ago
  This works well, I've done it before with Selenium and a headless Firefox with uBlock Origin and Bypass Paywalls installed.
cjr 3y ago
I work on a paid screenshot api[0] where we have features to either hide these banners and overlays using css, or alternatively we run some javascript to send a click event to what we detect as the 'accept' button in order to dismiss the popups. It's quite a painful problem and we screenshot many millions of sites a day, our success rate at detecting these is high but still not 100%. We have gotten quite far with heuristics and are exploring whether we can get better results by training a model. [0]:https://urlbox.com https://urlbox.com
- qwertox 3y ago
  Interesting service. Can the API provide a custom tag, comment or ID which will then be inserted in the output? Like in JPEG EXIF, PNG also knows Metadata, PDF description, HTML meta tag?
  - cjr 3y ago
    No - we don't currently support this feature as nobody has asked for it so far :) You could append a new meta tag by running some custom JS to add it, but we don't modify exif, metadata or pdf description at the moment.
simonw 3y ago
Try this: shot-scraper -h 800 'https://www.spiegel.de/international/' \ --wait-for "() => { const div = document.querySelector('[id^="sp_message_container"]'); if (div) { div.remove(); return true; } }" shot-scraper runs that --wait-for script until it returns true. In this case we're waiting for the cookie consent overlay div to show up and then removing it before we take the screenshot. Screenshots here: https://gist.github.com/simonw/de75355c39025f9a64548aa3366b16d7?permalink_comment_id=5000526#gistcomment-5000526 https://gist.github.com/simonw/de75355c39025f9a64548aa3366b1...

DANmode 3y ago

Screenshot the archive.org render?

grey8 3y ago

Just a thought, but what happens if you use orchestrate a browser instance with an installed ad blocker like uBlock Origin?

walthamstow 3y ago
This works well, I've done it before with Selenium and a headless Firefox with uBlock Origin and Bypass Paywalls installed.

cjr 3y ago

I work on a paid screenshot api[0] where we have features to either hide these banners and overlays using css, or alternatively we run some javascript to send a click event to what we detect as the 'accept' button in order to dismiss the popups. It's quite a painful problem and we screenshot many millions of sites a day, our success rate at detecting these is high but still not 100%. We have gotten quite far with heuristics and are exploring whether we can get better results by training a model. [0]:https://urlbox.com https://urlbox.com

qwertox 3y ago
Interesting service. Can the API provide a custom tag, comment or ID which will then be inserted in the output? Like in JPEG EXIF, PNG also knows Metadata, PDF description, HTML meta tag?
- cjr 3y ago
  No - we don't currently support this feature as nobody has asked for it so far :) You could append a new meta tag by running some custom JS to add it, but we don't modify exif, metadata or pdf description at the moment.

simonw 3y ago

Try this: shot-scraper -h 800 'https://www.spiegel.de/international/' \ --wait-for "() => { const div = document.querySelector('[id^="sp_message_container"]'); if (div) { div.remove(); return true; } }" shot-scraper runs that --wait-for script until it returns true. In this case we're waiting for the cookie consent overlay div to show up and then removing it before we take the screenshot. Screenshots here: https://gist.github.com/simonw/de75355c39025f9a64548aa3366b16d7?permalink_comment_id=5000526#gistcomment-5000526 https://gist.github.com/simonw/de75355c39025f9a64548aa3366b1...