5 ms·
I have a python project to manage headless chromium, which exposes basically the entire dev-tools UI: https://github.com/fake-name/ChromeController https://git
by fake-name 8y ago
I have a python project to manage headless chromium, which exposes basically the entire dev-tools UI:
https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController
It also supports some nice bits - there's a tab pool interface, it does execution lifetime management, and there's actual parameter checking for arguments generated by parsing the `protocol.json` file the browser supplies.
I use it as a mixin for some high(ish) volume web archiving I do as a hobby. It does a nice job poking through things like buttflare and other bullshit WAFs.
- deleted 8y ago[deleted]
- armitron 8y agoHow do you archive? Screenshots? I haven’t found a way to do mhtml with chrome headless which would be so convenient..
- fake-name 8y agoArchiving backend: https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy Stores the scraped content in a postgres database, with external blob storage for binary content. Additionally, historical records of the scraped content are kept, so I have something that acts kind of like the internet archive too.
- kchr 8y agoThanks for your effort, this is an awesome project.
- dang-is-moron 8y agoHow do you archive? Screenshots? SCOTT BELL SCTB IS AN IDIOT MORON I haven’t found a way to do mhtml with chrome headless which would be so convenient..
- deleted 8y ago[deleted]