5 ms·
Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any
by jl6 10y ago
Screenshotting or PDFing of a website is an increasingly important archiving tool, to supplement wget. I've come across a lot of websites that won't render any content if not connected to a live server.
- nikisweeting 10y agoI couldn't agree more. I wish more sites would load without needing multiple seconds of JS execution and AJAX. One of my TODOs is to get full-page screenshots working as well.
- AznHisoka 10y agoin some cases it leads to worse horrible experiences like my Linkedin feed. I have to wait a few seconss everytime before content loads.
- zulln 10y ago> One of my TODOs is to get full-page screenshots working as well. https://medium.com/@dschnr/using-headless-chrome-as-an-automated-screenshot-tool-4b07dffba79a https://medium.com/@dschnr/using-headless-chrome-as-an-autom...
- tomcooks 10y agoYou might find this useful if you use Firefox: - shift+f2 - screenshot urcodiaz.png --fullpage
- Drdrdrq 10y agoThank you for this tip! I have always installed Abduction plugin if I needed this, but the less plugins the better.
- jccalhoun 10y agoAgreed. I research new media and archive.org is invaluable to me. I worry that current web sites won't be able to be preserved. (much like many of the flash sites and real audio of the past are largely gone.)
- nikisweeting 10y agoSpeaking of audio, one of my highest priority TODO's for this project is to use youtube-dl for nicely archiving youtube & soundcloud links.
- driverdan 10y agoPDFs are really not suitable for archiving websites since they're designed around pages and the web does not have pages. A better option is to render a page with JS turned on and save the resulting HTML.
- TazeTSchnitzel 10y agoPDFs have the advantage of being a fixed format that should display the same everywhere. It's probably less fragile than modern CSS or HTML. However, as you point out, PDFs are designed around the printed page, not the flowing arbitrary-page-size documents of the web.
- nikisweeting 10y agoI was doing this before with the chrome --dump-dom flag but the output I was getting was garbage, and no more useful than the simpler wget download. PDF turned out to produce really nice, readable archives about 75% of the time, so I kept it in. Text-based sites tend to do a good job of having PDF-friendly styling.
- ianbicking 10y agoI've been trying to do something like this in this project: https://github.com/ianb/pagearchive https://github.com/ianb/pagearchive It came out of a screenshot/archiver I've been working at at Mozilla, but I've split it up as the screenshotting is shipping and DOM archiving is still way outside Mozilla's comfort zone.
- zulln 10y agoI googled it, and --dump-dom simply dumps the output of document.body.innerHTML. I have not used headless chrome at all, but I imagine it would not be that hard to get it to dump document.documentElement.outerHTML instead. (execute JavaScript on the page is most likely possible?)
- robzyb 10y agoWouldn't a copy of the DOM be even better than a screenshot? I.e. DOM copy > screenshot > wget?
- nikisweeting 10y agoI mentioned it below, but I tried getting DOM snapshots using chrome --dump-dom, but the output usually didn't render well without a <head> section. (chrome only outputs the <body>) I could attach the headers from the wget file... but then it starts getting messy and complicated.
- frik 10y agoBut what do you do when a website has a broken media query, essentially destroying the print layout? Then a PDF is useless. Well, I took a screenshot, better than nothing.