5 ms·

Ask HN: What is the best way to archive a webpage

16 points by badwolff 9y ago

I am working with some peers who have a website that links to and catalogs a number of resources (think blog posts). It would be ideal for the administrators to be able to archive or have a copy of that web page on their server in case of the original post being deleted, links moving, servers being down etc. Currently they are using http://archive.is to implement a half solution to this intent. It does not work for some websites and ideally they could host their own archived copy. What are easy solutions to do this? With Python I was thinking requests - but this would just grab the HTML and not images, or content generated by javascript. Thinking Selenium, you could take a screenshot of the content - not the most user friendly to read. What are some other solutions?

inceptionnames 9y ago
Save the page using the browser's Save feature and zip the created assets (a html file plus directory with graphics, js, css, etc) for ease of sharing.
cimmanom 9y ago
If it's not doing silly things like using Javascript to load static content, wget can do recursive crawls.
mdaniel 9y ago
I've enjoyed great success with various archiving proxies, including https://github.com/internetarchive/warcprox#readme https://github.com/internetarchive/warcprox#readme and https://github.com/zaproxy/zaproxy#readme https://github.com/zaproxy/zaproxy#readme (which saves the content to an embedded database, and can be easier to work with than warc files). The benefit of those approaches over just save-as from the browser is that almost by definition the proxy will save all the components required to re-render the page, whereas save will only grab the parts it sees at that time. If it's a public page, you can submit the URL to the Internet Archive, and benefit both you and them
anotheryou 9y ago
perma.cc looks sweet, but it's very limited for private people
- adultSwim 9y ago
  Run it yourself https://github.com/harvard-lil/perma https://github.com/harvard-lil/perma
  - anotheryou 9y ago
    I especially like them for their rescue plan and that it will stay online no matter what I do...
tyingq 9y ago
If you’re okay with easy, but saves at a third party: https://www.npmjs.com/package/archive.is https://www.npmjs.com/package/archive.is
ibn_ibid 9y ago
WebRecorder is your friend! https://webrecorder.io https://webrecorder.io It can capture content such as images and JS and also do offline playback. You can use the web interface or run your own instance https://github.com/webrecorder/webrecorder https://github.com/webrecorder/webrecorder
adultSwim 9y ago
Either curl or wget will get you pretty far. Learn one of them well. They are basically equivalent. I use curl. For current web apps, there is an interactive archiver written in Python, Web Recorder. It captures the full bi-directional traffic of a session. https://webrecorder.io/ https://webrecorder.io/ Web Recorder uses an internal Python library, pywb. That might be a good place to look. https://github.com/webrecorder/pywb https://github.com/webrecorder/pywb It looks like Selenium has done a lot of catching up on it's interface. I'd be curious how they compare now. Talk to librarians about archiving the web. They made Internet Archive and have a lot of experience.

16 points by badwolff 9y ago

inceptionnames 9y ago

Save the page using the browser's Save feature and zip the created assets (a html file plus directory with graphics, js, css, etc) for ease of sharing.

cimmanom 9y ago

If it's not doing silly things like using Javascript to load static content, wget can do recursive crawls.

mdaniel 9y ago

I've enjoyed great success with various archiving proxies, including https://github.com/internetarchive/warcprox#readme https://github.com/internetarchive/warcprox#readme and https://github.com/zaproxy/zaproxy#readme https://github.com/zaproxy/zaproxy#readme (which saves the content to an embedded database, and can be easier to work with than warc files). The benefit of those approaches over just save-as from the browser is that almost by definition the proxy will save all the components required to re-render the page, whereas save will only grab the parts it sees at that time. If it's a public page, you can submit the URL to the Internet Archive, and benefit both you and them

anotheryou 9y ago

perma.cc looks sweet, but it's very limited for private people

adultSwim 9y ago
Run it yourself https://github.com/harvard-lil/perma https://github.com/harvard-lil/perma
- anotheryou 9y ago
  I especially like them for their rescue plan and that it will stay online no matter what I do...

tyingq 9y ago

If you’re okay with easy, but saves at a third party: https://www.npmjs.com/package/archive.is https://www.npmjs.com/package/archive.is

ibn_ibid 9y ago

WebRecorder is your friend! https://webrecorder.io https://webrecorder.io It can capture content such as images and JS and also do offline playback. You can use the web interface or run your own instance https://github.com/webrecorder/webrecorder https://github.com/webrecorder/webrecorder

adultSwim 9y ago

Either curl or wget will get you pretty far. Learn one of them well. They are basically equivalent. I use curl. For current web apps, there is an interactive archiver written in Python, Web Recorder. It captures the full bi-directional traffic of a session. https://webrecorder.io/ https://webrecorder.io/ Web Recorder uses an internal Python library, pywb. That might be a good place to look. https://github.com/webrecorder/pywb https://github.com/webrecorder/pywb It looks like Selenium has done a lot of catching up on it's interface. I'd be curious how they compare now. Talk to librarians about archiving the web. They made Internet Archive and have a lot of experience.