4 ms·

I just launched a similar service called https://www.DailySiteSnap.com https://www.DailySiteSnap.com that screenshots, emails, and archives a specified web site

by derwiki 13y ago

I just launched a similar service called https://www.DailySiteSnap.com https://www.DailySiteSnap.com that screenshots, emails, and archives a specified web site on a daily basis. My use case is to be able to look back at any one day and see what my site looks like, since Archive.org doesn't refresh my page as often as I update it. Disclaimer: I'm really not trying to over-market myself, but I figured readers of this thread might be interested in my project. Happy to take down this post if it's read as too spammy.

toomuchtodo 13y ago
You can always package your site up in a format Archive.org can read: http://www.archiveteam.org/index.php?title=Wget_with_WARC_output http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou... http://www.archiveteam.org/index.php?title=The_WARC_Ecosystem http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste... http://justsolve.archiveteam.org/wiki/WARC http://justsolve.archiveteam.org/wiki/WARC http://warcreate.com/ http://warcreate.com/
- derwiki 13y ago
  Thanks for the info, was unaware of ARC/WARC formats. That said, I still think many people are looking for something simpler/easier, and a daily screenshot is good enough. Particularly, it will guarantee preserved formatting as browsers continue to evolve.
  - toomuchtodo 13y ago
    I agree that screenshots are easier; WARC files are future-proof.
  - Cameron_D 13y ago
    You can probably make your service do both screenshots and WARC, instead of loading a site directly, load it through WARC Proxy (https://github.com/odie5533/WarcProxy https://github.com/odie5533/WarcProxy), that will write out a WARC file and you can still store your screenshot. Once you have the WARCs you can upload them to Archive.org and they can be added to the wayback, or you can set up your own service for browsing them, built off something like warc-proxy https://github.com/alard/warc-proxy https://github.com/alard/warc-proxy (Yeah, same name different purpose...) There is also a MITM version of WARCProxy that will let you store HTTPS sites: https://github.com/odie5533/WarcMITMProxy https://github.com/odie5533/WarcMITMProxy
    - hncommenter13 13y ago
      As of version 1.14, wget natively supports warc (including built-in gzip and cdx index file generation). http://www.archiveteam.org/index.php?title=Wget_with_WARC_output http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou... This makes creating a browse-able mirror of a site in warc format fairly straightforward, as wget will automatically make links relative, as well as fetch requisite files (css, js, images) for each page.
      - Cameron_D 13y ago
        Yeah, but as far as I can guess, derwiki's service doesn't use wget, so running a proxy to store the WARCs is the next-simplest thing.
        toomuchtodo 13y ago
        If his service runs on any sort of Linux distro, its stupid simple to call wget with a system call. Wget comes standard with all of the most popular distros.
d0n 13y ago
That would be nice, if you mailed the screenshots. But it looks like you just send a link to the screenshot.

by derwiki 13y ago

toomuchtodo 13y ago

You can always package your site up in a format Archive.org can read: http://www.archiveteam.org/index.php?title=Wget_with_WARC_output http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou... http://www.archiveteam.org/index.php?title=The_WARC_Ecosystem http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste... http://justsolve.archiveteam.org/wiki/WARC http://justsolve.archiveteam.org/wiki/WARC http://warcreate.com/ http://warcreate.com/

derwiki 13y ago
Thanks for the info, was unaware of ARC/WARC formats. That said, I still think many people are looking for something simpler/easier, and a daily screenshot is good enough. Particularly, it will guarantee preserved formatting as browsers continue to evolve.
- toomuchtodo 13y ago
  I agree that screenshots are easier; WARC files are future-proof.
- Cameron_D 13y ago
  You can probably make your service do both screenshots and WARC, instead of loading a site directly, load it through WARC Proxy (https://github.com/odie5533/WarcProxy https://github.com/odie5533/WarcProxy), that will write out a WARC file and you can still store your screenshot. Once you have the WARCs you can upload them to Archive.org and they can be added to the wayback, or you can set up your own service for browsing them, built off something like warc-proxy https://github.com/alard/warc-proxy https://github.com/alard/warc-proxy (Yeah, same name different purpose...) There is also a MITM version of WARCProxy that will let you store HTTPS sites: https://github.com/odie5533/WarcMITMProxy https://github.com/odie5533/WarcMITMProxy
  - hncommenter13 13y ago
    As of version 1.14, wget natively supports warc (including built-in gzip and cdx index file generation). http://www.archiveteam.org/index.php?title=Wget_with_WARC_output http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou... This makes creating a browse-able mirror of a site in warc format fairly straightforward, as wget will automatically make links relative, as well as fetch requisite files (css, js, images) for each page.
    - Cameron_D 13y ago
      Yeah, but as far as I can guess, derwiki's service doesn't use wget, so running a proxy to store the WARCs is the next-simplest thing.
      - toomuchtodo 13y ago
        If his service runs on any sort of Linux distro, its stupid simple to call wget with a system call. Wget comes standard with all of the most popular distros.

d0n 13y ago

That would be nice, if you mailed the screenshots. But it looks like you just send a link to the screenshot.