6 ms·

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage!

Hi, I manage the Wayback Machine at the Internet Archive. Very happy so many people here care about preserving, and making available, our cultural heritage! Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine. As noted the Internet Archive is experimenting with filecoin.io and storj.io and is always open to suggestions about how we might do our jobs better, and improve our service. We also host regular meetups (and have hosted summits and a camp) related to the Decentralized Web. See: https://blog.archive.org/tag/dweb/ https://blog.archive.org/tag/dweb/ The Internet Archive also offers archive-it.org, a subscription service, for those who want a higher level of support and more features. We appreciate any support you can offer, financial and otherwise. Please share any bug reports, feature suggestions and other feedback with us via email to info@archive.org Oh/and… checkout the new PDF Search feature we just launched at the bottom of web.archive.org. More to come like that in 2022. Finally, you might also find some of the things I wrote here of interest: https://gijn.org/2021/05/05/tips-for-using-the-internet-archives-wayback-machine-in-your-next-investigation/ https://gijn.org/2021/05/05/tips-for-using-the-internet-arch...

sayhar 5y ago
Thank you for your service!
chaps 5y ago
I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals. Cheers.
- gjs278 5y ago
- bcherny 5y ago
  What’s recursive scraping (presumably, following internal links), and what’s broken about it?
  - chaps 5y ago
    Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.
- smarx007 5y ago
  Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.
  - eslaught 5y ago
    There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.
    - chaps 5y ago
i67vw3 5y ago
mysterydip 5y ago
Just wanted to say thank you for all the times you've let me continue researching when the original sources became dead!
cookiengineer 5y ago
Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? I know about the web.archive.org/https:// https://... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after receiving the whole HTML response. A REST API would help thirdparty clients to know about this in advance, and the API documentations I found were super unclear in whether something like this exists or not. Context: I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it. On top of that I have no idea how to "un-UI" the web archived versions. I know that the wget user agent somehow leads to this, but it's also kind of undocumented how the webserver of IA does this in the background and when exactly the UI is injected and all the URLs are rewritten. Something like maybe a http request header to get the raw actual source would be nice.
- LegionMammal978 5y ago
  Regarding your last point, adding "id_" to the end of the timestamp in the URL produces the original downloaded file. Adding "if_" produces the page with rewritten links but without the Wayback Machine header. Additionally, if you look at the HTTP response headers on any version of the archived page, they include copies of the original response headers.
- cxr 5y ago
  > Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? Have you looked at the headers that they send? GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2 [...] HTTP/2 200 OK [...] link: <http://arxiv.org/abs/0911.1112>; rel="original", <https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/http://arxiv.org/abs/0911.1112>; rel="timegate", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>; rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT", <https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>; rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT" [...] The Wayback Machine implements RFC 7089. <http://mementoweb.org/guide/quick-intro/ http://mementoweb.org/guide/quick-intro/>
truemotive 5y ago
Thank you!

sayhar 5y ago

Thank you for your service!

chaps 5y ago

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals. Cheers.

gjs278 5y ago
bcherny 5y ago
What’s recursive scraping (presumably, following internal links), and what’s broken about it?
- chaps 5y ago
  Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.
smarx007 5y ago
Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.
- eslaught 5y ago
  There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.
  - chaps 5y ago

i67vw3 5y ago

mysterydip 5y ago

Just wanted to say thank you for all the times you've let me continue researching when the original sources became dead!

cookiengineer 5y ago

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? I know about the web.archive.org/https:// https://... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after receiving the whole HTML response. A REST API would help thirdparty clients to know about this in advance, and the API documentations I found were super unclear in whether something like this exists or not. Context: I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it. On top of that I have no idea how to "un-UI" the web archived versions. I know that the wget user agent somehow leads to this, but it's also kind of undocumented how the webserver of IA does this in the background and when exactly the UI is injected and all the URLs are rewritten. Something like maybe a http request header to get the raw actual source would be nice.

LegionMammal978 5y ago
Regarding your last point, adding "id_" to the end of the timestamp in the URL produces the original downloaded file. Adding "if_" produces the page with rewritten links but without the Wayback Machine header. Additionally, if you look at the HTTP response headers on any version of the archived page, they include copies of the original response headers.
cxr 5y ago
> Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? Have you looked at the headers that they send? GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2 [...] HTTP/2 200 OK [...] link: <http://arxiv.org/abs/0911.1112>; rel="original", <https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/http://arxiv.org/abs/0911.1112>; rel="timegate", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>; rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT", <https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>; rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT" [...] The Wayback Machine implements RFC 7089. <http://mementoweb.org/guide/quick-intro/ http://mementoweb.org/guide/quick-intro/>

truemotive 5y ago

Thank you!