3 ms·

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are avai

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? I know about the web.archive.org/https:// https://... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after receiving the whole HTML response. A REST API would help thirdparty clients to know about this in advance, and the API documentations I found were super unclear in whether something like this exists or not. Context: I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it. On top of that I have no idea how to "un-UI" the web archived versions. I know that the wget user agent somehow leads to this, but it's also kind of undocumented how the webserver of IA does this in the background and when exactly the UI is injected and all the URLs are rewritten. Something like maybe a http request header to get the raw actual source would be nice.

LegionMammal978 5y ago
Regarding your last point, adding "id_" to the end of the timestamp in the URL produces the original downloaded file. Adding "if_" produces the page with rewritten links but without the Wayback Machine header. Additionally, if you look at the HTTP response headers on any version of the archived page, they include copies of the original response headers.
cxr 5y ago
> Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? Have you looked at the headers that they send? GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2 [...] HTTP/2 200 OK [...] link: <http://arxiv.org/abs/0911.1112>; rel="original", <https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/http://arxiv.org/abs/0911.1112>; rel="timegate", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>; rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT", <https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>; rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT" [...] The Wayback Machine implements RFC 7089. <http://mementoweb.org/guide/quick-intro/ http://mementoweb.org/guide/quick-intro/>
wolverine876 5y ago
> I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it. Brave browser does that, and I've seen Firefox add-ons that do it. Maybe you can look at their code.

LegionMammal978 5y ago

Regarding your last point, adding "id_" to the end of the timestamp in the URL produces the original downloaded file. Adding "if_" produces the page with rewritten links but without the Wayback Machine header. Additionally, if you look at the HTTP response headers on any version of the archived page, they include copies of the original response headers.

cxr 5y ago

> Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache? Have you looked at the headers that they send? GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2 [...] HTTP/2 200 OK [...] link: <http://arxiv.org/abs/0911.1112>; rel="original", <https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/http://arxiv.org/abs/0911.1112>; rel="timegate", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>; rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT", <https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>; rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT", <https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>; rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT" [...] The Wayback Machine implements RFC 7089. <http://mementoweb.org/guide/quick-intro/ http://mementoweb.org/guide/quick-intro/>

wolverine876 5y ago

> I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it. Brave browser does that, and I've seen Firefox add-ons that do it. Maybe you can look at their code.