6 ms·

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

npunt 6y ago
I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.
- vermilingua 6y ago
  And disk life.
  - philjohn 6y ago
    perversely, wouldn't spinning drives up and down impact drive life?
    - chemicalnovae 6y ago
      Probably yes, but only if it was happening a lot; I don’t know what the cross-over point would be though that you’re better off just keeping the drives spinning...
- bnewbold 6y ago
  The disks are spinning all the time, and most disks are seeing fairly frequent reads to some content or another. A lot of content is very rarely accesses, but almost every disk has some content which gets accessed. If spinning disks had only frequently-accessed content, they would be unable to keep up with the read rate or read throughput, things balance out reasonably on average. Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request. Note: I work at IA but am not on core infrastructure team
  - npunt 6y ago
    Interesting thanks for the insights! Then would the few second delay be more a matter of time it takes to decompress the contents, or that files are stored on disks which are being accessed a lot, or something else? Always been curious about it.
kilroy123 6y ago
This is my big gripe as well, it's so painfully slow. Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.
- beckman466 6y ago
  I'm surprised to read this and OP's comment, for me it's always quite fast. Are there specific websites your request - are they media heavy?
  - coldpie 6y ago
    I picked a random article from my browser's history, linked below. Just loading the snapshot year pages took about 10 seconds, then the snapshot hover took another 10 seconds. Finally, fully loading the snapshot page took about 50 seconds. So that's roughly 90 seconds to go from inserting a URL into the search bar and actually having the rendered page. Not unacceptable, but certainly slow by modern standards. http://web.archive.org/web/20120801000000*/http://blogs.msdn.com/b/larryosterman/archive/2005/06/24/432386.aspx http://web.archive.org/web/20120801000000*/http://blogs.msdn...
    - carapace 6y ago
      90s is better than ∞ (infinity) which is what it would be if the Archive didn't exist, eh?
      - coldpie 6y ago
        The original comment was "the Archive is painfully slow", the reply to that was "for me it's always quite fast", so I gave it some actual hard data explaining that 90s is in fact quite slow and maybe even painfully so in 2021. I have absolutely no idea how you got from that discussion to a hypothetical where the Archive doesn't exist. Obviously 90s is better than it not existing. I even said in my comment that it was not unacceptable. Why did you even make this comment?
ignoramous 6y ago
Wayback Machine started life at Amazon's Search project. May be it never recovered from that :) https://archive.is/2B2ts https://archive.is/2B2ts
shaunparker 6y ago
They want you to experience what it was like to browse wayback in the early 90s. Sorry, I couldn’t resist the joke :)
bnewbold 6y ago
Performance is fun! One aspect is that our data centers are in California, with no CDN. If you are on the other side of the world, you will have higher round-trip latency on every request, for all services. Another is layers of caching. Popular or recently requested Wayback content is more likely to be in either an explicit cache (eg, redis), or implicitly in kernel page caches across all layers of the request. Every wayback replay request hits several layers of index indexes (sorted by domain, path, and timestamp), which are huge and thus actually served from spinning disk over HTTP (!). This includes a timeline summary for the primary document, to display the banner. Then the actual raw records are fetched from another spinning disk over HTTP. This may result in one or more layers of internal redirect (additional fetches) if there was a "revisit" (identical HTTP body content, same URL, different timestamp). Then finally the record is re-written for replay (for HTML, CSS, Javascript, etc, unless the raw record was requested). Some pages will have many sub-resources, so this process is repeated many times, but that is the same as page load and you can see which resources are slow or not. As mentioned in the video, depending on where we are in the network hardware upgrade lifecycle, sometimes outbound bandwidth is tight also, which slows down transfer. And of course most of these services operate without a ton of overhead, so if there is a spike in traffic everything will slow down a bit. There is a lot of multi-tenancy-like situations also, so if there is a very popular zip file or Flash game getting served from the same storage disk as the WARC file holding a wayback resource, the replay for that specific resource will be slow due to disk I/O contention. If you are curious about why a specific HTML wayback replay was slow, you can look in the source code of the re-written document and see some timing numbers. Several organizations run large web archives that operate similarly to web.archive.org, and have described cost/benefit trade offs for different components. Eg, National Library of Australia has an alternative CDX index called OutbackCDX, which uses RocksDB on SSDs. I believe other folks store WARC files in S3 or S3-like object storage systems. The Wayback Machine is somewhat unique in the amount of (read) traffic it gets, the heterogeneity of archived content (from several crawlers, in older ARC as well as WARC), volume of live crawling ("save paper now" results show up pretty fast in the main site, which is black magic), running on "boring" general purpose hardware, and deep integration with our general purpose storage cluster. Note: I work at IA but not on the Wayback system
Bestia0728 6y ago
why

Probably a good place to ask: is there a reason why Wayback Machine's archives often take forever to load? Most of content on IA loads pretty fast, so WB is a notable exception.

npunt 6y ago

I get the impression Wayback Machine data is stored in powered down drives and they only spin them up when someone accesses the data. That would explain the several second delay and it'd make sense that an archive wouldn't need 95% of its data ready to go at a moments notice since that'd be a terrible waste of power.

vermilingua 6y ago
And disk life.
- philjohn 6y ago
  perversely, wouldn't spinning drives up and down impact drive life?
  - chemicalnovae 6y ago
    Probably yes, but only if it was happening a lot; I don’t know what the cross-over point would be though that you’re better off just keeping the drives spinning...
bnewbold 6y ago
The disks are spinning all the time, and most disks are seeing fairly frequent reads to some content or another. A lot of content is very rarely accesses, but almost every disk has some content which gets accessed. If spinning disks had only frequently-accessed content, they would be unable to keep up with the read rate or read throughput, things balance out reasonably on average. Wayback content is on the same disks as most other content, in the form of WARC files, with individual records fetched out of the middle of WARC files via HTTP range request. Note: I work at IA but am not on core infrastructure team
- npunt 6y ago
  Interesting thanks for the insights! Then would the few second delay be more a matter of time it takes to decompress the contents, or that files are stored on disks which are being accessed a lot, or something else? Always been curious about it.

kilroy123 6y ago

This is my big gripe as well, it's so painfully slow. Still, I strongly support the work they do and think it's very important work. I also think they do a good job for their size and resources.

beckman466 6y ago
I'm surprised to read this and OP's comment, for me it's always quite fast. Are there specific websites your request - are they media heavy?
- coldpie 6y ago
  I picked a random article from my browser's history, linked below. Just loading the snapshot year pages took about 10 seconds, then the snapshot hover took another 10 seconds. Finally, fully loading the snapshot page took about 50 seconds. So that's roughly 90 seconds to go from inserting a URL into the search bar and actually having the rendered page. Not unacceptable, but certainly slow by modern standards. http://web.archive.org/web/20120801000000*/http://blogs.msdn.com/b/larryosterman/archive/2005/06/24/432386.aspx http://web.archive.org/web/20120801000000*/http://blogs.msdn...
  - carapace 6y ago
    90s is better than ∞ (infinity) which is what it would be if the Archive didn't exist, eh?
    - coldpie 6y ago
      The original comment was "the Archive is painfully slow", the reply to that was "for me it's always quite fast", so I gave it some actual hard data explaining that 90s is in fact quite slow and maybe even painfully so in 2021. I have absolutely no idea how you got from that discussion to a hypothetical where the Archive doesn't exist. Obviously 90s is better than it not existing. I even said in my comment that it was not unacceptable. Why did you even make this comment?

ignoramous 6y ago

Wayback Machine started life at Amazon's Search project. May be it never recovered from that :) https://archive.is/2B2ts https://archive.is/2B2ts

shaunparker 6y ago

They want you to experience what it was like to browse wayback in the early 90s. Sorry, I couldn’t resist the joke :)

bnewbold 6y ago

Performance is fun! One aspect is that our data centers are in California, with no CDN. If you are on the other side of the world, you will have higher round-trip latency on every request, for all services. Another is layers of caching. Popular or recently requested Wayback content is more likely to be in either an explicit cache (eg, redis), or implicitly in kernel page caches across all layers of the request. Every wayback replay request hits several layers of index indexes (sorted by domain, path, and timestamp), which are huge and thus actually served from spinning disk over HTTP (!). This includes a timeline summary for the primary document, to display the banner. Then the actual raw records are fetched from another spinning disk over HTTP. This may result in one or more layers of internal redirect (additional fetches) if there was a "revisit" (identical HTTP body content, same URL, different timestamp). Then finally the record is re-written for replay (for HTML, CSS, Javascript, etc, unless the raw record was requested). Some pages will have many sub-resources, so this process is repeated many times, but that is the same as page load and you can see which resources are slow or not. As mentioned in the video, depending on where we are in the network hardware upgrade lifecycle, sometimes outbound bandwidth is tight also, which slows down transfer. And of course most of these services operate without a ton of overhead, so if there is a spike in traffic everything will slow down a bit. There is a lot of multi-tenancy-like situations also, so if there is a very popular zip file or Flash game getting served from the same storage disk as the WARC file holding a wayback resource, the replay for that specific resource will be slow due to disk I/O contention. If you are curious about why a specific HTML wayback replay was slow, you can look in the source code of the re-written document and see some timing numbers. Several organizations run large web archives that operate similarly to web.archive.org, and have described cost/benefit trade offs for different components. Eg, National Library of Australia has an alternative CDX index called OutbackCDX, which uses RocksDB on SSDs. I believe other folks store WARC files in S3 or S3-like object storage systems. The Wayback Machine is somewhat unique in the amount of (read) traffic it gets, the heterogeneity of archived content (from several crawlers, in older ARC as well as WARC), volume of live crawling ("save paper now" results show up pretty fast in the main site, which is black magic), running on "boring" general purpose hardware, and deep integration with our general purpose storage cluster. Note: I work at IA but not on the Wayback system

Bestia0728 6y ago

why