7 ms·

I'm really surprised they don't use more CDN for this. Anyone know the reason why it isn't served by something like cloudflare?

ajayyy 6y ago
It could be philosophical
- 1f60c 6y ago
  That's what I'm thinking. If they wanted to use Cloudflare, I'm pretty sure they'd jump at the opportunity.
adtac 6y ago
Please don't make the whole internet basically cloudflare. they've banned my VPN endpoint (Hetzner server) and as a result a huge chunk of websites already don't work for me despite me having done nothing wrong. I've heard reports of Tor users being restricted as well.
- SteveNuts 6y ago
  Okay, any one of the other CDNs then. It seems like the bandwidth costs and equipment are prohibitively expensive for them. There must be a reason they don't use them which is what I want to know.
  - dylz 6y ago
    Any of the other CDNs are absurdly prohibitively expensive, pay-per-GB, etc. And would still have to download from their source.
- judge2020 6y ago
  It's more likely that website owners are blocking the ASNs of hosting providers since those are often used for content scraping and exploit type attacks [since they're cheap]. There are public and private lists of hosting provider ASNs you can use to block them all if you generally only want visitors with a business/residential IP.
- manquer 6y ago
  It is usually nothing specific to your VPN setup or IP. It is mainly either for (D)DoS protection or Content Licensing related ( VPNs are easy way to bypass Geo-locked content). Many CDNs and services will allow you to access using only non commercial IP range
- mattl 6y ago
  So many of the VPS companies have bots galore doing credit card attacks on smaller sites that it’s very easy and convenient to simply block those VPS providers.
highclass 6y ago
At the bandwidth levels they are using they would need to use Cloudflare Enterprise, and in my experience that is way more expensive than other CDN providers. Also, since archive.org has so much content, the caching ratio is going to be very bad and kill CDN efficiency while still requiring lots of direct bandwidth. Cheap direct bandwidth in their case looks best(which is what they seem to be doing).
- highclass 6y ago
  Another aspect is they are actually using bandwidth to max and have a cap on usage spending. No CDN I know of allows capping bandwidth usage. The way they use bandwidth means the most efficent GB/dollar cost although at the cost of poor performance.
  - deleted 6y ago
    [deleted]
- unityByFreedom 6y ago
  Cloudflare does not have bandwidth limits.
  - deleted 6y ago
    [deleted]
EE84M3i 6y ago
The geographic distribution benefits for archive.org seem minimal to me - for the actual content high latency isn't a big deal.
- fomine3 6y ago
  It's really benefiticial for outside of US.
- iofiiiiiiiii 6y ago
  Geographic locationa affects far more than just latency. Even though you may have a 100 Mbps link on both client and server , only 10 Mbps might make it across the Atlantic Ocean. Connections between countries/ISPs are not unlimited in capacity!
jcrawfordor 6y ago
I don't foresee use of a CDN saving them much or any money. Cache hit ratio would probably be relatively low and the amount of storage required at the edge to cover an appreciable portion of traffic would be very expensive. E.g. their Akamai bill would be in the five figures per month if not six figures, depending on what kind of volume discount they could arrange, just based on outbound bandwidth. They're not the cheapest but also not the most expensive. Serving up that large of a media library at mid scale just isn't really a great use case for a CDN, those that have to due to transit costs becoming truly enormous (e.g. Netflix) make an enormous investment in hardware at the edge that probably isn't affordable to IA (or necessary at this point).
chronomex 6y ago
They serve everything in-house and are their own AS; this allows them to provide stronger guarantees of reader privacy than sites that use a CDN.
- Stratoscope 6y ago
  > They serve everything in-house and are their own AS Forgive me, but what is an AS?
  - Kalium 6y ago
    Autonomous System. https://en.wikipedia.org/wiki/Autonomous_system_(Internet) https://en.wikipedia.org/wiki/Autonomous_system_(Internet) An AS is an entity significant enough to be reasoned about at the BGP level.
    - Stratoscope 6y ago
      Thanks!
jonah-archive 6y ago
Sibling comments are exactly correct -- we're a library. We don't track or record the activity of our patrons (see e.g. https://archive.org/services/docs/api/views.html#footnote-what-is-the-deal-with-privacy-protecting-hashes-instead-of-ip-numbers-in-our-logs https://archive.org/services/docs/api/views.html#footnote-wh... ), and since most CDNs cannot offer the same guarantee, we can't use them.

ajayyy 6y ago

It could be philosophical

1f60c 6y ago
That's what I'm thinking. If they wanted to use Cloudflare, I'm pretty sure they'd jump at the opportunity.

adtac 6y ago

Please don't make the whole internet basically cloudflare. they've banned my VPN endpoint (Hetzner server) and as a result a huge chunk of websites already don't work for me despite me having done nothing wrong. I've heard reports of Tor users being restricted as well.

SteveNuts 6y ago
Okay, any one of the other CDNs then. It seems like the bandwidth costs and equipment are prohibitively expensive for them. There must be a reason they don't use them which is what I want to know.
- dylz 6y ago
  Any of the other CDNs are absurdly prohibitively expensive, pay-per-GB, etc. And would still have to download from their source.
judge2020 6y ago
It's more likely that website owners are blocking the ASNs of hosting providers since those are often used for content scraping and exploit type attacks [since they're cheap]. There are public and private lists of hosting provider ASNs you can use to block them all if you generally only want visitors with a business/residential IP.
manquer 6y ago
It is usually nothing specific to your VPN setup or IP. It is mainly either for (D)DoS protection or Content Licensing related ( VPNs are easy way to bypass Geo-locked content). Many CDNs and services will allow you to access using only non commercial IP range
mattl 6y ago
So many of the VPS companies have bots galore doing credit card attacks on smaller sites that it’s very easy and convenient to simply block those VPS providers.

highclass 6y ago

At the bandwidth levels they are using they would need to use Cloudflare Enterprise, and in my experience that is way more expensive than other CDN providers. Also, since archive.org has so much content, the caching ratio is going to be very bad and kill CDN efficiency while still requiring lots of direct bandwidth. Cheap direct bandwidth in their case looks best(which is what they seem to be doing).

highclass 6y ago
Another aspect is they are actually using bandwidth to max and have a cap on usage spending. No CDN I know of allows capping bandwidth usage. The way they use bandwidth means the most efficent GB/dollar cost although at the cost of poor performance.
- deleted 6y ago
  [deleted]
unityByFreedom 6y ago
Cloudflare does not have bandwidth limits.
- deleted 6y ago
  [deleted]

EE84M3i 6y ago

The geographic distribution benefits for archive.org seem minimal to me - for the actual content high latency isn't a big deal.

fomine3 6y ago
It's really benefiticial for outside of US.
iofiiiiiiiii 6y ago
Geographic locationa affects far more than just latency. Even though you may have a 100 Mbps link on both client and server , only 10 Mbps might make it across the Atlantic Ocean. Connections between countries/ISPs are not unlimited in capacity!

jcrawfordor 6y ago

I don't foresee use of a CDN saving them much or any money. Cache hit ratio would probably be relatively low and the amount of storage required at the edge to cover an appreciable portion of traffic would be very expensive. E.g. their Akamai bill would be in the five figures per month if not six figures, depending on what kind of volume discount they could arrange, just based on outbound bandwidth. They're not the cheapest but also not the most expensive. Serving up that large of a media library at mid scale just isn't really a great use case for a CDN, those that have to due to transit costs becoming truly enormous (e.g. Netflix) make an enormous investment in hardware at the edge that probably isn't affordable to IA (or necessary at this point).

chronomex 6y ago

They serve everything in-house and are their own AS; this allows them to provide stronger guarantees of reader privacy than sites that use a CDN.

Stratoscope 6y ago
> They serve everything in-house and are their own AS Forgive me, but what is an AS?
- Kalium 6y ago
  Autonomous System. https://en.wikipedia.org/wiki/Autonomous_system_(Internet) https://en.wikipedia.org/wiki/Autonomous_system_(Internet) An AS is an entity significant enough to be reasoned about at the BGP level.
  - Stratoscope 6y ago
    Thanks!

jonah-archive 6y ago

Sibling comments are exactly correct -- we're a library. We don't track or record the activity of our patrons (see e.g. https://archive.org/services/docs/api/views.html#footnote-what-is-the-deal-with-privacy-protecting-hashes-instead-of-ip-numbers-in-our-logs https://archive.org/services/docs/api/views.html#footnote-wh... ), and since most CDNs cannot offer the same guarantee, we can't use them.