4 ms·

60 Gbps is about 1/3 of the traffic served by a single Netflix CDN node.

notRobot 6y ago
Source?
est31 6y ago
You are comparing to one of the largest bandwidth using services world-wide. Of course archive.org will lose out. I should have added: 60 Gbit/s is a lot for a donation run service like archive.org. Not sure if there are donation run services on the internet with more traffic.
toomuchtodo 6y ago
> single Netflix CDN node. Looks like they max out at 100 Gbps, or as low as 40 Gbps, depending on appliance model and link aggregation configuration. No argument either way, just thought it was cool info. https://openconnect.zendesk.com/hc/en-us/articles/360034538352 https://openconnect.zendesk.com/hc/en-us/articles/3600345383...
- cperciva 6y ago
  That's out of date. Here's a presentation from last year where they talk about >190 Gbps: https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf
  - deleted 6y ago
    [deleted]
  - erk__ 6y ago
    The video accompanying this slide deck is also a great watch: https://www.youtube.com/watch?v=8NSzkYSX5nY https://www.youtube.com/watch?v=8NSzkYSX5nY
PudgePacket 6y ago
I'd say the "bytes" being transferred are very different. Netflix has a (relatively) small number of large content pieces, and a CDN node would hold the hottest items. IA has 430 billion individual pieces of data, on average much smaller than a piece of Netflix content. So each "byte" transferred from IA is "more work" to produce, and less likely to come from some kind of cache.
jonah-archive 6y ago
It's harder when you don't make money, and everyone's not downloading the same things.
- bits_n_bytes 6y ago
  I understand money is tight for a non-profit, but why are bandwidth costs a gating factor? Both your upstreams, Cogent and Hurricane Electric, offer 100G ports at fiveish grand per month in carrier neutral DCs. Given that your budget is in the millions, an outlay of this magnitude doesn't seem wildly out of the question. If you can explain what the problems are in getting more bandwidth, I'd be more than happy to see what I can do to help. Please let me know if you'd prefer to discuss the matter over email.
  - jonah-archive 6y ago
    Happy to chat in much greater detail over email if you like (in my profile) -- transit specifically is not generally our limiting cost (though of course in the context we operate in, every penny counts -- e.g. we run our own DCs, ambiently cooled, single-feed grid power with minimal backing, etc). We generally run very close to the edge, capacity-wise, in order to get the most out of what we have -- in this case, demand moved very quickly and we had to work a hardware deployment to catch up.
    - bits_n_bytes 6y ago
      Email sent.
    - MaxBarraclough 6y ago
      I'm sure I'm not the only one who'd be interested in reading more of this. Is there a reason your conversation can't proceed on HackerNews?
      - Aeolun 6y ago
        Or at least update us on the conclusion :)
ptman 6y ago
does archive.org use cdn or is it all served from one location?

Source?

You are comparing to one of the largest bandwidth using services world-wide. Of course archive.org will lose out. I should have added: 60 Gbit/s is a lot for a donation run service like archive.org. Not sure if there are donation run services on the internet with more traffic.

toomuchtodo 6y ago

> single Netflix CDN node. Looks like they max out at 100 Gbps, or as low as 40 Gbps, depending on appliance model and link aggregation configuration. No argument either way, just thought it was cool info. https://openconnect.zendesk.com/hc/en-us/articles/360034538352 https://openconnect.zendesk.com/hc/en-us/articles/3600345383...

cperciva 6y ago
That's out of date. Here's a presentation from last year where they talk about >190 Gbps: https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf
- deleted 6y ago
  [deleted]
- erk__ 6y ago
  The video accompanying this slide deck is also a great watch: https://www.youtube.com/watch?v=8NSzkYSX5nY https://www.youtube.com/watch?v=8NSzkYSX5nY

PudgePacket 6y ago

I'd say the "bytes" being transferred are very different. Netflix has a (relatively) small number of large content pieces, and a CDN node would hold the hottest items. IA has 430 billion individual pieces of data, on average much smaller than a piece of Netflix content. So each "byte" transferred from IA is "more work" to produce, and less likely to come from some kind of cache.

jonah-archive 6y ago

It's harder when you don't make money, and everyone's not downloading the same things.

bits_n_bytes 6y ago
I understand money is tight for a non-profit, but why are bandwidth costs a gating factor? Both your upstreams, Cogent and Hurricane Electric, offer 100G ports at fiveish grand per month in carrier neutral DCs. Given that your budget is in the millions, an outlay of this magnitude doesn't seem wildly out of the question. If you can explain what the problems are in getting more bandwidth, I'd be more than happy to see what I can do to help. Please let me know if you'd prefer to discuss the matter over email.
- jonah-archive 6y ago
  Happy to chat in much greater detail over email if you like (in my profile) -- transit specifically is not generally our limiting cost (though of course in the context we operate in, every penny counts -- e.g. we run our own DCs, ambiently cooled, single-feed grid power with minimal backing, etc). We generally run very close to the edge, capacity-wise, in order to get the most out of what we have -- in this case, demand moved very quickly and we had to work a hardware deployment to catch up.
  - bits_n_bytes 6y ago
    Email sent.
  - MaxBarraclough 6y ago
    I'm sure I'm not the only one who'd be interested in reading more of this. Is there a reason your conversation can't proceed on HackerNews?
    - Aeolun 6y ago
      Or at least update us on the conclusion :)

ptman 6y ago

does archive.org use cdn or is it all served from one location?