6 ms·

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and n

This is a recent video presentation by Jonah Edwards, who runs the Core Infrastructure Team at the Internet Archive. He explains the IA’s server, storage, and networking infrastructure, and then takes questions from other people at the Archive. I found it all interesting. But the main takeaway for me is his response to Brewster Kahle’s question, beginning at 13:06, about why the IA does everything in-house rather than having its storage and processing hosted by, for example, AWS. His answer: lower cost, greater control, and greater confidence that their users are not being tracked.

benaadams 6y ago
I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...
- charcircuit 6y ago
  I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.
  - dilyevsky 6y ago
    For lots of orgs I’ve seen it creeps up slowly until you’re paying 10-50x of full transit without any peers but by that point you’re too locked in to do anything
  - remus 6y ago
    Depends on whether bandwidth is important to what you're doing. In many applications it isn't so even at the inflated prices charged by AWS et al don't really matter in the context of other expenses.
  - polote 6y ago
    Because people have nothing to compare their AWS cost to. They don't know how much it would cost them to host their service outside of AWS. And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills
    - johannes1234321 6y ago
      Not only that: It is really hard to predict AWS cost. So many variables go in. And starting with a small side project in AWS is easy, and then each additional step is a small step ...
    - blacksmith_tb 6y ago
      Absolutely - AWS is this generation's IBM, no one was ever fired for buying services from them, so to speak.
- walrus01 6y ago
  On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering. 60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.
  - tutfbhuf 6y ago
    It seems that companies can be too big for the cloud and too small for the cloud (don't need k8s). I wonder what's the sweet spot.
    - 867-5309 6y ago
      horizontal and vertical scaling are the latest push, but diagonal would be the sweet spot
  - owenmarshall 6y ago
    IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.
    - Daho0n 6y ago
      So the small pay for the large?
      - jeromegv 6y ago
        Yes, that's typically how it works for any industry. You get discount on bulk orders.
  - 6y ago
- tutfbhuf 6y ago
  A hybrid solution might be possible. Put your core infra on AWS, but get a very cheap CDN (or custom solution) in front of it to handle the 60Gbps so that only a small fraction will hit your AWS infra. Do the same for storage, e.g. build your own ceph cluster on bare-metal instead of Amazon S3.
- Sebb767 6y ago
  To be fair here, when you're pouring that much money into AWS you probably have a better contract and can negotiate the price down quite a bit. Additionally, you could use CloudFront to further reduce your bandwidth costs. That's not to say that it wouldn't be incredibly expensive, but probably far less than what you see on the pricing page.
janjim 6y ago
Curious about the cost. Does that already include manpower and various acquisition cost of constructing their internal network (hardware, fiber link between site)? I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.
- Nextgrid 6y ago
  I've seen this argument a lot but I'm not sure how well it holds. The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available within minutes.
  - Sebb767 6y ago
    > The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. It really depends on what scale you're talking about. When you're a startup and suddenly land on the front page of HN, you might need 100x or 1000x your current capacity - in which case AWS will be useful to no end. If, on the other hand, you're an established name with quite a bit of traffic already and the maximum uptick you will reasonably experience is 2x-3x, the argument holds far less water.
    - ddorian43 6y ago
      Feel free to point to these cases where people scaled 1000x when they hit the HN frontpage. Especially their database.
- ddorian43 6y ago
  He said his storage pricing is 2-5x cheaper than google archive line. That is $1.2/3x= $0.4/TB/month. Compare that against $20/month S3. He has 50x less cost. He can afford to overprovision.
ignoramous 6y ago
Speaking of Infrastructure, it is amazing that the initial set of the Apache BigData projects started at Internet Archive [0] whilst Alexa Internet, a startup Brewster Kahle sold to Amazon in 1999, formed the basis of Alexa Web Information Service, one of the first ''AWS'' products [1] which is still up? https://aws.amazon.com/awis/ https://aws.amazon.com/awis/ [0] http://radar.oreilly.com/2015/04/coming-full-circle-with-bigtable-and-hbase.html http://radar.oreilly.com/2015/04/coming-full-circle-with-big... [1] https://web.stanford.edu/class/ee204/Publications/Amazon-EE353-2008-1.pdf https://web.stanford.edu/class/ee204/Publications/Amazon-EE3...
7800 6y ago
I have ADD and typically eschew watching video if I can get the same, quality content faster in text. I loved this and watched it to the end. To any that feel the need to hide or disparage this because it seems to promote doing things on your own vs in the cloud, this isn’t some tech ops Total Money Makeover where you read a book and you’re suddenly in some sort of anti-credit cult. This is hard shit, and it’s the basics of the hard shit that I grew up with as being the only shit. Yes, you can serve your own data. No one should fault you for doing that if you want. It takes the humble intelligence of the core team and everyone at IA to pull that off at this scale. If you don’t want to do the hard things, you could use the cloud. There are financial reasons also for one or the other, just as there are reasons people live with their family, rent, lease, and buy homes and office space- an imperfect analogy of course. I hope that some of the others that could go on to work at the big guys or have been working there and want a challenge consider applying to IA when there’s an opening. They’ve done an incredible job, and I look forward to the cool things they accomplish in the future.
- jonah-archive 6y ago
  Thank you for this! It was definitely geared towards an internal audience but it makes me very happy to know that it was enjoyed and appreciated more broadly. I am going to get a transcript done and up soon as well -- I just gave the talk on Friday so haven't had time to do so yet.
nodesocket 6y ago
What happens when there is a major earthquake in the bay area though? Not being geographically dispersed seems like a large risk.
- nikisweeting 6y ago
  They have backups in Canada and Egypt.
  - nikisweeting 6y ago
    partial* backups^
Ansil849 6y ago
> and greater confidence that their users are not being tracked. And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.
- samstave 6y ago
  Tracking should be STRICTLY illegal and ONLY acceptable with a verifiable OPT-IN and a transparency on to WHOM the data was sent/read-by/received. With STILL an option to selectively opt-out.
  - Shared404 6y ago
    I agree with the sentiment, but there needs to be some degree of leeway. For example are server logs considered tracking? It seems unreasonable to require that logs not be kept. Edit: On further thought, I don't even know that tracking should be banned. Instead, I would argue that advertising in the way enabled by tracking should be banned. That way the incentive is removed with less bureaucracy.
    - samstave 6y ago
      bureaucracy IS the incentive. We need to kill bureaucratic interests in EVERY THING. Politicians should be conscripted servants with no method for empowering or financing themselves. They should run on policy ALONE.
      - Shared404 6y ago
        > bureaucracy IS the incentive I don't agree there. There is no reason that someone has building bureaucracy as they're goal. The goal is to get something out of it, which is accomplished via the bureaucracy. > We need to kill bureaucratic interests in EVERY THING. While I don't necessarily disagree, this is orthogonal to the original issue. > Politicians should be conscripted servants with no method for empowering or financing themselves. I don't think conscripted means what you think it means... Also, that would be impossible to implement. Either you don't try to cut off every path, in which case you have the option of limiting bureaucracy, or you try to cut off every path, increasing bureaucracy. > They should run on policy ALONE. I agree. Also orthogonal to the topic at hand. Also, please let me know if you find a way of implementing this without requiring bureaucracy as a critical component.

benaadams 6y ago

I assume the 200PB of storage and 60Gbps egress bandwidth 24/365 they do would be _extremely_ pricy on AWS...

charcircuit 6y ago
I have no clue how people can afford what AWS charges for bandwidth. I did the math once for migrating a project to AWS and the bandwidth alone costed 10x my entire current infrastructure for that project, which is something I run for free.
- dilyevsky 6y ago
  For lots of orgs I’ve seen it creeps up slowly until you’re paying 10-50x of full transit without any peers but by that point you’re too locked in to do anything
- remus 6y ago
  Depends on whether bandwidth is important to what you're doing. In many applications it isn't so even at the inflated prices charged by AWS et al don't really matter in the context of other expenses.
- polote 6y ago
  Because people have nothing to compare their AWS cost to. They don't know how much it would cost them to host their service outside of AWS. And it is not only a cost comparison. You need different kind of people to manage in-house vs cloud, not that you need less or more, just different skills
  - johannes1234321 6y ago
    Not only that: It is really hard to predict AWS cost. So many variables go in. And starting with a small side project in AWS is easy, and then each additional step is a small step ...
  - blacksmith_tb 6y ago
    Absolutely - AWS is this generation's IBM, no one was ever fired for buying services from them, so to speak.
walrus01 6y ago
On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible. Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering. 60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.
- tutfbhuf 6y ago
  It seems that companies can be too big for the cloud and too small for the cloud (don't need k8s). I wonder what's the sweet spot.
  - 867-5309 6y ago
    horizontal and vertical scaling are the latest push, but diagonal would be the sweet spot
- owenmarshall 6y ago
  IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.
  - Daho0n 6y ago
    So the small pay for the large?
    - jeromegv 6y ago
      Yes, that's typically how it works for any industry. You get discount on bulk orders.
- 6y ago
tutfbhuf 6y ago
A hybrid solution might be possible. Put your core infra on AWS, but get a very cheap CDN (or custom solution) in front of it to handle the 60Gbps so that only a small fraction will hit your AWS infra. Do the same for storage, e.g. build your own ceph cluster on bare-metal instead of Amazon S3.
Sebb767 6y ago
To be fair here, when you're pouring that much money into AWS you probably have a better contract and can negotiate the price down quite a bit. Additionally, you could use CloudFront to further reduce your bandwidth costs. That's not to say that it wouldn't be incredibly expensive, but probably far less than what you see on the pricing page.

janjim 6y ago

Curious about the cost. Does that already include manpower and various acquisition cost of constructing their internal network (hardware, fiber link between site)? I guess the biggest downside is the speed of scaling that they can do. As it is limited by how fast they can purchase and install new storage device. But with the use case of Internet Archive, that shouldn't matter much.

Nextgrid 6y ago
I've seen this argument a lot but I'm not sure how well it holds. The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. Also, bare-metal doesn't necessarily mean owning the hardware. You can rent it too. There are providers that provide bare-metal in one-click and sometimes available within minutes.
- Sebb767 6y ago
  > The price for performance ratio on cloud providers is so poor that you can overprovision in advance (to mitigate the extra delay involved in adding extra hardware) and still come out ahead. It really depends on what scale you're talking about. When you're a startup and suddenly land on the front page of HN, you might need 100x or 1000x your current capacity - in which case AWS will be useful to no end. If, on the other hand, you're an established name with quite a bit of traffic already and the maximum uptick you will reasonably experience is 2x-3x, the argument holds far less water.
  - ddorian43 6y ago
    Feel free to point to these cases where people scaled 1000x when they hit the HN frontpage. Especially their database.
ddorian43 6y ago
He said his storage pricing is 2-5x cheaper than google archive line. That is $1.2/3x= $0.4/TB/month. Compare that against $20/month S3. He has 50x less cost. He can afford to overprovision.

ignoramous 6y ago

Speaking of Infrastructure, it is amazing that the initial set of the Apache BigData projects started at Internet Archive [0] whilst Alexa Internet, a startup Brewster Kahle sold to Amazon in 1999, formed the basis of Alexa Web Information Service, one of the first ''AWS'' products [1] which is still up? https://aws.amazon.com/awis/ https://aws.amazon.com/awis/ [0] http://radar.oreilly.com/2015/04/coming-full-circle-with-bigtable-and-hbase.html http://radar.oreilly.com/2015/04/coming-full-circle-with-big... [1] https://web.stanford.edu/class/ee204/Publications/Amazon-EE353-2008-1.pdf https://web.stanford.edu/class/ee204/Publications/Amazon-EE3...

7800 6y ago

I have ADD and typically eschew watching video if I can get the same, quality content faster in text. I loved this and watched it to the end. To any that feel the need to hide or disparage this because it seems to promote doing things on your own vs in the cloud, this isn’t some tech ops Total Money Makeover where you read a book and you’re suddenly in some sort of anti-credit cult. This is hard shit, and it’s the basics of the hard shit that I grew up with as being the only shit. Yes, you can serve your own data. No one should fault you for doing that if you want. It takes the humble intelligence of the core team and everyone at IA to pull that off at this scale. If you don’t want to do the hard things, you could use the cloud. There are financial reasons also for one or the other, just as there are reasons people live with their family, rent, lease, and buy homes and office space- an imperfect analogy of course. I hope that some of the others that could go on to work at the big guys or have been working there and want a challenge consider applying to IA when there’s an opening. They’ve done an incredible job, and I look forward to the cool things they accomplish in the future.

jonah-archive 6y ago
Thank you for this! It was definitely geared towards an internal audience but it makes me very happy to know that it was enjoyed and appreciated more broadly. I am going to get a transcript done and up soon as well -- I just gave the talk on Friday so haven't had time to do so yet.

nodesocket 6y ago

What happens when there is a major earthquake in the bay area though? Not being geographically dispersed seems like a large risk.

nikisweeting 6y ago
They have backups in Canada and Egypt.
- nikisweeting 6y ago
  partial* backups^

Ansil849 6y ago

> and greater confidence that their users are not being tracked. And that right there is why I continue to donate to IA. I am sick and tired of services offloading my data to destructive companies like Amazon.

samstave 6y ago
Tracking should be STRICTLY illegal and ONLY acceptable with a verifiable OPT-IN and a transparency on to WHOM the data was sent/read-by/received. With STILL an option to selectively opt-out.
- Shared404 6y ago
  I agree with the sentiment, but there needs to be some degree of leeway. For example are server logs considered tracking? It seems unreasonable to require that logs not be kept. Edit: On further thought, I don't even know that tracking should be banned. Instead, I would argue that advertising in the way enabled by tracking should be banned. That way the incentive is removed with less bureaucracy.
  - samstave 6y ago
    bureaucracy IS the incentive. We need to kill bureaucratic interests in EVERY THING. Politicians should be conscripted servants with no method for empowering or financing themselves. They should run on policy ALONE.
    - Shared404 6y ago
      > bureaucracy IS the incentive I don't agree there. There is no reason that someone has building bureaucracy as they're goal. The goal is to get something out of it, which is accomplished via the bureaucracy. > We need to kill bureaucratic interests in EVERY THING. While I don't necessarily disagree, this is orthogonal to the original issue. > Politicians should be conscripted servants with no method for empowering or financing themselves. I don't think conscripted means what you think it means... Also, that would be impossible to implement. Either you don't try to cut off every path, in which case you have the option of limiting bureaucracy, or you try to cut off every path, increasing bureaucracy. > They should run on policy ALONE. I agree. Also orthogonal to the topic at hand. Also, please let me know if you find a way of implementing this without requiring bureaucracy as a critical component.