6 ms·

Not including extra human cost in the analysis is just disingenuous. I think to manage private servers of that size you would need at least two extra experts to

Not including extra human cost in the analysis is just disingenuous. I think to manage private servers of that size you would need at least two extra experts totalling at least $20k/month.

ralgozino 5y ago
How many DevOps do you need to manage your cloud? How much do they cost?
- Dma54rhs 5y ago
  You need high skilled people who are comfortable in AWS or other cloud offering as well. Have to care of tons of things, set them up etc. They are not one button setups in real life when things get complicated.
capableweb 5y ago
What?! Maybe if you hire SV-skilled engineers on location in Sillicon Valley, but you can easily serve 2GB/second (on infra worth $2K/month) with one sys-admin dealing with it, and for way less than a whopping $10k/month.
- PragmaticPulp 5y ago
  > with one sys-admin dealing with it, And if that one sys-admin wants to go on vacation? Or travel away from a computer? Or takes another job? You can never have “just one” admin handling a server and being on-call 24/7. Would you really want a job where you could never, ever be away from a computer because you’re the only on-call person? This doesn’t work.
  - new_guy 5y ago
    > Would you really want a job where you could never, ever be away from a computer because you’re the only on-call person? This doesn’t work. Works fine for me. $20K a month for two people doing f*k all is insane.
  - Melkman 5y ago
    Why would it be necessary to have an engineer on call 24/7 ? If you do your risk calculations and an outage of 12 hours is acceptable in the expected frequency you just let the engineer have a nice evening and night an deal with the outage in the morning. If outages are only to be expected once a year an you can tolerate 48 hours of outage you don't need any on call engineer. Most outages are cause by changes. You can test those and plan putting these in production and have elevated monitoring after this to catch problems early. Only problem remaining is hardware outages. And those are very rare as long as you do decent lifecycle maintenance. As others said before. Not everyone is Google or Amazon and needs 99,999% uptime.
kasey_junk 5y ago
They didn’t mention a fleet size but 2GB/s is a single commodity server. You don’t even need a single employee to manage a single server…
- YetAnotherNick 5y ago
  Which "commodity server" could serve 2GB/s(16gbps) to public internet?
  - jleahy 5y ago
    Pretty much anything. How about a Dell R340 with a dual-10G NIC and some SSDs? That's not commodity, that's cheap, a commodity server would be a dual-Xeon but that's overkill for serving 16gbps.
    - ksec 5y ago
      I think that is the problem with modern day's Web Dev. ( Sorry ) With Cloud and SaaS. They are so abstracted from Hardware that their knowledge on basic hardware and server, everything from CPU, Storage I/O and Network are close to zero.
  - virtuallynathan 5y ago
    At Netflix we’re doing close to 400Gbps on 1U commodity hardware, and pretty inexpensive.
    - ksec 5y ago
      And that is 400Gbps encrypted! Side Note, aren't they 2U?
      - virtuallynathan 5y ago
        Yep! All TLS. The production boxes are 2U, but it can be done in a 1U box.
cblconfederate 5y ago
> extra human cost Where? Costs vary hugely across the world
rndgermandude 5y ago
> I think to manage private servers of that size you would need at least two extra experts totalling at least $20k/month. What? You set up the deployment once, and then you only need to touch it when things go horribly wrong, which is every couple of months, or to make minor quick tweaks and run some updates. Let's be generous, and say you need 10 h/month, which is about 1/16 of a person-month. And if things go horribly wrong, everybody drops what they are doing to fix things, anyway, no matter if you're on AWS, dedicated/colo or run your own data center. When you significantly change your architecture/deployment, then you need to put in more time again, but if you build your code with need to scale and such things in mind from the get-go, then that won't come up much or at all.
- PragmaticPulp 5y ago
  > What? You set up the deployment once, and then you only need to touch it when things go horribly wrong, which is every couple of months Right, which is exactly why people pay extra for cloud managed services. If things are going “horribly wrong” every couple of months then you must necessarily be on call 24/7 and never take vacation or time away. In practice, you need at least two people to manage on-call coverage so you’re not completely uncovered if someone gets sick, decides to take vacation, wants to travel away from a computer and so on.
  - southerntofu 5y ago
    We're talking volunteer-run projects, though. Who cares if it's not available 24/7? Best-effort is good enough. Those managed cloud services also fail often, you just have no information and no recourse about it.
  - rndgermandude 5y ago
    Things go horribly wrong with AWS hosted stuff as well. And a lot of companies have a single-point-of-failure AWS person. While you're not wrong in general, nothing you just wrote is specific to running on dedicated servers vs AWS.
  - lupire 5y ago
bezospen15 5y ago
Are you hiring? What infrastructure engineer are you paying $10,000/month for just to manage 4 servers? LOL
svelle 5y ago
But that AWS deployment doesn't manifest out of thin air either. Kubernetes and AWS knowledgable engineers aren't any less expensive.
BiteCode_dev 5y ago
No, the system is maintained by one single part time dev. That's the entire dev team.
ziggus 5y ago
What do you imagine these experts would do? Rebuild the servers from raw materials each month, and hand-code all software in ASM weekly? In the real world, once most hosting platforms are up and running, the maintenance overhead is pretty low.

Not including extra human cost in the analysis is just disingenuous. I think to manage private servers of that size you would need at least two extra experts totalling at least $20k/month.

ralgozino 5y ago

How many DevOps do you need to manage your cloud? How much do they cost?

Dma54rhs 5y ago
You need high skilled people who are comfortable in AWS or other cloud offering as well. Have to care of tons of things, set them up etc. They are not one button setups in real life when things get complicated.

capableweb 5y ago

What?! Maybe if you hire SV-skilled engineers on location in Sillicon Valley, but you can easily serve 2GB/second (on infra worth $2K/month) with one sys-admin dealing with it, and for way less than a whopping $10k/month.

PragmaticPulp 5y ago
> with one sys-admin dealing with it, And if that one sys-admin wants to go on vacation? Or travel away from a computer? Or takes another job? You can never have “just one” admin handling a server and being on-call 24/7. Would you really want a job where you could never, ever be away from a computer because you’re the only on-call person? This doesn’t work.
- new_guy 5y ago
  > Would you really want a job where you could never, ever be away from a computer because you’re the only on-call person? This doesn’t work. Works fine for me. $20K a month for two people doing f*k all is insane.
- Melkman 5y ago
  Why would it be necessary to have an engineer on call 24/7 ? If you do your risk calculations and an outage of 12 hours is acceptable in the expected frequency you just let the engineer have a nice evening and night an deal with the outage in the morning. If outages are only to be expected once a year an you can tolerate 48 hours of outage you don't need any on call engineer. Most outages are cause by changes. You can test those and plan putting these in production and have elevated monitoring after this to catch problems early. Only problem remaining is hardware outages. And those are very rare as long as you do decent lifecycle maintenance. As others said before. Not everyone is Google or Amazon and needs 99,999% uptime.

kasey_junk 5y ago

They didn’t mention a fleet size but 2GB/s is a single commodity server. You don’t even need a single employee to manage a single server…

YetAnotherNick 5y ago
Which "commodity server" could serve 2GB/s(16gbps) to public internet?
- jleahy 5y ago
  Pretty much anything. How about a Dell R340 with a dual-10G NIC and some SSDs? That's not commodity, that's cheap, a commodity server would be a dual-Xeon but that's overkill for serving 16gbps.
  - ksec 5y ago
    I think that is the problem with modern day's Web Dev. ( Sorry ) With Cloud and SaaS. They are so abstracted from Hardware that their knowledge on basic hardware and server, everything from CPU, Storage I/O and Network are close to zero.
- virtuallynathan 5y ago
  At Netflix we’re doing close to 400Gbps on 1U commodity hardware, and pretty inexpensive.
  - ksec 5y ago
    And that is 400Gbps encrypted! Side Note, aren't they 2U?
    - virtuallynathan 5y ago
      Yep! All TLS. The production boxes are 2U, but it can be done in a 1U box.

cblconfederate 5y ago

> extra human cost Where? Costs vary hugely across the world

rndgermandude 5y ago

> I think to manage private servers of that size you would need at least two extra experts totalling at least $20k/month. What? You set up the deployment once, and then you only need to touch it when things go horribly wrong, which is every couple of months, or to make minor quick tweaks and run some updates. Let's be generous, and say you need 10 h/month, which is about 1/16 of a person-month. And if things go horribly wrong, everybody drops what they are doing to fix things, anyway, no matter if you're on AWS, dedicated/colo or run your own data center. When you significantly change your architecture/deployment, then you need to put in more time again, but if you build your code with need to scale and such things in mind from the get-go, then that won't come up much or at all.

PragmaticPulp 5y ago
> What? You set up the deployment once, and then you only need to touch it when things go horribly wrong, which is every couple of months Right, which is exactly why people pay extra for cloud managed services. If things are going “horribly wrong” every couple of months then you must necessarily be on call 24/7 and never take vacation or time away. In practice, you need at least two people to manage on-call coverage so you’re not completely uncovered if someone gets sick, decides to take vacation, wants to travel away from a computer and so on.
- southerntofu 5y ago
  We're talking volunteer-run projects, though. Who cares if it's not available 24/7? Best-effort is good enough. Those managed cloud services also fail often, you just have no information and no recourse about it.
- rndgermandude 5y ago
  Things go horribly wrong with AWS hosted stuff as well. And a lot of companies have a single-point-of-failure AWS person. While you're not wrong in general, nothing you just wrote is specific to running on dedicated servers vs AWS.
- lupire 5y ago

bezospen15 5y ago

Are you hiring? What infrastructure engineer are you paying $10,000/month for just to manage 4 servers? LOL

svelle 5y ago

But that AWS deployment doesn't manifest out of thin air either. Kubernetes and AWS knowledgable engineers aren't any less expensive.

BiteCode_dev 5y ago

No, the system is maintained by one single part time dev. That's the entire dev team.

ziggus 5y ago

What do you imagine these experts would do? Rebuild the servers from raw materials each month, and hand-code all software in ASM weekly? In the real world, once most hosting platforms are up and running, the maintenance overhead is pretty low.