3 ms·

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Har

by warent 4y ago

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Hardware these days is really good. People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie. It would still be a huge cost saving and trivial for them to contract someone to maintain the servers for like 1 hour a month with emergency on-call. Of which there are numerous such people who are not me.

senttoschool 4y ago
I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy. I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU for the same price! It's a no brainer! Then I realized that if I wanted to build custom PCs for all the workers, I'd have to provide support for all of them. I'd have to be as good as CDW, who was supplying the company with PCs. This means next-day repairs and replacements. Drivers. Security. Firmware updates. For hundreds of PCs. Needless to say, a lot of people under appreciate all the things that vendors like AWS or CDW provide. They think they can do it because they know how it's done. But they're ignoring a lot of other factors that can come back to bite you. I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.
- sshine 4y ago
  > I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer. It’s not either AWS or “bring your soldering iron”. Self-hosting K8s on managed hardware is still insanely cheaper than AWS. GitOps, infrastructure as code, DevOps platforms, CNCF best practice setups are entirely doable without paying the Amazon tax. If your lead DevOps quits, you’re right, it’s easier to find someone who does AWS. The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs. It’s not much different than outsourcing DevOps, which I think is what some hackers frown upon, because that’s their skillset you buy in town at a premium.
  - nostrebored 4y ago
    Running K8S is itself a giant tax. The complexity is absurd and the components are not as reliable as having a literal team of people who are responsible for ops and making a service feature complete. > The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs. This means nothing to me -- what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? How flexible are the different systems for the future? What are the reliability/uptime requirements and the marginal cost of outage? How replaceable are the dev ops engineers who are running your K8s/managed infra? People moving to cloud isn't irrational. There's a reason that it's pervasive.
    - sshine 4y ago
      > what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? I can only speak of one PHP/TypeScript app with a poor performance profile: from $250/mo to $5.000/mo in running costs, not counting the migration. The team that made the app couldn’t maintain the Helm chart or the Terraform resources, adding to a DevOps team bottleneck that appears to limit how teams innovate (kind of how like creating new branches was expensive on CVS). > How replaceable are the dev ops engineers who are running your K8s/managed infra? I haven’t met a competent AWS DevOps engineer who wasn’t also competent to run the clusters; it’s just that they prefer not to. > People moving to cloud isn't irrational. There's a reason that it's pervasive. Yes, it’s made really solid. AWS makes sense to reduce operational risk at a large scale. The price is not my main argument against using AWS, although you’ve gotta make good money to justify the expense. My motivation is to own one’s own infrastructure. I like to work on things that are both important and contented enough that you can’t, as a principle, outsource the operation.
- Symbiote 4y ago
  You can obviously buy servers from HP or Dell. I have about 80 in a colo datacentre. Power, air-conditioning, security etc isn't my problem. That's what we pay the colo for. I go there about once a year to install new servers, and about once at some other point in the year to replace a failed disk. I've taken various developers who are interested each time, so if I'm out of town there are several people who can cover, plus the colo staff. You can also pay for people to do this routine work. And then there's still the option of renting managed servers, so that company handles all the hardware.
  - topicseed 4y ago
    To be fair, this sounds a lot of to do items that I don't want to do.
    - Symbiote 4y ago
      I think it's nice to have a completely different type of work a couple of days per year. I've found there's generally developers who are into building custom gaming PCs or Raspberry Pis or whatever who are interested to see the servers and lend a hand installing new ones.
      - topicseed 4y ago
        I actually am into hacking at home but I'd definitely not play around with my business' production environment. Wouldn't know where to start if sh* hits the fan.
        senttoschool 4y ago
        Not sure why you're being downvoted. But I love building PCs too. There's no way I'd want to put together my company's server hardware though even though the basics are the same. I do not want to be on call 24/7.
        OccamsMirror 4y ago
- amluto 4y ago
  In the last few years, build-your-own storage got annoying. Most server disks were SATA, mostly the same form factor and SATA ports are available in abundance. You could get various more complex SFF-xyzw connectors, but the adapters were straightforwardly available. Now there’s NVMe and there are quite a few form factors. And NVMe is generally a point-to-point link to the CPU, so you either need a motherboard with the right number and types of connectors or you need a PCIe/NVMe splitter or switch. And the latter don’t seem widely available on the non-sketchy-hardware market. So the situation is fine for system integrators who make their own PCBs, but not amazing for DIY builders.
manv1 4y ago
Well, it depends on the colo. I mean, it's not like a colo doesn't have issues. At one point our colo was running some cable and we discovered that their cables to our rack were wiggly...after our stuff went offline intermittently. That was hard to track down. And you need to set up the LOMs, the management network (hey, you don't want that on the public internet), VPN, serial cables (backup to network LOM), maybe a load balancer. And you have to hope that when you reboot the box it doesn't fail POST. And you have to remember to turn off power saving in the BIOS (how many people forget to do this?). And you have to set up the RAID 5/10. And you have to do this sitting on the floor, because not a lot of places provide chairs in the racks. Oh, do you have a rack-mounted KVM/monitor/serial console or are you using your laptop with a serial cable to do all this? I mean, it's not hard, it's just a lot of detail work. And I suspect that these skills are going away.
- warent 4y ago
  These are great points, thank you for the insight. A small note I will add is that these days there are remote BIOS options available for servers like iDRAC
  - manv1 4y ago
    Do they need a reboot to take effect? I haven't touched a LOM in more than 8 years!
    - hotpotamus 4y ago
      DRACs are a Dell specific term for what is generically known as IPMI or out of band management. They were quite well developed when I worked on servers well over a decade ago now. The good ones are basically as independent of the system they're attached to as possible and essentially give you the equivalent of physical access to a server.
      - manv1 4y ago
        I know what DRACs are (it's Dell's LOM). I asked whether the DRAC bios stuff required a reboot.

by warent 4y ago

senttoschool 4y ago

I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy. I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU for the same price! It's a no brainer! Then I realized that if I wanted to build custom PCs for all the workers, I'd have to provide support for all of them. I'd have to be as good as CDW, who was supplying the company with PCs. This means next-day repairs and replacements. Drivers. Security. Firmware updates. For hundreds of PCs. Needless to say, a lot of people under appreciate all the things that vendors like AWS or CDW provide. They think they can do it because they know how it's done. But they're ignoring a lot of other factors that can come back to bite you. I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.

sshine 4y ago
> I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer. It’s not either AWS or “bring your soldering iron”. Self-hosting K8s on managed hardware is still insanely cheaper than AWS. GitOps, infrastructure as code, DevOps platforms, CNCF best practice setups are entirely doable without paying the Amazon tax. If your lead DevOps quits, you’re right, it’s easier to find someone who does AWS. The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs. It’s not much different than outsourcing DevOps, which I think is what some hackers frown upon, because that’s their skillset you buy in town at a premium.
- nostrebored 4y ago
  Running K8S is itself a giant tax. The complexity is absurd and the components are not as reliable as having a literal team of people who are responsible for ops and making a service feature complete. > The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs. This means nothing to me -- what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? How flexible are the different systems for the future? What are the reliability/uptime requirements and the marginal cost of outage? How replaceable are the dev ops engineers who are running your K8s/managed infra? People moving to cloud isn't irrational. There's a reason that it's pervasive.
  - sshine 4y ago
    > what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? I can only speak of one PHP/TypeScript app with a poor performance profile: from $250/mo to $5.000/mo in running costs, not counting the migration. The team that made the app couldn’t maintain the Helm chart or the Terraform resources, adding to a DevOps team bottleneck that appears to limit how teams innovate (kind of how like creating new branches was expensive on CVS). > How replaceable are the dev ops engineers who are running your K8s/managed infra? I haven’t met a competent AWS DevOps engineer who wasn’t also competent to run the clusters; it’s just that they prefer not to. > People moving to cloud isn't irrational. There's a reason that it's pervasive. Yes, it’s made really solid. AWS makes sense to reduce operational risk at a large scale. The price is not my main argument against using AWS, although you’ve gotta make good money to justify the expense. My motivation is to own one’s own infrastructure. I like to work on things that are both important and contented enough that you can’t, as a principle, outsource the operation.
Symbiote 4y ago
You can obviously buy servers from HP or Dell. I have about 80 in a colo datacentre. Power, air-conditioning, security etc isn't my problem. That's what we pay the colo for. I go there about once a year to install new servers, and about once at some other point in the year to replace a failed disk. I've taken various developers who are interested each time, so if I'm out of town there are several people who can cover, plus the colo staff. You can also pay for people to do this routine work. And then there's still the option of renting managed servers, so that company handles all the hardware.
- topicseed 4y ago
  To be fair, this sounds a lot of to do items that I don't want to do.
  - Symbiote 4y ago
    I think it's nice to have a completely different type of work a couple of days per year. I've found there's generally developers who are into building custom gaming PCs or Raspberry Pis or whatever who are interested to see the servers and lend a hand installing new ones.
    - topicseed 4y ago
      I actually am into hacking at home but I'd definitely not play around with my business' production environment. Wouldn't know where to start if sh* hits the fan.
      - senttoschool 4y ago
        Not sure why you're being downvoted. But I love building PCs too. There's no way I'd want to put together my company's server hardware though even though the basics are the same. I do not want to be on call 24/7.
        OccamsMirror 4y ago
amluto 4y ago
In the last few years, build-your-own storage got annoying. Most server disks were SATA, mostly the same form factor and SATA ports are available in abundance. You could get various more complex SFF-xyzw connectors, but the adapters were straightforwardly available. Now there’s NVMe and there are quite a few form factors. And NVMe is generally a point-to-point link to the CPU, so you either need a motherboard with the right number and types of connectors or you need a PCIe/NVMe splitter or switch. And the latter don’t seem widely available on the non-sketchy-hardware market. So the situation is fine for system integrators who make their own PCBs, but not amazing for DIY builders.

manv1 4y ago

Well, it depends on the colo. I mean, it's not like a colo doesn't have issues. At one point our colo was running some cable and we discovered that their cables to our rack were wiggly...after our stuff went offline intermittently. That was hard to track down. And you need to set up the LOMs, the management network (hey, you don't want that on the public internet), VPN, serial cables (backup to network LOM), maybe a load balancer. And you have to hope that when you reboot the box it doesn't fail POST. And you have to remember to turn off power saving in the BIOS (how many people forget to do this?). And you have to set up the RAID 5/10. And you have to do this sitting on the floor, because not a lot of places provide chairs in the racks. Oh, do you have a rack-mounted KVM/monitor/serial console or are you using your laptop with a serial cable to do all this? I mean, it's not hard, it's just a lot of detail work. And I suspect that these skills are going away.

warent 4y ago
These are great points, thank you for the insight. A small note I will add is that these days there are remote BIOS options available for servers like iDRAC
- manv1 4y ago
  Do they need a reboot to take effect? I haven't touched a LOM in more than 8 years!
  - hotpotamus 4y ago
    DRACs are a Dell specific term for what is generically known as IPMI or out of band management. They were quite well developed when I worked on servers well over a decade ago now. The good ones are basically as independent of the system they're attached to as possible and essentially give you the equivalent of physical access to a server.
    - manv1 4y ago
      I know what DRACs are (it's Dell's LOM). I asked whether the DRAC bios stuff required a reboot.