6 ms·

When consulting for a seed funded startup, I suggested they buy their own servers and colocate them. It could save them almost 100k a year. Everyone looked at

• senttoschool 4y ago
  They'd have to keep you employed to manage the hardware. What happens if you get run over by a bus (which I never hope will happen)? What happens if you decide to move to another city? There are a lot more "what ifs" by employing you to build and manage the hardware. Saving $100k would not be worth the hassle/risk for most well funded startups.
  • warent 4y ago
    Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years. Hardware these days is really good. People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie. It would still be a huge cost saving and trivial for them to contract someone to maintain the servers for like 1 hour a month with emergency on-call. Of which there are numerous such people who are not me.
    • senttoschool 4y ago
      I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy. I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU for the same price! It's a no brainer! Then I realized that if I wanted to build custom PCs for all the workers, I'd have to provide support for all of them. I'd have to be as good as CDW, who was supplying the company with PCs. This means next-day repairs and replacements. Drivers. Security. Firmware updates. For hundreds of PCs. Needless to say, a lot of people under appreciate all the things that vendors like AWS or CDW provide. They think they can do it because they know how it's done. But they're ignoring a lot of other factors that can come back to bite you. I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.
      • sshine 4y ago
        > I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer. It’s not either AWS or “bring your soldering iron”. Self-hosting K8s on managed hardware is still insanely cheaper than AWS. GitOps, infrastructure as code, DevOps platforms, CNCF best practice setups are entirely doable without paying the Amazon tax. If your lead DevOps quits, you’re right, it’s easier to find someone who does AWS. The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs. It’s not much different than outsourcing DevOps, which I think is what some hackers frown upon, because that’s their skillset you buy in town at a premium.
        • nostrebored 4y ago
          Running K8S is itself a giant tax. The complexity is absurd and the components are not as reliable as having a literal team of people who are responsible for ops and making a service feature complete. > The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs. This means nothing to me -- what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? How flexible are the different systems for the future? What are the reliability/uptime requirements and the marginal cost of outage? How replaceable are the dev ops engineers who are running your K8s/managed infra? People moving to cloud isn't irrational. There's a reason that it's pervasive.
          • sshine 4y ago
            > what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? I can only speak of one PHP/TypeScript app with a poor performance profile: from $250/mo to $5.000/mo in running costs, not counting the migration. The team that made the app couldn’t maintain the Helm chart or the Terraform resources, adding to a DevOps team bottleneck that appears to limit how teams innovate (kind of how like creating new branches was expensive on CVS). > How replaceable are the dev ops engineers who are running your K8s/managed infra? I haven’t met a competent AWS DevOps engineer who wasn’t also competent to run the clusters; it’s just that they prefer not to. > People moving to cloud isn't irrational. There's a reason that it's pervasive. Yes, it’s made really solid. AWS makes sense to reduce operational risk at a large scale. The price is not my main argument against using AWS, although you’ve gotta make good money to justify the expense. My motivation is to own one’s own infrastructure. I like to work on things that are both important and contented enough that you can’t, as a principle, outsource the operation.
    • manv1 4y ago
      Well, it depends on the colo. I mean, it's not like a colo doesn't have issues. At one point our colo was running some cable and we discovered that their cables to our rack were wiggly...after our stuff went offline intermittently. That was hard to track down. And you need to set up the LOMs, the management network (hey, you don't want that on the public internet), VPN, serial cables (backup to network LOM), maybe a load balancer. And you have to hope that when you reboot the box it doesn't fail POST. And you have to remember to turn off power saving in the BIOS (how many people forget to do this?). And you have to set up the RAID 5/10. And you have to do this sitting on the floor, because not a lot of places provide chairs in the racks. Oh, do you have a rack-mounted KVM/monitor/serial console or are you using your laptop with a serial cable to do all this? I mean, it's not hard, it's just a lot of detail work. And I suspect that these skills are going away.
      • warent 4y ago
        These are great points, thank you for the insight. A small note I will add is that these days there are remote BIOS options available for servers like iDRAC
        • manv1 4y ago
          Do they need a reboot to take effect? I haven't touched a LOM in more than 8 years!
          • hotpotamus 4y ago
            DRACs are a Dell specific term for what is generically known as IPMI or out of band management. They were quite well developed when I worked on servers well over a decade ago now. The good ones are basically as independent of the system they're attached to as possible and essentially give you the equivalent of physical access to a server.
            • manv1 4y ago
              I know what DRACs are (it's Dell's LOM). I asked whether the DRAC bios stuff required a reboot.
• Cpoll 4y ago
  I agree that there's a point where co-locating is a good thing to think about. But I don't think $100k is it. To put it in perspective, $100k is probably how much an intermediate engineer costs a company (not just salary). In exchange, you give up the usual cloud conveniences like easier budgeting, easier scale-up (and scale-down), lower up-front costs (which is a big deal for most early startups), better SLAs, being able to use cloud object storage without paying transfer fees, access to support, fewer things that can go wrong, etc.
• oxfordmale 4y ago
  Buying your own server requires an upfront investment. AS 90% of start-ups fail, hiring your computing resource flexibly is far better, even if it comes at a higher cost. Once your company is mature and relatively stable, it is worth considering if hosting your infrastructure is more cost-effective.
  • hinkley 4y ago
    But now you have a culture that has no experience or value with self hosted hardware. How are you going to fix that cultural problem? Hire a bunch of outsiders? How’s that gonna fly?
    • oxfordmale 4y ago
      Bringing your infrastructure from Cloud to on-perm is a project; you would have to hire a dedicated team. Once you have proven your start-up is providing a genuine solution, it is a lot easier to get funding for this.
      • hinkley 4y ago
        And you haven't had turf wars where people conflate the issues and costs so badly that nobody really knows what the truth is anymore? Apparently I'm getting your share.
      • sshine 4y ago
        You're missing several points on the spectrum. You can provision dedicated servers with Terraform. They can still be cloudy in the way you deal with hardware maintenance and risk of failure, and how you authenticate with them and how you name them, and how you hook them up on VPNs. But they're physical units without the AWS bells and whistles or the AWS premium. Not everyone needs their own colocated rack, and manage their own UPS and network peering.
        • oxfordmale 4y ago
          The main problem is OPEX versus APEX, not whether you can Terraform your infrastructure. It depends on your business model whether self-hosting is more cost effective than the Cloud. Cloud is more flexible and as a result you pay a premium, however, the cost differences are not as large as claimed in this article