6 ms·

It’s sad that racking and maintaining your own physical hardware is becoming such a lost art… I appreciate the up-front simplicity of cloud offerings as much as

It’s sad that racking and maintaining your own physical hardware is becoming such a lost art… I appreciate the up-front simplicity of cloud offerings as much as anyone, but there’s something to be said for owning your own hardware and avoiding the continual rent payments you’re sending the cloud providers. The wisdom is that cloud providers are better at infra than you, and that the economies of scale make it better to piggy back on what they’re doing, but… AWS is the most profitable part of Amazon for a reason. They’re overcharging you.

cherryswitch 3y ago
Sometimes the flexibility and time savings is worth the added upfront cost. Similar to how companies like to hire consultants or lease office space. Being able to walk away is better for short term, because companies value profits in the short term.
ilc 3y ago
For most orgs: AWS is not overcharging IMHO. When you look at the cost of the hardware + hosting. Yes, it certainly looks and feels that way. But if you've dealt with corporate IT, and had to deal with 3-6 month lead times on getting hardware, or politics to get your hands on hardware to get stuff done. AWS is cheap. It gives you velocity. If your company is large enough that it can offer the elasticity of resources that Amazon offers or even 1/4 of it... and you have an IT org that will let it happen. Yes, AWS is a waste. But with AWS... when a project dies, you can wipe its costs out, people won't hold onto hardware so they have hardware for the next project, etc... Trust me. I've been IT, I can spec and build rack systems. I am a software dev. And I've been a dev most all my career. For 90%+ of orgs... they don't have the maturity and skills to handle that type of infra without substantially distracting from their primary business.
- ttt3ts 3y ago
  I find at AWS I am always wasting engineer time optimizing dumb things. Do you know how much a TB of ram costs? Or 10TB of blazing fast NVME costs? Less than $5K. How much does that cost at AWS?! This is not even considering bandwidth which AWS overcharges soo much. Yet I waste time. Also, maintaining servers is not hard at a proper data center. It is often more hands off than the migrations cloud providers force on their customers.
  - wredue 3y ago
    Yeah. Also, to be honest, we still have process and approvals to get through to spin up AWS stuff. It’s not like process disappears just cause you’re not on your own hardware. Infra is still its own team with its own budgets poking and prodding at every damn turn for every little thing till rejecting your requests, you escalate and then have a 4 week battle over needing the space.
    - JCharante 3y ago
      Really? At my old company each division had its own budget and account and then you’d be an iam member of an account and spin up services under that account, but there’s no central authority to send a request to. There were tools to analyze underused services across all accounts (like EC2 instances constantly under 2% cpu load meaning they were flagged for downsizing if possible).
    - ilc 3y ago
      Then your company should have stayed on prem. If you are paying the price of being on prem, which is really lack of ability to provision and de-provision infra quickly. There's little point to the cloud, unless you just have no infra to begin with (small companies). I'm in a small firm now. I can't imagine having an approval process to spin up a few instances to run my tests and spin them down after. That'd be silly.
- sitkack 3y ago
  I work in cloud and you sound like a shill.
  - ilc 3y ago
    More a realist. If you have the scale to go on prem. Do it. Most firms don't. Or don't have the skills. Also, the cloud can help an IT project recover from errors. Let's say, I'm about to buy 500k of hardware to setup some storage. I get my requirements, I architect it, do my design work, and then buy the hardware. I have to over provision a bit because of reality and human error... But when I discover that the requirements, shift 2mo in my project, and I've already ordered the hardware... I may be hosed. This isn't hypothetical, this is what happens. Things evolve and shift. The cloud allows for more agility. If your firm is large enough, or has its stuff together enough, go for it on-prem. I've got 20+ years on prem.. I've seen it fail all over. I've seen cloud be a mess too. But if you told me to clean up one. I'll take the cloud.
  - throwaway2990 3y ago
    Anyone who thinks the blanket statement “cloud is expensive” really doesn’t know what they are talking about.
- benreesman 3y ago
  Getting good results on AWS/GCP is neither easy nor simple: it’s a different set of headaches. It’s still a win for a lot of use cases and I still do it quite often, but the meme that it’s this “click and you’ve just hired the best ops team in the world to work for you” and so the 50-500% markup is actually a bargain is horseshit. A Bizon box in your living room fucks AWS up on flops/$ on most instance types and pays for itself in 30 days. It is one of the best ops teams on Earth: but they’re working for you like the Google search team is working for the user.
  - mannyv 3y ago
    The problem with using a cloud provider is that you still need to know what you're doing. Your application isn't going to magically become HA/DR. You still have to make it that way, from your application design/coding up through the deployment. I mean, if you're not storing your session IDs in a data store that's reachable by all the nodes behind your load balancer then no amount of infrastructure is going to save you.
    - TheNewsIsHere 3y ago
      A great illustration of this is the GitHub outage a few years back. They had a fairly well distributed application layer but the database topography at the time didn’t consider the failure mode, even though the application layer did. That’s a realistic scenario no matter whether you’re bare metal, building out your own cloud, or using someone else’s. No amount of AWS/GCP/Azure/et al marketing changes that.
  - ilc 3y ago
    And when your comcast link dies, GG man. Oh, what about when you drop a drive? Yes, you have to learn things to goto the cloud, and I won't say it is all roses, it ain't. But... AWS is less likely to fsck it up. If you have the constant load to burn the flops 24x7x365... go for it. If you have the ops team to do it... go for it. If you don't... take a bit of time and learn the cloud which is much easier than getting on-prem right. Especially for smaller firms, this isn't even a close call IMHO.
- ChrisMarshallNY 3y ago
  For me, I prefer hosted/cloud, preferably managed. I’m quite capable of setting up whole server stacks. I did it for years, but I stopped, some time ago, and consider myself to be, for want of a better word, incompetent at being a modern admin. I think I’d screw the pooch, so I prefer that someone who does it every day, handle it. But I write Swift code, every day, so I’m not incompetent at everything.
- deleted 3y ago
  [deleted]
solardev 3y ago
Where does this stop? Do you produce your own electricity? Farm your own food? Make your own silverware and shoes? Sometimes it's just easier to outsource the things you don't want to (or aren't good at) doing yourself. If I wanted to host a website, sure, I can build a server out of parts and negotiate with my ISP and get a business pipe and handle all caching and such. Or like I can pay a provider $5/mo and get better performance and reliability with no management overhead. Yeah, maybe over 5 years I'd save more money doing it myself... but it's not worth the time. If I wanted to generate a photo or a dozen, or a few paragraphs of text, that's like a few cents worth of cloud AI. Maybe low single-digit dollars. Or I could spend thousands on fat GPUs or a Macbook, spend forever training it, and still end up with a sub-par result. AWS is profitable not just because they're overcharging you but because they are providing a hugely useful service for millions of businesses that don't want to deal with that infrastructure themselves, any more than they'd want to manage their own plumbing or electrical grid or roads and bridges leading to their office. DIY makes sense if you're doing it as a hobby or if your scale is so big that you would incur significant savings to in-house it, but for millions of small and medium businesses, it's just not the most practical approach. Nothing wrong with that. I mean, it's like saying development is such a lost art... why hire a dev if you can learn to code yourself? Sure, but not everyone wants to, can, or has time.
- Spooky23 3y ago
  Depends on the financials. My employer has generated its own electricity and steam for decades. For a small business - different story.
- ClimaxGravely 3y ago
  I hate to say this but it has gotten to the point where I'm starting to farm some of my own food (I'm starting to get fed up with produce quality issues in my hometown). Still haven't started on the silverware or shoes yet. I do agree with you though. If you are a non-tech company or a company that lacks the human resources you might as well go with the cloud.
  - solardev 3y ago
sbarre 3y ago
The question I've often heard asked when deciding on build vs buy (which can apply to cloud vs. bare metal) is: Are we in the business of building, maintaining and operating <thing to build> or do we want to buy that as a service instead and focus on our actual core business? There's more to the cost of building and operating than just the hard costs. Retaining good modern IT talent is getting harder and harder - and I'm not even talking about salaries.. You need a whole department including strong leaders who can hire, train, and lead the right people, etc.. This is something most companies wouldn't even know where to start with.
- hamandcheese 3y ago
  I don't think you need a whole department to rack a few minis and run some buildkite agents on them.
  - londons_explore 3y ago
    You need one good motivated sysadmin. But if you can motivate that same sysadmin to spend his skills on something more directly benefiting your company, then you should still buy it in.
    - c0pium 3y ago
      Now your business depends on that one person, congratulations. Hope they don’t realize that their skills are better suited to working at AWS/Azure etc. for 2x the money. Which they are.
      - londons_explore 3y ago
        It's not core stuff. If the one sysadmin leaves, you buy in a solution.
        c0pium 3y ago
        I mean, fair enough I guess. Why play the OpEx vs CapEx game when you can just pay both.
whalesalad 3y ago
You’ve never had to drive to the colo at 3am huh?
- ninkendo 3y ago
  No, I had proper OOB access and could do any kind of power cycling or remote console access from my desktop.
  - hnfong 3y ago
    Never seen a hardware failure, I see.
    - ninkendo 3y ago
      The snark is getting a bit tiring. No, plenty of hardware failures but we had redundancy. Our availability wasn’t dependent on how fast I cold drive to the data center at 3am. Drive dies, who cares, there are plenty of hot spares, we’ll deal with it during business hours. Server dies, who cares, there are lots of them. We have remote hands too, so simple hardware replacement is something you can get cheap onsite labor to do. If your operations halt until some poor sysadmin has to drive to the colo, you are absolutely doing it wrong.
ClimaxGravely 3y ago
I'm a former build engineer and used to do everything on prem and I gotta say I miss it (not being a build engineer, the on prem experience). Since those days pretty much every company I've worked at moved their CI/CD to the cloud and I gotta say it feels so much slower, even when working from home. I remember twice switching from an in-house jenkins/teamcity/whatever type of CI to Azure devops and the thing I remember the most was how much longer it took a build to complete as well as the massively longer time downloading a build from Azure vs from within the office. Even when working from home the on-prem stuff was faster. The thing is, the build/devops teams seem to be about the same size in both cases. It's just kind of worse in pretty much every case when we do CI in the cloud. Notes - My experiences are largely for game development so the build times and artifact sizes can be quite large. - I've only ever had CI/CD experience with Azure, I've not tried other cloud providers - Since this is game development and we're using CI downtime is more acceptable than other cases. That said, I don't remember much downtime when I was working as a build engineer. I have seen periods of 1-2 hours of downtime once in a blue moon but then again I've seen that with Azure. In both cases it wasn't so much the setup but a build script deployment issue. Also being able to cool off in the rack room when it's a hot day is always a treat :)

cherryswitch 3y ago

Sometimes the flexibility and time savings is worth the added upfront cost. Similar to how companies like to hire consultants or lease office space. Being able to walk away is better for short term, because companies value profits in the short term.

ilc 3y ago

For most orgs: AWS is not overcharging IMHO. When you look at the cost of the hardware + hosting. Yes, it certainly looks and feels that way. But if you've dealt with corporate IT, and had to deal with 3-6 month lead times on getting hardware, or politics to get your hands on hardware to get stuff done. AWS is cheap. It gives you velocity. If your company is large enough that it can offer the elasticity of resources that Amazon offers or even 1/4 of it... and you have an IT org that will let it happen. Yes, AWS is a waste. But with AWS... when a project dies, you can wipe its costs out, people won't hold onto hardware so they have hardware for the next project, etc... Trust me. I've been IT, I can spec and build rack systems. I am a software dev. And I've been a dev most all my career. For 90%+ of orgs... they don't have the maturity and skills to handle that type of infra without substantially distracting from their primary business.

ttt3ts 3y ago
I find at AWS I am always wasting engineer time optimizing dumb things. Do you know how much a TB of ram costs? Or 10TB of blazing fast NVME costs? Less than $5K. How much does that cost at AWS?! This is not even considering bandwidth which AWS overcharges soo much. Yet I waste time. Also, maintaining servers is not hard at a proper data center. It is often more hands off than the migrations cloud providers force on their customers.
- wredue 3y ago
  Yeah. Also, to be honest, we still have process and approvals to get through to spin up AWS stuff. It’s not like process disappears just cause you’re not on your own hardware. Infra is still its own team with its own budgets poking and prodding at every damn turn for every little thing till rejecting your requests, you escalate and then have a 4 week battle over needing the space.
  - JCharante 3y ago
    Really? At my old company each division had its own budget and account and then you’d be an iam member of an account and spin up services under that account, but there’s no central authority to send a request to. There were tools to analyze underused services across all accounts (like EC2 instances constantly under 2% cpu load meaning they were flagged for downsizing if possible).
  - ilc 3y ago
    Then your company should have stayed on prem. If you are paying the price of being on prem, which is really lack of ability to provision and de-provision infra quickly. There's little point to the cloud, unless you just have no infra to begin with (small companies). I'm in a small firm now. I can't imagine having an approval process to spin up a few instances to run my tests and spin them down after. That'd be silly.
sitkack 3y ago
I work in cloud and you sound like a shill.
- ilc 3y ago
  More a realist. If you have the scale to go on prem. Do it. Most firms don't. Or don't have the skills. Also, the cloud can help an IT project recover from errors. Let's say, I'm about to buy 500k of hardware to setup some storage. I get my requirements, I architect it, do my design work, and then buy the hardware. I have to over provision a bit because of reality and human error... But when I discover that the requirements, shift 2mo in my project, and I've already ordered the hardware... I may be hosed. This isn't hypothetical, this is what happens. Things evolve and shift. The cloud allows for more agility. If your firm is large enough, or has its stuff together enough, go for it on-prem. I've got 20+ years on prem.. I've seen it fail all over. I've seen cloud be a mess too. But if you told me to clean up one. I'll take the cloud.
- throwaway2990 3y ago
  Anyone who thinks the blanket statement “cloud is expensive” really doesn’t know what they are talking about.
benreesman 3y ago
Getting good results on AWS/GCP is neither easy nor simple: it’s a different set of headaches. It’s still a win for a lot of use cases and I still do it quite often, but the meme that it’s this “click and you’ve just hired the best ops team in the world to work for you” and so the 50-500% markup is actually a bargain is horseshit. A Bizon box in your living room fucks AWS up on flops/$ on most instance types and pays for itself in 30 days. It is one of the best ops teams on Earth: but they’re working for you like the Google search team is working for the user.
- mannyv 3y ago
  The problem with using a cloud provider is that you still need to know what you're doing. Your application isn't going to magically become HA/DR. You still have to make it that way, from your application design/coding up through the deployment. I mean, if you're not storing your session IDs in a data store that's reachable by all the nodes behind your load balancer then no amount of infrastructure is going to save you.
  - TheNewsIsHere 3y ago
    A great illustration of this is the GitHub outage a few years back. They had a fairly well distributed application layer but the database topography at the time didn’t consider the failure mode, even though the application layer did. That’s a realistic scenario no matter whether you’re bare metal, building out your own cloud, or using someone else’s. No amount of AWS/GCP/Azure/et al marketing changes that.
- ilc 3y ago
  And when your comcast link dies, GG man. Oh, what about when you drop a drive? Yes, you have to learn things to goto the cloud, and I won't say it is all roses, it ain't. But... AWS is less likely to fsck it up. If you have the constant load to burn the flops 24x7x365... go for it. If you have the ops team to do it... go for it. If you don't... take a bit of time and learn the cloud which is much easier than getting on-prem right. Especially for smaller firms, this isn't even a close call IMHO.
ChrisMarshallNY 3y ago
For me, I prefer hosted/cloud, preferably managed. I’m quite capable of setting up whole server stacks. I did it for years, but I stopped, some time ago, and consider myself to be, for want of a better word, incompetent at being a modern admin. I think I’d screw the pooch, so I prefer that someone who does it every day, handle it. But I write Swift code, every day, so I’m not incompetent at everything.
deleted 3y ago
[deleted]

solardev 3y ago

Where does this stop? Do you produce your own electricity? Farm your own food? Make your own silverware and shoes? Sometimes it's just easier to outsource the things you don't want to (or aren't good at) doing yourself. If I wanted to host a website, sure, I can build a server out of parts and negotiate with my ISP and get a business pipe and handle all caching and such. Or like I can pay a provider $5/mo and get better performance and reliability with no management overhead. Yeah, maybe over 5 years I'd save more money doing it myself... but it's not worth the time. If I wanted to generate a photo or a dozen, or a few paragraphs of text, that's like a few cents worth of cloud AI. Maybe low single-digit dollars. Or I could spend thousands on fat GPUs or a Macbook, spend forever training it, and still end up with a sub-par result. AWS is profitable not just because they're overcharging you but because they are providing a hugely useful service for millions of businesses that don't want to deal with that infrastructure themselves, any more than they'd want to manage their own plumbing or electrical grid or roads and bridges leading to their office. DIY makes sense if you're doing it as a hobby or if your scale is so big that you would incur significant savings to in-house it, but for millions of small and medium businesses, it's just not the most practical approach. Nothing wrong with that. I mean, it's like saying development is such a lost art... why hire a dev if you can learn to code yourself? Sure, but not everyone wants to, can, or has time.

Spooky23 3y ago
Depends on the financials. My employer has generated its own electricity and steam for decades. For a small business - different story.
ClimaxGravely 3y ago
I hate to say this but it has gotten to the point where I'm starting to farm some of my own food (I'm starting to get fed up with produce quality issues in my hometown). Still haven't started on the silverware or shoes yet. I do agree with you though. If you are a non-tech company or a company that lacks the human resources you might as well go with the cloud.
- solardev 3y ago

sbarre 3y ago

The question I've often heard asked when deciding on build vs buy (which can apply to cloud vs. bare metal) is: Are we in the business of building, maintaining and operating <thing to build> or do we want to buy that as a service instead and focus on our actual core business? There's more to the cost of building and operating than just the hard costs. Retaining good modern IT talent is getting harder and harder - and I'm not even talking about salaries.. You need a whole department including strong leaders who can hire, train, and lead the right people, etc.. This is something most companies wouldn't even know where to start with.

hamandcheese 3y ago
I don't think you need a whole department to rack a few minis and run some buildkite agents on them.
- londons_explore 3y ago
  You need one good motivated sysadmin. But if you can motivate that same sysadmin to spend his skills on something more directly benefiting your company, then you should still buy it in.
  - c0pium 3y ago
    Now your business depends on that one person, congratulations. Hope they don’t realize that their skills are better suited to working at AWS/Azure etc. for 2x the money. Which they are.
    - londons_explore 3y ago
      It's not core stuff. If the one sysadmin leaves, you buy in a solution.
      - c0pium 3y ago
        I mean, fair enough I guess. Why play the OpEx vs CapEx game when you can just pay both.

whalesalad 3y ago

You’ve never had to drive to the colo at 3am huh?

ninkendo 3y ago
No, I had proper OOB access and could do any kind of power cycling or remote console access from my desktop.
- hnfong 3y ago
  Never seen a hardware failure, I see.
  - ninkendo 3y ago
    The snark is getting a bit tiring. No, plenty of hardware failures but we had redundancy. Our availability wasn’t dependent on how fast I cold drive to the data center at 3am. Drive dies, who cares, there are plenty of hot spares, we’ll deal with it during business hours. Server dies, who cares, there are lots of them. We have remote hands too, so simple hardware replacement is something you can get cheap onsite labor to do. If your operations halt until some poor sysadmin has to drive to the colo, you are absolutely doing it wrong.

ClimaxGravely 3y ago

I'm a former build engineer and used to do everything on prem and I gotta say I miss it (not being a build engineer, the on prem experience). Since those days pretty much every company I've worked at moved their CI/CD to the cloud and I gotta say it feels so much slower, even when working from home. I remember twice switching from an in-house jenkins/teamcity/whatever type of CI to Azure devops and the thing I remember the most was how much longer it took a build to complete as well as the massively longer time downloading a build from Azure vs from within the office. Even when working from home the on-prem stuff was faster. The thing is, the build/devops teams seem to be about the same size in both cases. It's just kind of worse in pretty much every case when we do CI in the cloud. Notes - My experiences are largely for game development so the build times and artifact sizes can be quite large. - I've only ever had CI/CD experience with Azure, I've not tried other cloud providers - Since this is game development and we're using CI downtime is more acceptable than other cases. That said, I don't remember much downtime when I was working as a build engineer. I have seen periods of 1-2 hours of downtime once in a blue moon but then again I've seen that with Azure. In both cases it wasn't so much the setup but a build script deployment issue. Also being able to cool off in the rack room when it's a hot day is always a treat :)