15 ms·

How are such savings not obvious after putting the amounts in an Excel sheet, and spending an hour over it (and most importantly doing this before spending half

How are such savings not obvious after putting the amounts in an Excel sheet, and spending an hour over it (and most importantly doing this before spending half a million/year on AWS)?

Spivak 3y ago
I would be surprised if people didn't know that coloing was cheaper. I certainly evangelize it for workloads that are particularly expensive on AWS. It's not entirely without downsides though and I think many shops are willing to pay more for a different set of them. It is incredibly rewarding work though. You get to do magic. * You do need more experienced people, there's no way around it and the skills are hard to come by sometimes. We spent probably 3 years looking to hire a senior dba before we found one. Networking people are also unicorns. * Having to deal with the full full stack is a lot more work and needing manage IRL hardware is a PITA. I hated driving 50 miles to swap some hard drives. Rather than using those nice cloud APIs you are also on the other side implementing them. And all the VM management software sucks in their own unique ways. * Storage will make you lose sleep. Ceph is a wonder of the technological world but it will also follow you in a dark alleyway and ruin your sleep. * Building true redundancy is harder than you think it should be. "What if your ceph cluster dies?" "What if your ESXi shits the bed?" "What if Consul?" Setting things up so that you don't accidentally have single points of failure is tedious work. * You have to constantly be looking at your horizons. We made a stupid little doomsday clock web app that we put all the "in the next x days/weeks/months we have to do x or we'll have an outage." Because it will take more time than you think it should to buy equipment.
- theLiminator 3y ago
  It's great when you don't need instant elasticity and traffic is very predictable. I think it's very useful for batch processing, especially owning a GPU cluster could be great for ML startups. Hybrid cloud + bare metal is probably the way to go (though that does incur the complexity of dealing with both, which is also hard).
- deleted 3y ago
  [deleted]
mschuster91 3y ago
> and most importantly doing this before spending half a million/year on AWS AWS is... incentivizing scope creep, to put it mildly. In ye olde days, you had your ESXi blades, and if you were lucky some decent storage attached to it, and you gotta made do with what you had - if you needed more resources, you'd have to go through the entire usual corporate bullshit. Get quotes from at least three comparable vendors, line up contract details, POs, get approval from multiple levels... Now? Who cares if you spin up entire servers worth of instances for feature branch environments, and look, isn't that new AI chatbot something we could use... you get the idea. The reason why cloud (not just AWS) is so popular in corporate hellscapes is because it eliminates a lot of the busybody impeders. Shadow IT as a Service.
- tqi 3y ago
  Those busybodies are also there to keep rogue engineers from burning money on useless features (like AI chat bots) that only serve to bolster their promo packet...
  - mschuster91 3y ago
    That depends on how the incentive structures for your corporate purchase department are set up - and there's really a ton of variance there, with results ranging from everyone being happy in the best case to frustrated employees quitting in droves or the company getting burned at employer rating portals.
    - tqi 3y ago
      > That depends on how the incentive structures for your corporate purchase department are set up Sure, but that seems orthogonal to the pros and cons of having more layers of oversight (busybodies, to use your term) on infra spend. Badly run companies are badly run, and I don't think having the increased flexibility that comes from cloud providers changes that.
  - withinboredom 3y ago
    I literally started laughing at this. I worked at a bare-metal shop fairly recently and a guy on my team used a corporate credit card to set up an AWS account and create an AI chatbot. The dude nearly got fired, but your comment hit the spot. You made my night, thank you.
ldargin 3y ago
Bare-metal solutions save money, but are costly in terms of development time and lost agility. Basically, they have much more friction.
- withinboredom 3y ago
  huh, say wut? I guess before Amazon invented "the cloud" there wasn't any software companies...
  - threeseed 3y ago
    AWS isn't just IaaS they are PaaS. So it's a fact that for most use cases it will be significantly easier to manage than bare metal. Because much of it is being managed for you e.g. object store, databases etc.
    - withinboredom 3y ago
      Setting up k3s: 2 hours Setting up Garage for obj store: 1 hour. Setting up Longhorn for storage: .25 hour. Setting up db: 30 minutes. Setting up Cilium with a pool of ips to use as a lb: 45 mins. All in: ~5 hours and I'm ready to deploy and spending 300 bucks a month, just renting bare metal servers. AWS, for far less compute and same capabilities: approximately 800-1000 bucks a month, and takes about 3 hours -- we aren't even counting egress costs yet. So, for two extra hours on your initial setup, you can save a ridiculous amount of money. Maintenance is actually less work than AWS too. (source: I'm working on a youtube video)
      - threeseed 3y ago
        You should stick to making Youtube videos then. Because there is a world of difference between installing some software and making it robust enough to support a multi-million dollar business. I would be surprised if you can setup and test a proper highly-available database with automated backup in < 30 mins.
        3y ago
RhodesianHunter 3y ago
Because the amount of time your engineers will spend maintaining your little herd of pet servers, and the opportunity cost of not being able to spin up manged service X to try an experiment, are not measurable.
- yjftsjthsd-h 3y ago
  > maintaining your little herd of pet servers You know bare metal can be an automated fleet of cattle too, right?
- withinboredom 3y ago
  Have you ever heard of pxe boot? You should check out Harvester made by Rancher (IIRC). Basically, manage bare metal machines using standard k8s tooling.
- deleted 3y ago
  [deleted]
dilyevsky 3y ago
Cloud is putting spend decisions into individual EMs’ or even devs’ hands. With baremetal one team (“infra” or whatever) will own all compute and thus spend decisions need to be justified by EMs which they usually dont like ;)

How are such savings not obvious after putting the amounts in an Excel sheet, and spending an hour over it (and most importantly doing this before spending half a million/year on AWS)?

Spivak 3y ago

I would be surprised if people didn't know that coloing was cheaper. I certainly evangelize it for workloads that are particularly expensive on AWS. It's not entirely without downsides though and I think many shops are willing to pay more for a different set of them. It is incredibly rewarding work though. You get to do magic. * You do need more experienced people, there's no way around it and the skills are hard to come by sometimes. We spent probably 3 years looking to hire a senior dba before we found one. Networking people are also unicorns. * Having to deal with the full full stack is a lot more work and needing manage IRL hardware is a PITA. I hated driving 50 miles to swap some hard drives. Rather than using those nice cloud APIs you are also on the other side implementing them. And all the VM management software sucks in their own unique ways. * Storage will make you lose sleep. Ceph is a wonder of the technological world but it will also follow you in a dark alleyway and ruin your sleep. * Building true redundancy is harder than you think it should be. "What if your ceph cluster dies?" "What if your ESXi shits the bed?" "What if Consul?" Setting things up so that you don't accidentally have single points of failure is tedious work. * You have to constantly be looking at your horizons. We made a stupid little doomsday clock web app that we put all the "in the next x days/weeks/months we have to do x or we'll have an outage." Because it will take more time than you think it should to buy equipment.

theLiminator 3y ago
It's great when you don't need instant elasticity and traffic is very predictable. I think it's very useful for batch processing, especially owning a GPU cluster could be great for ML startups. Hybrid cloud + bare metal is probably the way to go (though that does incur the complexity of dealing with both, which is also hard).
deleted 3y ago
[deleted]

mschuster91 3y ago

> and most importantly doing this before spending half a million/year on AWS AWS is... incentivizing scope creep, to put it mildly. In ye olde days, you had your ESXi blades, and if you were lucky some decent storage attached to it, and you gotta made do with what you had - if you needed more resources, you'd have to go through the entire usual corporate bullshit. Get quotes from at least three comparable vendors, line up contract details, POs, get approval from multiple levels... Now? Who cares if you spin up entire servers worth of instances for feature branch environments, and look, isn't that new AI chatbot something we could use... you get the idea. The reason why cloud (not just AWS) is so popular in corporate hellscapes is because it eliminates a lot of the busybody impeders. Shadow IT as a Service.

tqi 3y ago
Those busybodies are also there to keep rogue engineers from burning money on useless features (like AI chat bots) that only serve to bolster their promo packet...
- mschuster91 3y ago
  That depends on how the incentive structures for your corporate purchase department are set up - and there's really a ton of variance there, with results ranging from everyone being happy in the best case to frustrated employees quitting in droves or the company getting burned at employer rating portals.
  - tqi 3y ago
    > That depends on how the incentive structures for your corporate purchase department are set up Sure, but that seems orthogonal to the pros and cons of having more layers of oversight (busybodies, to use your term) on infra spend. Badly run companies are badly run, and I don't think having the increased flexibility that comes from cloud providers changes that.
- withinboredom 3y ago
  I literally started laughing at this. I worked at a bare-metal shop fairly recently and a guy on my team used a corporate credit card to set up an AWS account and create an AI chatbot. The dude nearly got fired, but your comment hit the spot. You made my night, thank you.

ldargin 3y ago

Bare-metal solutions save money, but are costly in terms of development time and lost agility. Basically, they have much more friction.

withinboredom 3y ago
huh, say wut? I guess before Amazon invented "the cloud" there wasn't any software companies...
- threeseed 3y ago
  AWS isn't just IaaS they are PaaS. So it's a fact that for most use cases it will be significantly easier to manage than bare metal. Because much of it is being managed for you e.g. object store, databases etc.
  - withinboredom 3y ago
    Setting up k3s: 2 hours Setting up Garage for obj store: 1 hour. Setting up Longhorn for storage: .25 hour. Setting up db: 30 minutes. Setting up Cilium with a pool of ips to use as a lb: 45 mins. All in: ~5 hours and I'm ready to deploy and spending 300 bucks a month, just renting bare metal servers. AWS, for far less compute and same capabilities: approximately 800-1000 bucks a month, and takes about 3 hours -- we aren't even counting egress costs yet. So, for two extra hours on your initial setup, you can save a ridiculous amount of money. Maintenance is actually less work than AWS too. (source: I'm working on a youtube video)
    - threeseed 3y ago
      You should stick to making Youtube videos then. Because there is a world of difference between installing some software and making it robust enough to support a multi-million dollar business. I would be surprised if you can setup and test a proper highly-available database with automated backup in < 30 mins.
      - 3y ago

RhodesianHunter 3y ago

Because the amount of time your engineers will spend maintaining your little herd of pet servers, and the opportunity cost of not being able to spin up manged service X to try an experiment, are not measurable.

yjftsjthsd-h 3y ago
> maintaining your little herd of pet servers You know bare metal can be an automated fleet of cattle too, right?
withinboredom 3y ago
Have you ever heard of pxe boot? You should check out Harvester made by Rancher (IIRC). Basically, manage bare metal machines using standard k8s tooling.
deleted 3y ago
[deleted]

dilyevsky 3y ago

Cloud is putting spend decisions into individual EMs’ or even devs’ hands. With baremetal one team (“infra” or whatever) will own all compute and thus spend decisions need to be justified by EMs which they usually dont like ;)