5 ms·

Nice writing! My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally h

Nice writing! My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally hard for them to even consider having physical hardware (yet on colocation!) and managing such systems. It's just out of equation. On the article itself, billions scale payments are astonishing. And how much paying for traffic would add here too.

RicoElectrico 2y ago
Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?
- CoolCold 2y ago
  > Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort. > Second, connecting all the piping gives an illusion of doing work and feeling "important". This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
  - electric_mayhem 2y ago
    I’ve been doing infrastructure for 27 years. Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups. One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility. Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility. They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands. But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money. So they moved to Amazon. Their first month’s bill was $50,000. And it only went up from there. The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
    - CoolCold 2y ago
      > But their new VC-installed CTO was like… No! Largely the point I'm trying to put here > Their first month’s bill was $50,000. And it only went up from there. Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
      - Nullabillity 2y ago
        You're mixing yearly salary figures and monthly infra bills.
        throwaway11460 2y ago
        50k*12=600k, equivalent to 1-3 SWE employees; practically not even a single small team. It's nothing. Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
        electric_mayhem 2y ago
        I’ve seen ICs lose promotions they were on course for and get wrecked on their bonuses for losing track of a single $50k/mo cluster in AWS and leaving it running though unused. At a FAANG company. So, ICs who sweat that kind of waste are right to do so. If management sees that someone thinks 50k/mo is no big deal to squander, bet your ass there’s a good chance they’ll take at least a month’s worth out of that person’s annual bonus since that kind of money matters so little to them.
        throwaway11460 2y ago
        Sure, that money shouldn't be wasted - but thinking it's a significant part of the budget is misguided.
        2y ago
RamblingCTO 2y ago
Running your own data center is just a PITA no one wants to do. Non-C-levels just love to ignore the costs you have when doing such a thing by just comparing hardware costs to the cloud costs. There's always the liability aspect, managing less people, externalizing some complexity to another company etc. etc. All worth a lot of money, at least for me personally. I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
- CoolCold 2y ago
  > Running your own data center is just a PITA I believe in the article they clearly say they just use colo, not running own DCs > I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV. No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
  - RamblingCTO 2y ago
    Maybe my point didn't come across properly. Me personally, I prefer to keep the org complexity down and externalize some headcaches if I can. Having to also manage hardware, people that manage this hardware and not being able to keep other people liable for a given SLA. It's just yet another thing that other companies are really good at and for me it's just not worth it to do on our own. If I don't really need it, why would I? And again, there are more costs associated with this approach than just infra vs hardware + people costs. Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way. In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.
SkyPuncher 2y ago
In my eyes, physical hardware is something non-growth companies do to improve margins. When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.
- nojvek 2y ago
  Exactly. Dropbox was cloud and then later moved to on-prem to save on storage costs. Anyone running >1000 servers with >80% utilization should consider going partially colocated. Comma.ai and Tesla come to mind who run their own training clusters. If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy. Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.
barryrandall 2y ago
It's investors and the one-size-fits-all metrics many rely on. They want a company's financial statements to look like the financial statements from other financial instruments (companies) they're familiar with. If your industry has 30% EBITDA, your company needs to have ~30% EBITDA. If your industry is 100% cloud, your company is expected to be ~100% cloud. If your numbers deviate from expectations, they'll demand an explanation. If that explanation isn't your company's secret to success, then it's probably not going to be accepted.

RicoElectrico 2y ago

Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?

CoolCold 2y ago
> Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort. > Second, connecting all the piping gives an illusion of doing work and feeling "important". This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
- electric_mayhem 2y ago
  I’ve been doing infrastructure for 27 years. Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups. One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility. Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility. They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands. But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money. So they moved to Amazon. Their first month’s bill was $50,000. And it only went up from there. The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
  - CoolCold 2y ago
    > But their new VC-installed CTO was like… No! Largely the point I'm trying to put here > Their first month’s bill was $50,000. And it only went up from there. Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
    - Nullabillity 2y ago
      You're mixing yearly salary figures and monthly infra bills.
      - throwaway11460 2y ago
        50k*12=600k, equivalent to 1-3 SWE employees; practically not even a single small team. It's nothing. Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
        electric_mayhem 2y ago
        I’ve seen ICs lose promotions they were on course for and get wrecked on their bonuses for losing track of a single $50k/mo cluster in AWS and leaving it running though unused. At a FAANG company. So, ICs who sweat that kind of waste are right to do so. If management sees that someone thinks 50k/mo is no big deal to squander, bet your ass there’s a good chance they’ll take at least a month’s worth out of that person’s annual bonus since that kind of money matters so little to them.
        throwaway11460 2y ago
        Sure, that money shouldn't be wasted - but thinking it's a significant part of the budget is misguided.
      - 2y ago

RamblingCTO 2y ago

Running your own data center is just a PITA no one wants to do. Non-C-levels just love to ignore the costs you have when doing such a thing by just comparing hardware costs to the cloud costs. There's always the liability aspect, managing less people, externalizing some complexity to another company etc. etc. All worth a lot of money, at least for me personally. I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.

CoolCold 2y ago
> Running your own data center is just a PITA I believe in the article they clearly say they just use colo, not running own DCs > I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV. No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
- RamblingCTO 2y ago
  Maybe my point didn't come across properly. Me personally, I prefer to keep the org complexity down and externalize some headcaches if I can. Having to also manage hardware, people that manage this hardware and not being able to keep other people liable for a given SLA. It's just yet another thing that other companies are really good at and for me it's just not worth it to do on our own. If I don't really need it, why would I? And again, there are more costs associated with this approach than just infra vs hardware + people costs. Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way. In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.

SkyPuncher 2y ago

In my eyes, physical hardware is something non-growth companies do to improve margins. When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.

nojvek 2y ago
Exactly. Dropbox was cloud and then later moved to on-prem to save on storage costs. Anyone running >1000 servers with >80% utilization should consider going partially colocated. Comma.ai and Tesla come to mind who run their own training clusters. If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy. Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.

barryrandall 2y ago

It's investors and the one-size-fits-all metrics many rely on. They want a company's financial statements to look like the financial statements from other financial instruments (companies) they're familiar with. If your industry has 30% EBITDA, your company needs to have ~30% EBITDA. If your industry is 100% cloud, your company is expected to be ~100% cloud. If your numbers deviate from expectations, they'll demand an explanation. If that explanation isn't your company's secret to success, then it's probably not going to be accepted.