4 ms·

This is FUD unless you're running a stock exchange or payment processor where every minute of downtime will cost you hundreds of thousands. For most businesses

This is FUD unless you're running a stock exchange or payment processor where every minute of downtime will cost you hundreds of thousands. For most businesses this is fear-mongering to keep the DevOps & cloud industry going and ensure continued careers in this field.

The_Colonel 2y ago
It's not just about downtime, but also about not getting your systems hacked, not losing your data if sh1t hits the fan, regulation compliance, flexibility (e.g. ability to quickly spin-out new test envs) etc. My preferred solution to this problem is different, though. For most businesses, apps, a monolith (maybe with a few extra services) + 1 relational DB is all you need. In such a simple setup, many of the problems faced either disappear or get much smaller.
- packetlost 2y ago
  > also about not getting your systems hacked... The only systems I have ever seen get compromised firsthand were in public clouds and because they were in public clouds. Most of my career has been at shops that, for one reason or another, primarily own their own infrastructure, cloud represents a rather small fraction. It's far easier to secure a few servers behind a firewall than figure out the Rube Goldberg Machine that is cloud configuration. > not losing your data if sh1t hits the fan You can use off-site backup without using cloud systems, you know? Backblaze, AWS Glacier, etc. are all pretty reasonable solutions. Most of the time when I've seen the need to exercise the backup strategy it's because of some software fuckup, not something like a disk dying. Using a managed database isn't going to save you when the intern TRUNCATEs the prod database on accident (and if something like that happens, it means you fucked up elsewhere). > regulation compliance Most shops would be way better suited to paying a payment processor like Stripe, or other equivalent vendors for similarly protected data. Defense is a whole can of worms, "government clouds" are a scam that make you more vulnerable to an unauthorized export than less. > flexibility (e.g. ability to quickly spin-out new test envs) etc. You actually lose flexibility by buying into a particular cloud provider, not gain it. Some things become easier, but many things become harder. Also, IME the hard part of creating reasonable test envs is configuring your edge (ingress, logging infra) and data.
HolyLampshade 2y ago
Speaking of the exchanges (at least the sanely operated ones), there’s a reason the stack is simplified compared to most of what is being described here. When some component fails you absolutely do not want to spend time trying to figure out the underlying cause. Almost all the cases you hear in media of exchange outages are due to unnecessary complexity added to what is already a remarkably complex distributed (in most well designed cases) state machine. You generally want things to be as simple and streamlined as possible so when something does pop (and it will) your mean time to resolution is inside of a minute.
almost 2y ago
I run a business that is a long long way from a stock exchange or a payment processor. And while a few minutes of downtime is fine 30 minutes or a few hours at the wrong time will really make my customers quite sad. I've been woken in the small hours with technical problems maybe a couple of times over the last 8 years of running it and am quite willing to pay more for my hosting to avoid that happening again. Not for Heroku, they're absolute garbage these days, but definitely for a better run PaaS. Plenty of situations where running it yourself makes sense of course. If you have the people and the skills available (and the cost tradeoffs make sense) or if downtime really doesn't matter much at all to you then go ahead and consider things like this (or possibly simpler self hosting options, it depdns).But no, "you gotta run kubernettes yourself unless you're a stock exchange" is not a sensible position.
- Maxion 2y ago
  I don't know why people don't value their time at all. PaaS are so cheap these days for the majority of projects, that it just is not worth it to spend your own time to manage the whole infrastructure stack. If you're forced by regulation or if you just want to do it to learn, than yeah. But if your business is not running infra, or if your infra demands aren't crazy, then PaaS and what-have-you-flavored-cloud-container products will cost you ~1-2 work weeks of a single developer annually.
  - sgarland 2y ago
    Unless you already know how to run infra quickly and efficiently. Which – spoiler – you can achieve if you want to learn.
gspencley 2y ago
It's not FUD, it's pointing out a very real fact that most problems are not engineering problems that you can fix by choosing the one "magical" engineering solution that will work for all (or even most) situations. You need to understand your business and your requirements. Us engineers love to think that we can solve everything with the right tools or right engineering solutions. That's not true. There is no "perfect framework." No one sized fits all solution that will magically solve everything. What "stack" you choose, what programming language, which frameworks, which hosting providers ... these are all as much business decisions as they are engineering decisions. Good engineering isn't just about finding the simplest or cheapest solution. It is about understanding the business requirements and finding the right solution for the business.
- pphysch 2y ago
  Having managers (business people) make technical decisions based on marketing copy is how you get 10 technical problems that metastasize into 100 business problems, usually with little awareness of how we got there in the first place.
  - gspencley 2y ago
    Nice straw-man. I never once suggested that business people should be making technical decisions. What I said was that engineering solutions need to serve the needs of the business. Those are insanely different statements. They are so different that I think that you actively tried to misinterpret my comment so that you could shoot down something I didn't say.
    - pphysch 2y ago
      Well, you're using an overbroad definition of "business decisions", so forgive my interpretation. Of course everyone that goes on in a business could be conflated as a "business decision". But not everyone at the business is an MBA, so to speak. "Business" has particular semantics in this case, otherwise "engineering/technical" becomes an empty descriptor.

The_Colonel 2y ago

It's not just about downtime, but also about not getting your systems hacked, not losing your data if sh1t hits the fan, regulation compliance, flexibility (e.g. ability to quickly spin-out new test envs) etc. My preferred solution to this problem is different, though. For most businesses, apps, a monolith (maybe with a few extra services) + 1 relational DB is all you need. In such a simple setup, many of the problems faced either disappear or get much smaller.

packetlost 2y ago
> also about not getting your systems hacked... The only systems I have ever seen get compromised firsthand were in public clouds and because they were in public clouds. Most of my career has been at shops that, for one reason or another, primarily own their own infrastructure, cloud represents a rather small fraction. It's far easier to secure a few servers behind a firewall than figure out the Rube Goldberg Machine that is cloud configuration. > not losing your data if sh1t hits the fan You can use off-site backup without using cloud systems, you know? Backblaze, AWS Glacier, etc. are all pretty reasonable solutions. Most of the time when I've seen the need to exercise the backup strategy it's because of some software fuckup, not something like a disk dying. Using a managed database isn't going to save you when the intern TRUNCATEs the prod database on accident (and if something like that happens, it means you fucked up elsewhere). > regulation compliance Most shops would be way better suited to paying a payment processor like Stripe, or other equivalent vendors for similarly protected data. Defense is a whole can of worms, "government clouds" are a scam that make you more vulnerable to an unauthorized export than less. > flexibility (e.g. ability to quickly spin-out new test envs) etc. You actually lose flexibility by buying into a particular cloud provider, not gain it. Some things become easier, but many things become harder. Also, IME the hard part of creating reasonable test envs is configuring your edge (ingress, logging infra) and data.

HolyLampshade 2y ago

Speaking of the exchanges (at least the sanely operated ones), there’s a reason the stack is simplified compared to most of what is being described here. When some component fails you absolutely do not want to spend time trying to figure out the underlying cause. Almost all the cases you hear in media of exchange outages are due to unnecessary complexity added to what is already a remarkably complex distributed (in most well designed cases) state machine. You generally want things to be as simple and streamlined as possible so when something does pop (and it will) your mean time to resolution is inside of a minute.

almost 2y ago

I run a business that is a long long way from a stock exchange or a payment processor. And while a few minutes of downtime is fine 30 minutes or a few hours at the wrong time will really make my customers quite sad. I've been woken in the small hours with technical problems maybe a couple of times over the last 8 years of running it and am quite willing to pay more for my hosting to avoid that happening again. Not for Heroku, they're absolute garbage these days, but definitely for a better run PaaS. Plenty of situations where running it yourself makes sense of course. If you have the people and the skills available (and the cost tradeoffs make sense) or if downtime really doesn't matter much at all to you then go ahead and consider things like this (or possibly simpler self hosting options, it depdns).But no, "you gotta run kubernettes yourself unless you're a stock exchange" is not a sensible position.

Maxion 2y ago
I don't know why people don't value their time at all. PaaS are so cheap these days for the majority of projects, that it just is not worth it to spend your own time to manage the whole infrastructure stack. If you're forced by regulation or if you just want to do it to learn, than yeah. But if your business is not running infra, or if your infra demands aren't crazy, then PaaS and what-have-you-flavored-cloud-container products will cost you ~1-2 work weeks of a single developer annually.
- sgarland 2y ago
  Unless you already know how to run infra quickly and efficiently. Which – spoiler – you can achieve if you want to learn.

gspencley 2y ago

It's not FUD, it's pointing out a very real fact that most problems are not engineering problems that you can fix by choosing the one "magical" engineering solution that will work for all (or even most) situations. You need to understand your business and your requirements. Us engineers love to think that we can solve everything with the right tools or right engineering solutions. That's not true. There is no "perfect framework." No one sized fits all solution that will magically solve everything. What "stack" you choose, what programming language, which frameworks, which hosting providers ... these are all as much business decisions as they are engineering decisions. Good engineering isn't just about finding the simplest or cheapest solution. It is about understanding the business requirements and finding the right solution for the business.

pphysch 2y ago
Having managers (business people) make technical decisions based on marketing copy is how you get 10 technical problems that metastasize into 100 business problems, usually with little awareness of how we got there in the first place.
- gspencley 2y ago
  Nice straw-man. I never once suggested that business people should be making technical decisions. What I said was that engineering solutions need to serve the needs of the business. Those are insanely different statements. They are so different that I think that you actively tried to misinterpret my comment so that you could shoot down something I didn't say.
  - pphysch 2y ago
    Well, you're using an overbroad definition of "business decisions", so forgive my interpretation. Of course everyone that goes on in a business could be conflated as a "business decision". But not everyone at the business is an MBA, so to speak. "Business" has particular semantics in this case, otherwise "engineering/technical" becomes an empty descriptor.