6 ms·

And here it is, the main problem with outsourcing critical infrastructure. Remote server providers like Microsoft Azure should at most be used as twins/redundan

And here it is, the main problem with outsourcing critical infrastructure. Remote server providers like Microsoft Azure should at most be used as twins/redundant systems to a locally managed system. Governments of the world: pay your IT people more money to prevent brain drain.

aynsof 6y ago
Do you think locally managed systems are immune to outages? Or that governments are capable of resourcing their teams sufficiently to do a better job at availability than Microsoft, Google, or Amazon?
- walrus01 6y ago
  Your comment is a fine example of the standard rhetoric from the 'move everything to the cloud' marketing people, but perhaps organizations should consider that they don't have to go with one cloud provider as a single point of failure. It's the lazy way of abstracting away responsibility and blame to some other party, and in my experience often does not result in better availability than a properly implemented "belt and suspenders" approach.
  - deleted 6y ago
    [deleted]
  - mattzito 6y ago
    There’s a complexity cost, though, right? It’s not free to run two different systems, manage two different billing systems, different tools, pay for cross-provider bandwidth, etc.? I’m sure there are some cases where the cost is worth the complexity, but I don’t think it’s cut and dried as written here. Most cloud providers are very reliable, and my guess is that you are more likely to have a self-inflicted outage due to the complexity of your infrastructure than the cloud provider having an outage.
  - emodendroket 6y ago
    Or it’s borne of experience of doing it oneself being much less reliable than cloud providers even accounting for these rare outages.
    - deleted 6y ago
      [deleted]
- cm2187 6y ago
  I can think of a few good reasons. Strategic (you don’t want to hand over to a foreign power your critical infrastructures). Diversification (if all your banks run on aws, the day aws goes down you don’t have a banking system anymore). Not being at the mercy of a capricious tech billionaire (what happened to Parler could very well happen to a state if the said billionaire doesn’t like your policy).
  - austinheap 6y ago
    I’m unaware of any Core Banking Systems (CBS) that runs on a cloud provider with the exception being Finastra (Azure). Other parts of retail banking stacks? Sure. Not their cores.
    - thetinguy 6y ago
      The major cloud providers all aim for 99.999% uptime. Keep in mind that 99.99% uptime means ~4 minutes of downtime a month. I think there are other reasons that banks may not want or have the ability to run their core services on the cloud.
      - austinheap 6y ago
        The vast majority of retail banking institutions do not run their core. They outsource it to the major core vendors to ‘protect themselves from themselves’. It’s atypical motivation but one of the few verticals I know of where the cloud is largely out of the equation.
      - medium_burrito 6y ago
        Let's be honest, it's Azure. The only reason you choose it is because either you compete with Amazon in some way, or you hate your infrastructure engineers, or don't know better.
        emodendroket 6y ago
        Quite a few businesses are still simply more comfortable with any idea if Microsoft does it than otherwise.
- dec0dedab0de 6y ago
  The comment you're replying to says the cloud should be used as a redundant system. So they are inherently saying they are not immune to outages.
- grey-area 6y ago
  On the evidence, yes. Cloud providers frequently have outages lasting multiple hours affecting millions of clients. Some governments manage to run critical servers without downtime.
  - aynsof 6y ago
    While I accept of course that there are government-owned systems that have uptime measured in years (decades?), I think that extrapolating that into an argument about the overall reliability of those systems falls into survivorship bias. But if I had to make an a priori prediction about which systems would have the better reliability over the long term - cloud-based or those run out of a corporate DC - my money would be on the cloud-based systems. The cloud providers are going to have better management of power/network/hardware than the most mature government agency, simply because it's a core capability for them.
    - grey-area 6y ago
      Change is a leading cause of outages. The cloud providers are constantly changing and upgrading their services, and constantly have downtime ranging from minutes to hours. In contrast UK gov servers like hmrc.gov.uk (for example) are pretty stable, I can't remember the last time I heard about an outage. Cloud providers are certainly better at some things (like staying up to date with latest tech), but I'd contend reliability is not one of those things based on their prominent and frequent outages (at least several a year).
- DougN7 6y ago
  I’m starting to think the answer is yes. My small systems have not had outages like Azure has.
spydum 6y ago
So you think local IT can acheive the same high availability and elasticity? Sorry that isn't usually my experience. Lots of anecdotal local IT get lucky, but on average I think this is the wrong lesson to learn.
- mhh__ 6y ago
  > can acheive the same high availability and elasticity Maybe not, but even just a "flick the switch to go back to a dumb system" option is worth maintaining
  - thinkharderdev 6y ago
    Is it really? You would maintain an entire parallel stack built on an entirely separate infrastructure stack, with completely different deployment patterns and all the data synced? DR is really hard if you just want to fail over to another AWS/Azure/GCP region. I can't imagine what a nightmare it would be to maintain an on-prem DR standby. To mitigate a couple hours per year (in a really bad year) of cloud downtime?
    - mhh__ 6y ago
      Well it depends on the system, doesn't it? For filing taxes, it doesn't matter that much if there's downtime as long as you have a plan to move the demand around (i.e. don't fine people when it's down), but for 911 calls you're going to have a lot of fairly niche infrastructure that you need to plumb in any way, so the actual cloud parts should be comparatively easy to replace.
      - thinkharderdev 6y ago
        For those sorts of systems, my bet is that the DR strategy is to fall back to what they did before computers existed at all. And honestly that may be the best option. As long as you have pencils, paper and telephones then you can probably approximate what you were doing in 1995 anyway.
- 6y ago
nine_k 6y ago
I'd say that you need critical infrastructure redundantly deployed everywhere, and experiencing a training outage on one of the platform each quarter. But wanting is one thing, having the money to implement and maintain such a solution, a quite another thing :-/
deleted 6y ago
[deleted]
908B64B197 6y ago
They would need to triple the salaries for new grads to make it on par with Microsoft. Junior employees over there make around 45K USD[0], compare to Microsoft 110K[1]. Not including bonuses and stock, of course. I don't think they could build infrastructure even if they wanted to. [0] https://www.tbs-sct.gc.ca/agreements-conventions/view-visualiser-eng.aspx?id=1#tocxx312259 https://www.tbs-sct.gc.ca/agreements-conventions/view-visual... [1] https://www.levels.fyi/company/Microsoft/salaries/Software-Engineer/ https://www.levels.fyi/company/Microsoft/salaries/Software-E...
deleted 6y ago
[deleted]

aynsof 6y ago

Do you think locally managed systems are immune to outages? Or that governments are capable of resourcing their teams sufficiently to do a better job at availability than Microsoft, Google, or Amazon?

walrus01 6y ago
Your comment is a fine example of the standard rhetoric from the 'move everything to the cloud' marketing people, but perhaps organizations should consider that they don't have to go with one cloud provider as a single point of failure. It's the lazy way of abstracting away responsibility and blame to some other party, and in my experience often does not result in better availability than a properly implemented "belt and suspenders" approach.
- deleted 6y ago
  [deleted]
- mattzito 6y ago
  There’s a complexity cost, though, right? It’s not free to run two different systems, manage two different billing systems, different tools, pay for cross-provider bandwidth, etc.? I’m sure there are some cases where the cost is worth the complexity, but I don’t think it’s cut and dried as written here. Most cloud providers are very reliable, and my guess is that you are more likely to have a self-inflicted outage due to the complexity of your infrastructure than the cloud provider having an outage.
- emodendroket 6y ago
  Or it’s borne of experience of doing it oneself being much less reliable than cloud providers even accounting for these rare outages.
  - deleted 6y ago
    [deleted]
cm2187 6y ago
I can think of a few good reasons. Strategic (you don’t want to hand over to a foreign power your critical infrastructures). Diversification (if all your banks run on aws, the day aws goes down you don’t have a banking system anymore). Not being at the mercy of a capricious tech billionaire (what happened to Parler could very well happen to a state if the said billionaire doesn’t like your policy).
- austinheap 6y ago
  I’m unaware of any Core Banking Systems (CBS) that runs on a cloud provider with the exception being Finastra (Azure). Other parts of retail banking stacks? Sure. Not their cores.
  - thetinguy 6y ago
    The major cloud providers all aim for 99.999% uptime. Keep in mind that 99.99% uptime means ~4 minutes of downtime a month. I think there are other reasons that banks may not want or have the ability to run their core services on the cloud.
    - austinheap 6y ago
      The vast majority of retail banking institutions do not run their core. They outsource it to the major core vendors to ‘protect themselves from themselves’. It’s atypical motivation but one of the few verticals I know of where the cloud is largely out of the equation.
    - medium_burrito 6y ago
      Let's be honest, it's Azure. The only reason you choose it is because either you compete with Amazon in some way, or you hate your infrastructure engineers, or don't know better.
      - emodendroket 6y ago
        Quite a few businesses are still simply more comfortable with any idea if Microsoft does it than otherwise.
dec0dedab0de 6y ago
The comment you're replying to says the cloud should be used as a redundant system. So they are inherently saying they are not immune to outages.
grey-area 6y ago
On the evidence, yes. Cloud providers frequently have outages lasting multiple hours affecting millions of clients. Some governments manage to run critical servers without downtime.
- aynsof 6y ago
  While I accept of course that there are government-owned systems that have uptime measured in years (decades?), I think that extrapolating that into an argument about the overall reliability of those systems falls into survivorship bias. But if I had to make an a priori prediction about which systems would have the better reliability over the long term - cloud-based or those run out of a corporate DC - my money would be on the cloud-based systems. The cloud providers are going to have better management of power/network/hardware than the most mature government agency, simply because it's a core capability for them.
  - grey-area 6y ago
    Change is a leading cause of outages. The cloud providers are constantly changing and upgrading their services, and constantly have downtime ranging from minutes to hours. In contrast UK gov servers like hmrc.gov.uk (for example) are pretty stable, I can't remember the last time I heard about an outage. Cloud providers are certainly better at some things (like staying up to date with latest tech), but I'd contend reliability is not one of those things based on their prominent and frequent outages (at least several a year).
DougN7 6y ago
I’m starting to think the answer is yes. My small systems have not had outages like Azure has.

spydum 6y ago

So you think local IT can acheive the same high availability and elasticity? Sorry that isn't usually my experience. Lots of anecdotal local IT get lucky, but on average I think this is the wrong lesson to learn.

mhh__ 6y ago
> can acheive the same high availability and elasticity Maybe not, but even just a "flick the switch to go back to a dumb system" option is worth maintaining
- thinkharderdev 6y ago
  Is it really? You would maintain an entire parallel stack built on an entirely separate infrastructure stack, with completely different deployment patterns and all the data synced? DR is really hard if you just want to fail over to another AWS/Azure/GCP region. I can't imagine what a nightmare it would be to maintain an on-prem DR standby. To mitigate a couple hours per year (in a really bad year) of cloud downtime?
  - mhh__ 6y ago
    Well it depends on the system, doesn't it? For filing taxes, it doesn't matter that much if there's downtime as long as you have a plan to move the demand around (i.e. don't fine people when it's down), but for 911 calls you're going to have a lot of fairly niche infrastructure that you need to plumb in any way, so the actual cloud parts should be comparatively easy to replace.
    - thinkharderdev 6y ago
      For those sorts of systems, my bet is that the DR strategy is to fall back to what they did before computers existed at all. And honestly that may be the best option. As long as you have pencils, paper and telephones then you can probably approximate what you were doing in 1995 anyway.
6y ago

nine_k 6y ago

I'd say that you need critical infrastructure redundantly deployed everywhere, and experiencing a training outage on one of the platform each quarter. But wanting is one thing, having the money to implement and maintain such a solution, a quite another thing :-/

deleted 6y ago

[deleted]

908B64B197 6y ago

They would need to triple the salaries for new grads to make it on par with Microsoft. Junior employees over there make around 45K USD[0], compare to Microsoft 110K[1]. Not including bonuses and stock, of course. I don't think they could build infrastructure even if they wanted to. [0] https://www.tbs-sct.gc.ca/agreements-conventions/view-visualiser-eng.aspx?id=1#tocxx312259 https://www.tbs-sct.gc.ca/agreements-conventions/view-visual... [1] https://www.levels.fyi/company/Microsoft/salaries/Software-Engineer/ https://www.levels.fyi/company/Microsoft/salaries/Software-E...

deleted 6y ago

[deleted]