3 ms·

The cloud angle is interesting; on one hand, it creates an even-more-centralized single point of failure. On the other hand, given that virtually every computin

The cloud angle is interesting; on one hand, it creates an even-more-centralized single point of failure. On the other hand, given that virtually every computing system out there is a house of cards, letting the experts focus on securing (and updating!) just a single one might be the best defense.

mywittyname 6y ago
The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.
- brundolf 6y ago
  Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
  - belval 6y ago
    I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
    - kaliszad 6y ago
      Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
      - jacquesm 6y ago
        Clearly the latter, but unfortunately the economic incentives favor centralization and that in turn pretty much nixes the chances of significant resources being allocated to create those solutions. In fact, there are significant incentives to keep such solutions out of the marketplace entirely. It's quite funny in a way: regular mail worked for two hundred or so years without too much in terms of trouble, ok, we had some spam but that was about it. And now mail delivery has become so complicated that the mere act of accepting mail can lead to your corporate secrets being made public or lifted without your knowledge.
        lotsofpulp 6y ago
        Physical mail could easily be stolen though. But you can still send it if you want that kind of security. There’s trade offs galore, but obviously the cheapness and convenience of email seems to have won out versus security concerns.
        winkeltripel 6y ago
        But its like with electoral ballots. The paper mail is hard to copy without the carrier finding out, and has to be intercepted by a local. Email cam be copied and sent wherever without the operators knowledge, from anywhere with internet, if they break into the mail daemon.
        asdff 6y ago
        The path of the vector of attach changes but it doesn't go away. Corp B could just bribe an overworked and underpaid mail room worker at Corp A to make some copies of sensitive looking info before they deliver it upstairs. Even today, who is to say that this doesn't happen with some overworked and underpaid sysadmins? or secretaries with their bosses email account password? I wouldn't be surprised if bribes for emails happen pretty frequently in the business world even today.
        6y ago
    - brundolf 6y ago
      Yes, that's how things are now. My point was that things shouldn't be that way, and I see no reason why they always had to be that way, though at the same time that's where we are and there isn't a clear path out of it.
  - g_p 6y ago
    You're right, but I fear we have built up an ecosystem of talent that sees the world in one of two very binary ways. One camp assumes if you don't expose it to the internet, and keep it on-prem, it's secure. Think exchange server on-prem (but let's overlook the gaping internet exposed parts - they don't see those, they see the fact it runs in their office). On the other hand, it's public cloud, hosted service, rely on a big company with the resources (but accept loss of tenant isolation when something big goes badly wrong, and hope the cloud host has the skills to mitigate and detect issues). We need more secure systems, but if they're publicly exposed then you'll require that team of experts around the clock simply to detect the potential of a compromise. Something I see a lot of confusion around is knowing when something is compromised. Responding is then "easy" in comparison for them, but they don't know what they should be looking for. With complex exposed services (mixed user and management plane over HTTPS, email interfaces for multiple protocols with different versions and authentication mechanisms), the likelihood of serious comprise tends towards 1. Better hardening services would help to get some way towards the world you describe, but that has to filter through the whole supply chain and ecosystem - no, you shouldn't be able to manage the exchange server from outside, nor should any such interfaces be exposed. No, the exchange service shouldn't execute aspx code from folders on the local filesystem that can be modified other than through a privileged updater service.
  - TheDudeMan 6y ago
    Yes, what we need is reasonably secure systems. But what we pay for is features.
    - rorykoehler 6y ago
      Eventually security becomes a feature
- pmlnr 6y ago
  Riiiight, because cloud sw can't have 0 days.
  - sangnoir 6y ago
    The argument is cloud sw have better chances of remediating their 0-days compared to countless sys-admins/help-desk-admins
- LilBytes 6y ago
  I remember reading this. When the Dev did it a second time there was a txt file on the host (container? Can't remember) saying "Hey this is cool, we're about to patch this, thanks for letting us know".
- theobeers 6y ago
  Yeah, here's the blog post you're thinking of, from August 2020: https://offensi.com/2020/08/18/how-to-contact-google-sre-dropping-a-shell-in-cloud-sql/ https://offensi.com/2020/08/18/how-to-contact-google-sre-dro... And the HN thread: https://news.ycombinator.com/item?id=24216009 https://news.ycombinator.com/item?id=24216009
u678u 6y ago
I think this was the conclusion from the Sony hack (2014- wow nearly 7 years already). People were scared of cloud security but Sony showed that on prem isn't any better.
Veserv 6y ago
That is only accurate if they can provide a meaningful defense against expected attacks otherwise all you are doing is creating a single central target. Unfortunately, the cloud providers can not mount even a token defense against an attacker funding an attack at the $100M level, so I see no reason to assume they can defend against credible threats to a single Fortune 500 company given that they can not even stop an attack with such a meager amount of resources allocated to it relative to the size of a Fortune 500 company. That is not to say that the teams in a Fortune 500 company are any better, merely that everybody is completely inadequate. By consolidating targets when you can not even reach the level to protect a single one you are making the situation worse, not better by consolidating. For it to make any real amount of sense they would first need to demonstrate an ability to prevent attacks at least in the correct order of magnitude and then demonstrate that they can scale up without creating correlated risk. Only then does it make any sense to actually centralize on a single solution, let alone a single provider.
- brundolf 6y ago
  I mean... they prevented this one in the cloud version. I'm not advocating for a single provider, and I'm not necessarily advocating for cloud hosting as a solution, I'm just pointing out that in this case the cloud fared better than practically all of the self-hosted systems
  - jlgaddis 6y ago
    The "cloud version" is an entirely different piece of software. It's not the same OWA that one hosts on-premises. That one's still vulnerable even if it's hosted "in the cloud". On a different note, if they could prevent this in "the cloud version", why couldn't they -- why didn't they -- prevent it in "the non-cloud version"?
    - Godel_unicode 6y ago
      You already said the answer to your own question; it's different and therefore it's not the same.
  - 6y ago
koolba 6y ago
Cloud providers are also more likely to have true off site backups in place. Your vanilla SMB running an exchange server on a pc in the closet doesn’t.
perl4ever 6y ago
“Put all your eggs in one basket and then watch that basket.” - Andrew Carnegie
collsni 6y ago
imagine if aws, gcp, or azure went down for one day.
- tonyedgecombe 6y ago
  Or a week ...
- sofixa 6y ago
  That could never happen baring an unlikely critical even such as a nuclear world war. Any of them being fully down is nearly impossible - remember, we're talking about autonomous regions across hundreds of DCs across the globe.

mywittyname 6y ago

The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.

brundolf 6y ago
Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
- belval 6y ago
  I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
  - kaliszad 6y ago
    Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
    - jacquesm 6y ago
      Clearly the latter, but unfortunately the economic incentives favor centralization and that in turn pretty much nixes the chances of significant resources being allocated to create those solutions. In fact, there are significant incentives to keep such solutions out of the marketplace entirely. It's quite funny in a way: regular mail worked for two hundred or so years without too much in terms of trouble, ok, we had some spam but that was about it. And now mail delivery has become so complicated that the mere act of accepting mail can lead to your corporate secrets being made public or lifted without your knowledge.
      - lotsofpulp 6y ago
        Physical mail could easily be stolen though. But you can still send it if you want that kind of security. There’s trade offs galore, but obviously the cheapness and convenience of email seems to have won out versus security concerns.
        winkeltripel 6y ago
        But its like with electoral ballots. The paper mail is hard to copy without the carrier finding out, and has to be intercepted by a local. Email cam be copied and sent wherever without the operators knowledge, from anywhere with internet, if they break into the mail daemon.
        asdff 6y ago
        The path of the vector of attach changes but it doesn't go away. Corp B could just bribe an overworked and underpaid mail room worker at Corp A to make some copies of sensitive looking info before they deliver it upstairs. Even today, who is to say that this doesn't happen with some overworked and underpaid sysadmins? or secretaries with their bosses email account password? I wouldn't be surprised if bribes for emails happen pretty frequently in the business world even today.
        6y ago
  - brundolf 6y ago
    Yes, that's how things are now. My point was that things shouldn't be that way, and I see no reason why they always had to be that way, though at the same time that's where we are and there isn't a clear path out of it.
- g_p 6y ago
  You're right, but I fear we have built up an ecosystem of talent that sees the world in one of two very binary ways. One camp assumes if you don't expose it to the internet, and keep it on-prem, it's secure. Think exchange server on-prem (but let's overlook the gaping internet exposed parts - they don't see those, they see the fact it runs in their office). On the other hand, it's public cloud, hosted service, rely on a big company with the resources (but accept loss of tenant isolation when something big goes badly wrong, and hope the cloud host has the skills to mitigate and detect issues). We need more secure systems, but if they're publicly exposed then you'll require that team of experts around the clock simply to detect the potential of a compromise. Something I see a lot of confusion around is knowing when something is compromised. Responding is then "easy" in comparison for them, but they don't know what they should be looking for. With complex exposed services (mixed user and management plane over HTTPS, email interfaces for multiple protocols with different versions and authentication mechanisms), the likelihood of serious comprise tends towards 1. Better hardening services would help to get some way towards the world you describe, but that has to filter through the whole supply chain and ecosystem - no, you shouldn't be able to manage the exchange server from outside, nor should any such interfaces be exposed. No, the exchange service shouldn't execute aspx code from folders on the local filesystem that can be modified other than through a privileged updater service.
- TheDudeMan 6y ago
  Yes, what we need is reasonably secure systems. But what we pay for is features.
  - rorykoehler 6y ago
    Eventually security becomes a feature
pmlnr 6y ago
Riiiight, because cloud sw can't have 0 days.
- sangnoir 6y ago
  The argument is cloud sw have better chances of remediating their 0-days compared to countless sys-admins/help-desk-admins
LilBytes 6y ago
I remember reading this. When the Dev did it a second time there was a txt file on the host (container? Can't remember) saying "Hey this is cool, we're about to patch this, thanks for letting us know".
theobeers 6y ago
Yeah, here's the blog post you're thinking of, from August 2020: https://offensi.com/2020/08/18/how-to-contact-google-sre-dropping-a-shell-in-cloud-sql/ https://offensi.com/2020/08/18/how-to-contact-google-sre-dro... And the HN thread: https://news.ycombinator.com/item?id=24216009 https://news.ycombinator.com/item?id=24216009

u678u 6y ago

I think this was the conclusion from the Sony hack (2014- wow nearly 7 years already). People were scared of cloud security but Sony showed that on prem isn't any better.

Veserv 6y ago

That is only accurate if they can provide a meaningful defense against expected attacks otherwise all you are doing is creating a single central target. Unfortunately, the cloud providers can not mount even a token defense against an attacker funding an attack at the $100M level, so I see no reason to assume they can defend against credible threats to a single Fortune 500 company given that they can not even stop an attack with such a meager amount of resources allocated to it relative to the size of a Fortune 500 company. That is not to say that the teams in a Fortune 500 company are any better, merely that everybody is completely inadequate. By consolidating targets when you can not even reach the level to protect a single one you are making the situation worse, not better by consolidating. For it to make any real amount of sense they would first need to demonstrate an ability to prevent attacks at least in the correct order of magnitude and then demonstrate that they can scale up without creating correlated risk. Only then does it make any sense to actually centralize on a single solution, let alone a single provider.

brundolf 6y ago
I mean... they prevented this one in the cloud version. I'm not advocating for a single provider, and I'm not necessarily advocating for cloud hosting as a solution, I'm just pointing out that in this case the cloud fared better than practically all of the self-hosted systems
- jlgaddis 6y ago
  The "cloud version" is an entirely different piece of software. It's not the same OWA that one hosts on-premises. That one's still vulnerable even if it's hosted "in the cloud". On a different note, if they could prevent this in "the cloud version", why couldn't they -- why didn't they -- prevent it in "the non-cloud version"?
  - Godel_unicode 6y ago
    You already said the answer to your own question; it's different and therefore it's not the same.
- 6y ago

koolba 6y ago

Cloud providers are also more likely to have true off site backups in place. Your vanilla SMB running an exchange server on a pc in the closet doesn’t.

perl4ever 6y ago

“Put all your eggs in one basket and then watch that basket.” - Andrew Carnegie

collsni 6y ago

imagine if aws, gcp, or azure went down for one day.

tonyedgecombe 6y ago
Or a week ...
sofixa 6y ago
That could never happen baring an unlikely critical even such as a nuclear world war. Any of them being fully down is nearly impossible - remember, we're talking about autonomous regions across hundreds of DCs across the globe.