5 ms·

The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker contain

The cloud providers can afford to hire and train elite teams to handle security. I remember seeing a post about a guy trying to break out of the docker container used by Cloud SQL on GCP, and apparently the GCP admins made it known that he was being watched pretty early on. I believe the issue was patched fairly quickly too. It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.

brundolf 6y ago
Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt
- belval 6y ago
  I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
  - kaliszad 6y ago
    Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
    - jacquesm 6y ago
      Clearly the latter, but unfortunately the economic incentives favor centralization and that in turn pretty much nixes the chances of significant resources being allocated to create those solutions. In fact, there are significant incentives to keep such solutions out of the marketplace entirely. It's quite funny in a way: regular mail worked for two hundred or so years without too much in terms of trouble, ok, we had some spam but that was about it. And now mail delivery has become so complicated that the mere act of accepting mail can lead to your corporate secrets being made public or lifted without your knowledge.
      - lotsofpulp 6y ago
        Physical mail could easily be stolen though. But you can still send it if you want that kind of security. There’s trade offs galore, but obviously the cheapness and convenience of email seems to have won out versus security concerns.
        winkeltripel 6y ago
        But its like with electoral ballots. The paper mail is hard to copy without the carrier finding out, and has to be intercepted by a local. Email cam be copied and sent wherever without the operators knowledge, from anywhere with internet, if they break into the mail daemon.
        asdff 6y ago
        The path of the vector of attach changes but it doesn't go away. Corp B could just bribe an overworked and underpaid mail room worker at Corp A to make some copies of sensitive looking info before they deliver it upstairs. Even today, who is to say that this doesn't happen with some overworked and underpaid sysadmins? or secretaries with their bosses email account password? I wouldn't be surprised if bribes for emails happen pretty frequently in the business world even today.
        jacquesm 6y ago
        Yes, but in actual fact the chances are much bigger that that person turns out to be ethical and will report the attempted breach. In the digital world there is no such sentry.
      - kaliszad 6y ago
        Actually, you see Google hiring ever more people. My colleague has worked for Google and said on multiple occasions (it is even on video) you can barely get anything done using the systems and procedures they have in place. The infrastructure/ tools for anything they have programmed there was very high maintenance and almost begins to crumble under its own weight. It is almost like an oil tanker dragging its anchor - it is still rolling, but it sure isn't a sustainable development if what you want is efficient movement forward.
    - tablespoon 6y ago
      > Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution Also, at some point the cloud provider may figure out that they can increase profitability by hiring more and more below-average people and just market them as world-class.
      - acdha 6y ago
        This is true but every IT outsourcing company is decades ahead of them. Doing that will hurt the cloud provider’s business while most other F500 CIOs will get a bonus for saving money as long as they leave before an incident too blatant to blame on anything else.
        tablespoon 6y ago
        Those outsourcing companies have a reputation for providing cheap low skill workers. What I described is a situation of basically converting reputation into cash. Once you're known for having "armies of above-average developers" and then cut back on employee quality, it's going to take a long time for the market to figure it out (and you can probably extend that time significantly with slick marketing). In the mean time, you profit margins are increased.
        cmckn 6y ago
        Interesting idea, but I don’t think that’s a gamble any of the current clouds would make. The business is by nature a long game, so a few years of increased profit followed by <danger> doesn’t seem like it’d be enticing to a cloud. Besides, it’s not really true today that clouds only employ “above average” developers. I mean hell, they employed me!
        acdha 6y ago
        Again, I’m not saying it’s impossible — only that, say, AWS or GCP have a lot more at risk cutting into a core market pitch and failing to deliver a promised service – and that’s entirely on them to deliver. In part, this is the different service model: if I go to AWS and buy, say, S3 they have a very clear responsibility not to lose your data and to serve it quickly. If my CIO picks one of the bargain basement outsourcers and the centralized storage service fails badly, each different group will be saying that the failure wasn’t due to them but the company management, outsourced project management, the contractors who set it up/operate/monitor/secure, vendor products, vendor professional staff, Microsoft, etc. Since truckloads of cash will have been spent by then, many of those parties only care if it’ll reach the point of a lawsuit and everyone in the approval chain who didn’t say it was troubled before has an incentive to say the failure was unforeseeable and the solution is not to hold anyone accountable.
    - ckozlowski 6y ago
      "Simpler" is often at odds with "more features" here though. And while I anticipate the "I only need a handful of features" argument, great swaths of users feel differently. When you proceed to the logical end of enforcing simplicity to achieve security, you get OpenBSD. That's great for certain applications, but I think we can agree it doesn't check a lot of boxes for contemporary feature set demands. My point being, achieving that is way harder than it sounds.
      - kaliszad 6y ago
        That is exactly my point. You should be quite picky about the tools you rely on as a business or individual. Most users in one company, if you ask them, are fine using about 10 features with a great overlap, so maybe 20 features overall. That will have very large overlap with other companies in the industry as well. You have a few employees, that use like 50 features all the time. You know them by name as a sysadmin usually :-) and you tend to those people often and maybe even become friends. Those people will do just fine using a very professional/ complicated tool because they will invest time to learn using it. The other people would be overwhelmed. The solution is to use different tools for those two groups. Speaking of OpenBSD, that might actually be a better OS for most stuff on the shop floor in companies that I have seen from the inside, where Windows is used almost exclusively. The plus being, nobody can really mess around with it. There is usually exactly one app that needs to run 24/7/365 with occasional opportunity to update e.g. during a maintenance window and that's it, anything that causes the app to close is lost time on the shop floor. OpenBSD being minimal is a large plus here.
    - CuriousNinja 6y ago
      In an ideal world the latter would be best. However in practice systems tend to get complicated over time as they evolve and more features etc.. are added. I think one way to look at it is that as a business to outsourcing the non-differentiated heavy lifting to another entity that has more expertise in it would let you focus on your core products. In this specific example why is using a cloud email provider any different from deciding to use power from the power grid instead of generating your own electricity.
      - kaliszad 6y ago
        In a healthy company, I don't think you can outsource any work on the core product without making the end result worse. Even things like translations need lots of back and forth and the feedback loop needs to be really tight. It has to be clear to the user in the culture what you want to say and the things are very subtle at times, like the eye icon in the password field to show the password. In some cultures, that might be associated with something creepy or negative maybe, but you just don't know. Show or hide may be twice as long as in the languages you know which breaks your assumptions and makes the slick UI less slick or the error messages less clear in that particular setting.
        bearbawl 6y ago
        Most IT services should definitely be outsourced. Let’s say you are a big airline company, there is absolutely not a single reason you should manage your email system. Your job is to fly airplanes not to manage some goddam emails. The really fun part in that is that most of the big airlines actually outsourced some key part of their core job (IT wise I mean), like how they manage seats and load, this kind of stuff, while keeping some absolute non-core IT services internal, like an internal Exchange system with dozens and dozens of people to manage it.
    - bearbawl 6y ago
      The first one, 100%. The state where big companies are make the second option impossible. That may be unfortunate, I don’t know, but that’s really where we’re at. There is absolutely no way to cure big companies from all the shit they have accumulated. For them, the actual restart is to go to Cloud. Hopefully they will not go simply bare metal, because then they can recreate the exact same shit but in the Cloud.
  - brundolf 6y ago
    Yes, that's how things are now. My point was that things shouldn't be that way, and I see no reason why they always had to be that way, though at the same time that's where we are and there isn't a clear path out of it.
- g_p 6y ago
  You're right, but I fear we have built up an ecosystem of talent that sees the world in one of two very binary ways. One camp assumes if you don't expose it to the internet, and keep it on-prem, it's secure. Think exchange server on-prem (but let's overlook the gaping internet exposed parts - they don't see those, they see the fact it runs in their office). On the other hand, it's public cloud, hosted service, rely on a big company with the resources (but accept loss of tenant isolation when something big goes badly wrong, and hope the cloud host has the skills to mitigate and detect issues). We need more secure systems, but if they're publicly exposed then you'll require that team of experts around the clock simply to detect the potential of a compromise. Something I see a lot of confusion around is knowing when something is compromised. Responding is then "easy" in comparison for them, but they don't know what they should be looking for. With complex exposed services (mixed user and management plane over HTTPS, email interfaces for multiple protocols with different versions and authentication mechanisms), the likelihood of serious comprise tends towards 1. Better hardening services would help to get some way towards the world you describe, but that has to filter through the whole supply chain and ecosystem - no, you shouldn't be able to manage the exchange server from outside, nor should any such interfaces be exposed. No, the exchange service shouldn't execute aspx code from folders on the local filesystem that can be modified other than through a privileged updater service.
- TheDudeMan 6y ago
  Yes, what we need is reasonably secure systems. But what we pay for is features.
  - rorykoehler 6y ago
    Eventually security becomes a feature
pmlnr 6y ago
Riiiight, because cloud sw can't have 0 days.
- sangnoir 6y ago
  The argument is cloud sw have better chances of remediating their 0-days compared to countless sys-admins/help-desk-admins
LilBytes 6y ago
I remember reading this. When the Dev did it a second time there was a txt file on the host (container? Can't remember) saying "Hey this is cool, we're about to patch this, thanks for letting us know".
theobeers 6y ago
Yeah, here's the blog post you're thinking of, from August 2020: https://offensi.com/2020/08/18/how-to-contact-google-sre-dropping-a-shell-in-cloud-sql/ https://offensi.com/2020/08/18/how-to-contact-google-sre-dro... And the HN thread: https://news.ycombinator.com/item?id=24216009 https://news.ycombinator.com/item?id=24216009

brundolf 6y ago

Really what we need is the ability to self-host reasonably secure systems without a team of experts working round the clock... but that doesn't appear to be the hand we've been dealt

belval 6y ago
I might be biased because I work at AWS, but I really doubt that there are enough sys admins that know what they are doing and keep up to date let alone find vulnerabilities in the software they use to protect all companies. A Fortune 500 maybe, but at some point you simply can't afford someone who knows what he's doing and at that point you might as well have everything in the cloud so you can focus on your actual money making business.
- kaliszad 6y ago
  Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution to actually sit down and make simpler systems that a few skilled people can fit in their heads and actually understand?
  - jacquesm 6y ago
    Clearly the latter, but unfortunately the economic incentives favor centralization and that in turn pretty much nixes the chances of significant resources being allocated to create those solutions. In fact, there are significant incentives to keep such solutions out of the marketplace entirely. It's quite funny in a way: regular mail worked for two hundred or so years without too much in terms of trouble, ok, we had some spam but that was about it. And now mail delivery has become so complicated that the mere act of accepting mail can lead to your corporate secrets being made public or lifted without your knowledge.
    - lotsofpulp 6y ago
      Physical mail could easily be stolen though. But you can still send it if you want that kind of security. There’s trade offs galore, but obviously the cheapness and convenience of email seems to have won out versus security concerns.
      - winkeltripel 6y ago
        But its like with electoral ballots. The paper mail is hard to copy without the carrier finding out, and has to be intercepted by a local. Email cam be copied and sent wherever without the operators knowledge, from anywhere with internet, if they break into the mail daemon.
        asdff 6y ago
        The path of the vector of attach changes but it doesn't go away. Corp B could just bribe an overworked and underpaid mail room worker at Corp A to make some copies of sensitive looking info before they deliver it upstairs. Even today, who is to say that this doesn't happen with some overworked and underpaid sysadmins? or secretaries with their bosses email account password? I wouldn't be surprised if bribes for emails happen pretty frequently in the business world even today.
        jacquesm 6y ago
        Yes, but in actual fact the chances are much bigger that that person turns out to be ethical and will report the attempted breach. In the digital world there is no such sentry.
    - kaliszad 6y ago
      Actually, you see Google hiring ever more people. My colleague has worked for Google and said on multiple occasions (it is even on video) you can barely get anything done using the systems and procedures they have in place. The infrastructure/ tools for anything they have programmed there was very high maintenance and almost begins to crumble under its own weight. It is almost like an oil tanker dragging its anchor - it is still rolling, but it sure isn't a sustainable development if what you want is efficient movement forward.
  - tablespoon 6y ago
    > Is the "cloud" with armies of above-average developers, SREs/ sysadmins/ systems engineers and security specialists really the solution or is the solution Also, at some point the cloud provider may figure out that they can increase profitability by hiring more and more below-average people and just market them as world-class.
    - acdha 6y ago
      This is true but every IT outsourcing company is decades ahead of them. Doing that will hurt the cloud provider’s business while most other F500 CIOs will get a bonus for saving money as long as they leave before an incident too blatant to blame on anything else.
      - tablespoon 6y ago
        Those outsourcing companies have a reputation for providing cheap low skill workers. What I described is a situation of basically converting reputation into cash. Once you're known for having "armies of above-average developers" and then cut back on employee quality, it's going to take a long time for the market to figure it out (and you can probably extend that time significantly with slick marketing). In the mean time, you profit margins are increased.
        cmckn 6y ago
        Interesting idea, but I don’t think that’s a gamble any of the current clouds would make. The business is by nature a long game, so a few years of increased profit followed by <danger> doesn’t seem like it’d be enticing to a cloud. Besides, it’s not really true today that clouds only employ “above average” developers. I mean hell, they employed me!
        acdha 6y ago
        Again, I’m not saying it’s impossible — only that, say, AWS or GCP have a lot more at risk cutting into a core market pitch and failing to deliver a promised service – and that’s entirely on them to deliver. In part, this is the different service model: if I go to AWS and buy, say, S3 they have a very clear responsibility not to lose your data and to serve it quickly. If my CIO picks one of the bargain basement outsourcers and the centralized storage service fails badly, each different group will be saying that the failure wasn’t due to them but the company management, outsourced project management, the contractors who set it up/operate/monitor/secure, vendor products, vendor professional staff, Microsoft, etc. Since truckloads of cash will have been spent by then, many of those parties only care if it’ll reach the point of a lawsuit and everyone in the approval chain who didn’t say it was troubled before has an incentive to say the failure was unforeseeable and the solution is not to hold anyone accountable.
  - ckozlowski 6y ago
    "Simpler" is often at odds with "more features" here though. And while I anticipate the "I only need a handful of features" argument, great swaths of users feel differently. When you proceed to the logical end of enforcing simplicity to achieve security, you get OpenBSD. That's great for certain applications, but I think we can agree it doesn't check a lot of boxes for contemporary feature set demands. My point being, achieving that is way harder than it sounds.
    - kaliszad 6y ago
      That is exactly my point. You should be quite picky about the tools you rely on as a business or individual. Most users in one company, if you ask them, are fine using about 10 features with a great overlap, so maybe 20 features overall. That will have very large overlap with other companies in the industry as well. You have a few employees, that use like 50 features all the time. You know them by name as a sysadmin usually :-) and you tend to those people often and maybe even become friends. Those people will do just fine using a very professional/ complicated tool because they will invest time to learn using it. The other people would be overwhelmed. The solution is to use different tools for those two groups. Speaking of OpenBSD, that might actually be a better OS for most stuff on the shop floor in companies that I have seen from the inside, where Windows is used almost exclusively. The plus being, nobody can really mess around with it. There is usually exactly one app that needs to run 24/7/365 with occasional opportunity to update e.g. during a maintenance window and that's it, anything that causes the app to close is lost time on the shop floor. OpenBSD being minimal is a large plus here.
  - CuriousNinja 6y ago
    In an ideal world the latter would be best. However in practice systems tend to get complicated over time as they evolve and more features etc.. are added. I think one way to look at it is that as a business to outsourcing the non-differentiated heavy lifting to another entity that has more expertise in it would let you focus on your core products. In this specific example why is using a cloud email provider any different from deciding to use power from the power grid instead of generating your own electricity.
    - kaliszad 6y ago
      In a healthy company, I don't think you can outsource any work on the core product without making the end result worse. Even things like translations need lots of back and forth and the feedback loop needs to be really tight. It has to be clear to the user in the culture what you want to say and the things are very subtle at times, like the eye icon in the password field to show the password. In some cultures, that might be associated with something creepy or negative maybe, but you just don't know. Show or hide may be twice as long as in the languages you know which breaks your assumptions and makes the slick UI less slick or the error messages less clear in that particular setting.
      - bearbawl 6y ago
        Most IT services should definitely be outsourced. Let’s say you are a big airline company, there is absolutely not a single reason you should manage your email system. Your job is to fly airplanes not to manage some goddam emails. The really fun part in that is that most of the big airlines actually outsourced some key part of their core job (IT wise I mean), like how they manage seats and load, this kind of stuff, while keeping some absolute non-core IT services internal, like an internal Exchange system with dozens and dozens of people to manage it.
  - bearbawl 6y ago
    The first one, 100%. The state where big companies are make the second option impossible. That may be unfortunate, I don’t know, but that’s really where we’re at. There is absolutely no way to cure big companies from all the shit they have accumulated. For them, the actual restart is to go to Cloud. Hopefully they will not go simply bare metal, because then they can recreate the exact same shit but in the Cloud.
- brundolf 6y ago
  Yes, that's how things are now. My point was that things shouldn't be that way, and I see no reason why they always had to be that way, though at the same time that's where we are and there isn't a clear path out of it.
g_p 6y ago
You're right, but I fear we have built up an ecosystem of talent that sees the world in one of two very binary ways. One camp assumes if you don't expose it to the internet, and keep it on-prem, it's secure. Think exchange server on-prem (but let's overlook the gaping internet exposed parts - they don't see those, they see the fact it runs in their office). On the other hand, it's public cloud, hosted service, rely on a big company with the resources (but accept loss of tenant isolation when something big goes badly wrong, and hope the cloud host has the skills to mitigate and detect issues). We need more secure systems, but if they're publicly exposed then you'll require that team of experts around the clock simply to detect the potential of a compromise. Something I see a lot of confusion around is knowing when something is compromised. Responding is then "easy" in comparison for them, but they don't know what they should be looking for. With complex exposed services (mixed user and management plane over HTTPS, email interfaces for multiple protocols with different versions and authentication mechanisms), the likelihood of serious comprise tends towards 1. Better hardening services would help to get some way towards the world you describe, but that has to filter through the whole supply chain and ecosystem - no, you shouldn't be able to manage the exchange server from outside, nor should any such interfaces be exposed. No, the exchange service shouldn't execute aspx code from folders on the local filesystem that can be modified other than through a privileged updater service.
TheDudeMan 6y ago
Yes, what we need is reasonably secure systems. But what we pay for is features.
- rorykoehler 6y ago
  Eventually security becomes a feature

pmlnr 6y ago

Riiiight, because cloud sw can't have 0 days.

sangnoir 6y ago
The argument is cloud sw have better chances of remediating their 0-days compared to countless sys-admins/help-desk-admins

LilBytes 6y ago

I remember reading this. When the Dev did it a second time there was a txt file on the host (container? Can't remember) saying "Hey this is cool, we're about to patch this, thanks for letting us know".

theobeers 6y ago

Yeah, here's the blog post you're thinking of, from August 2020: https://offensi.com/2020/08/18/how-to-contact-google-sre-dropping-a-shell-in-cloud-sql/ https://offensi.com/2020/08/18/how-to-contact-google-sre-dro... And the HN thread: https://news.ycombinator.com/item?id=24216009 https://news.ycombinator.com/item?id=24216009