6 ms·

That must've been such an exciting moment to have someone say hello. Chilling too.

londons_explore 6y ago
More importantly, it means those same employees probably have unaudited access to the data in your cloud SQL database. If SRE can write a file to the filesystem, they can totally copy the database files out too. I completely support SRE being able to log into instances for debugging, but them being able to do that without leaving an audit trail visible to me, the data owner, isn't up to modern standards IMO.
- londons_explore 6y ago
  Cloud SRE should have a button to click to grant themselves access to an instance. That button should create for them a fully logged SSH connection to the VM, and the data owner should be notified that SRE has connected. The data owner should then be able to see the commands executed via the SSH connection, and also request an explanation of what the SRE was doing. The vast majority of instances are going to be things like "your instance hit a SIGSEGV, we logged in to grab the crash dump" or "your instance had a much higher than usage than all other users. We logged in to run tools to find where is was going".
  - zaarn 6y ago
    That would be acceptable since I could immediately close my account once SRE had connected due to the data breach. To be more serious, it would be unacceptable for me that an employee can, for whatever reason, access my instance without my express permission and oversight. Even crash dumps could contain very sensitive information.
    - pgo 6y ago
      Well its called managed infrastructure for a reason.
      - zaarn 6y ago
        Even in managed infrastructure, taking data out of the infrastructure is unacceptable.
        londons_explore 6y ago
        I don't think there is a good solution yet for products like cloud SQL to make it theoretically impossible for an evil hosting company to steal your data. But it should be possible for a client to audit all accesses, and the audit logging system be robust enough to catch evil employees.
        icedchai 6y ago
        Who audits the audit system? If I'm an evil employee I'll make sure my requests don't show up there.
        joshuamorton 6y ago
        So a company goes to the effort of creating an audit system, but gives everyone the ability to modify the history? If your threat model is a single rogue employee, auditing systems address the issues. If your threat model is "the whole company works against you", you probably shouldn't be using any company to host your infra, so host it yourself, because there's no way to be secure in that situation.
        zaarn 6y ago
        I doesn't have to be impossible to access but the hoster shouldn't just log into my system and download arbitrary data from it. I'm not asking for perfect confidentiality, just contractual confidentiality.
    - icedchai 6y ago
      Sounds like you should operate your own infrastructure.
      - john-shaffer 6y ago
        He could just use AWS.
        icedchai 6y ago
        How do we know the same thing can't happen there, with respect to employees accessing data, etc? My point is unless you manage it yourself, you don't know.
- delroth 6y ago
  As far as I can tell, there is nothing in the writeup that suggests the file wasn't there from the beginning and present on all Cloud SQL instances. I know this was a standard practice in a few of the products I worked on where we expected the first few layers of security/obfuscation would be peeled off by curious outsiders. This seems much more likely than OP's exploration being detected in realtime, then someone SSHing into the container manually to put a message. Disclaimer: I work at Google, not on Cloud SQL, and don't know anything specific about that greetings.txt.
  - biddlesby 6y ago
    > Not long after we started exploring the environment we landed our shell in we noticed a new file in the /mysql/tmp directory named ‘greetings.txt’ I interpreted the "new" as meaning that it wasn't there before
  - speckler 6y ago
    I work on Cloud SQL, we definitely don't add a file like that.
    - wtm_offensi 6y ago
      It could be the case that SRE recognized our accounts since we are hunting on a daily basis. In rare cases SRE's reach out to ISE's who contact us on their behalf. In this case a file was added. I personally don't see any reason why this would be a bad thing, it's a managed service and we tried to attack it after all. No data was removed, nor is there any indication that there are no audit trails. I hold both the ISE's and SRE's in extremely high regard. (If you try to Hunt for bugs in GCP you will find out why)

More importantly, it means those same employees probably have unaudited access to the data in your cloud SQL database. If SRE can write a file to the filesystem, they can totally copy the database files out too. I completely support SRE being able to log into instances for debugging, but them being able to do that without leaving an audit trail visible to me, the data owner, isn't up to modern standards IMO.

londons_explore 6y ago
Cloud SRE should have a button to click to grant themselves access to an instance. That button should create for them a fully logged SSH connection to the VM, and the data owner should be notified that SRE has connected. The data owner should then be able to see the commands executed via the SSH connection, and also request an explanation of what the SRE was doing. The vast majority of instances are going to be things like "your instance hit a SIGSEGV, we logged in to grab the crash dump" or "your instance had a much higher than usage than all other users. We logged in to run tools to find where is was going".
- zaarn 6y ago
  That would be acceptable since I could immediately close my account once SRE had connected due to the data breach. To be more serious, it would be unacceptable for me that an employee can, for whatever reason, access my instance without my express permission and oversight. Even crash dumps could contain very sensitive information.
  - pgo 6y ago
    Well its called managed infrastructure for a reason.
    - zaarn 6y ago
      Even in managed infrastructure, taking data out of the infrastructure is unacceptable.
      - londons_explore 6y ago
        I don't think there is a good solution yet for products like cloud SQL to make it theoretically impossible for an evil hosting company to steal your data. But it should be possible for a client to audit all accesses, and the audit logging system be robust enough to catch evil employees.
        icedchai 6y ago
        Who audits the audit system? If I'm an evil employee I'll make sure my requests don't show up there.
        joshuamorton 6y ago
        So a company goes to the effort of creating an audit system, but gives everyone the ability to modify the history? If your threat model is a single rogue employee, auditing systems address the issues. If your threat model is "the whole company works against you", you probably shouldn't be using any company to host your infra, so host it yourself, because there's no way to be secure in that situation.
        zaarn 6y ago
        I doesn't have to be impossible to access but the hoster shouldn't just log into my system and download arbitrary data from it. I'm not asking for perfect confidentiality, just contractual confidentiality.
  - icedchai 6y ago
    Sounds like you should operate your own infrastructure.
    - john-shaffer 6y ago
      He could just use AWS.
      - icedchai 6y ago
        How do we know the same thing can't happen there, with respect to employees accessing data, etc? My point is unless you manage it yourself, you don't know.
delroth 6y ago
As far as I can tell, there is nothing in the writeup that suggests the file wasn't there from the beginning and present on all Cloud SQL instances. I know this was a standard practice in a few of the products I worked on where we expected the first few layers of security/obfuscation would be peeled off by curious outsiders. This seems much more likely than OP's exploration being detected in realtime, then someone SSHing into the container manually to put a message. Disclaimer: I work at Google, not on Cloud SQL, and don't know anything specific about that greetings.txt.
- biddlesby 6y ago
  > Not long after we started exploring the environment we landed our shell in we noticed a new file in the /mysql/tmp directory named ‘greetings.txt’ I interpreted the "new" as meaning that it wasn't there before
- speckler 6y ago
  I work on Cloud SQL, we definitely don't add a file like that.
  - wtm_offensi 6y ago
    It could be the case that SRE recognized our accounts since we are hunting on a daily basis. In rare cases SRE's reach out to ISE's who contact us on their behalf. In this case a file was added. I personally don't see any reason why this would be a bad thing, it's a managed service and we tried to attack it after all. No data was removed, nor is there any indication that there are no audit trails. I hold both the ISE's and SRE's in extremely high regard. (If you try to Hunt for bugs in GCP you will find out why)