4 ms·

XXXX == what types of things? I’m curious why there was no auth required for his calls.

londons_explore 8y ago
* Grab nearly all of googles source code (no extra auth required for that, since so many libraries read config etc from the source code repo) * Make the right requests to one endpoint he found and retrieve company financials, number of hits to every google service, the name of every application running in every datacenter, etc. * With the above two things, you know the location of services and every RPC endpoint on them, and all access control configs. You can take your sweet time to audit the 10's of millions of lines of code to find vulnerabilities and get to attack as an authenticated (albeit low privilege) user. A lot of stuff is open to all authenticated internal users. * For example, you could take down any google service by quitting all the application servers at the same time by calling the right debugging RPC. You'd be caught obviously tho.
- eyeareque 8y ago
  Wow, that is quite significant. 36k is not a small bounty for an RCE, but I feel like this is more critical to Google than the highest Android payout, for which they pay up to 200k for: https://www.google.com/about/appsecurity/android-rewards/ https://www.google.com/about/appsecurity/android-rewards/
  - lathiat 8y ago
    Android is probably one of those markets that are more liquid than most for "black market" sources (as talked about elsewhere in the comments for this)
  - londons_explore 8y ago
    Android is wormable, and potentially not repairable by google. For example, with a decent remote android exploit, I could distribute a patched Google Play Services to all vulnerable handsets which disables updates and then listens to my own command and control infrastructure for further actions. I can now hold the phones hostage and extort google for money to regain control of them.
    - eyeareque 8y ago
      That would be pretty brutal and cause people to quit trusting android phones. But I think the same could be accomplished with the access he had, or worse, but would have taken a lot more work. He also would have needed to avoid detection too. His access sounds more troubling than the Aroura attacks they had years ago.
- puzzle 8y ago
  Production code has no unauthenticated access to source. In all the cases I know, there was a data push for dynamic configurations from source to production, usually adding access control, auditing, validation, etc. Static configurations would be baked into the container image. That said, it looks like in this case he would been able to turn on access to google3, through a special flag that is probably meant for internal GAE apps that work on source code. Presumably, the flag allows the app to use a GAE proxy that authenticates to Piper and provides a filesystem-like interface. It's not clear what would have happened at this point, because it is likely there exists a finer grained quota for source access than just all of GAE. These kinds of new RPC traffic would have been visible in Dapper and Census data, plus there are booby traps everywhere aka defense in depth.
  - girvo 8y ago
    Right, but he wasn’t doing this in a production environment, so I wonder if he could’ve got access to source still.. thoughts?
    - puzzle 8y ago
      At the end of the day, even a non-production GAE environment runs on Borg, talks to GFS and Bigtable, etc. They're not going to run the entire stack on bare metal. The environments will have different configurations, will be backed by different Borg jobs, hopefully will run under different Borg identities, etc. The articles about Piper and co. have made it clear that it ran on top of Bigtable first and then Spanner, i.e. in the production network, spread over ten locations. There's file level auditing and access control. It would be very dumb of them not to have intrusion and anomaly detection in place.
- bryandollery 8y ago
  Software architecture isn't like that. Nobody has a list of all the software running in every datacenter -- not in any large corporation anywhere in the world, and an API isn't going to change that. And, you can't quit all application servers at the same time. These are distributed, self-healing, and highly-redundant -- meaning that there are thousands of copies of each service, and the system will bring up new copies to replace the ones you kill, without loosing service (though you could, potentially, affect quality of service). Even the control mechanisms that allow all this magic to happen are run on the same platform, meaning that you can't affect the managers in this way either. The sort of things you can do are: affect billing and reporting, find sensitive company data, and, potentially, execute code remotely (though that will probably be in a container, and not have access to much else). Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. The code is huge, so downloading it will take forever, and then you'd have to read it. Finally, it's written in dozens of programming languages, some, like Golang, are unreadable to anyone but experts.
  - ponyfleisch 8y ago
    > Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. Google is using a monorepo.
z3t4 8y ago
Google API Auth/Access is extremely tedious, so I guess someone just didn't bother. Basically you have to get an token, format it, make a couple of http posts, and finally you will have a token to make an access-token-token ... Now image all the steps you would have to make to create a new type of access to some internal API, that should not have public access anyway. Probably saved six months work. And the libraries are probably hiding all the obscurity so no-one did notice until this guy started digging.
- eyeareque 8y ago
  Sounds plausible. Now I’m curious how much extra work this finding has caused teams at Google. There are probably many other similar insecure paths that need to be cleaned up.

* Grab nearly all of googles source code (no extra auth required for that, since so many libraries read config etc from the source code repo) * Make the right requests to one endpoint he found and retrieve company financials, number of hits to every google service, the name of every application running in every datacenter, etc. * With the above two things, you know the location of services and every RPC endpoint on them, and all access control configs. You can take your sweet time to audit the 10's of millions of lines of code to find vulnerabilities and get to attack as an authenticated (albeit low privilege) user. A lot of stuff is open to all authenticated internal users. * For example, you could take down any google service by quitting all the application servers at the same time by calling the right debugging RPC. You'd be caught obviously tho.

eyeareque 8y ago
Wow, that is quite significant. 36k is not a small bounty for an RCE, but I feel like this is more critical to Google than the highest Android payout, for which they pay up to 200k for: https://www.google.com/about/appsecurity/android-rewards/ https://www.google.com/about/appsecurity/android-rewards/
- lathiat 8y ago
  Android is probably one of those markets that are more liquid than most for "black market" sources (as talked about elsewhere in the comments for this)
- londons_explore 8y ago
  Android is wormable, and potentially not repairable by google. For example, with a decent remote android exploit, I could distribute a patched Google Play Services to all vulnerable handsets which disables updates and then listens to my own command and control infrastructure for further actions. I can now hold the phones hostage and extort google for money to regain control of them.
  - eyeareque 8y ago
    That would be pretty brutal and cause people to quit trusting android phones. But I think the same could be accomplished with the access he had, or worse, but would have taken a lot more work. He also would have needed to avoid detection too. His access sounds more troubling than the Aroura attacks they had years ago.
puzzle 8y ago
Production code has no unauthenticated access to source. In all the cases I know, there was a data push for dynamic configurations from source to production, usually adding access control, auditing, validation, etc. Static configurations would be baked into the container image. That said, it looks like in this case he would been able to turn on access to google3, through a special flag that is probably meant for internal GAE apps that work on source code. Presumably, the flag allows the app to use a GAE proxy that authenticates to Piper and provides a filesystem-like interface. It's not clear what would have happened at this point, because it is likely there exists a finer grained quota for source access than just all of GAE. These kinds of new RPC traffic would have been visible in Dapper and Census data, plus there are booby traps everywhere aka defense in depth.
- girvo 8y ago
  Right, but he wasn’t doing this in a production environment, so I wonder if he could’ve got access to source still.. thoughts?
  - puzzle 8y ago
    At the end of the day, even a non-production GAE environment runs on Borg, talks to GFS and Bigtable, etc. They're not going to run the entire stack on bare metal. The environments will have different configurations, will be backed by different Borg jobs, hopefully will run under different Borg identities, etc. The articles about Piper and co. have made it clear that it ran on top of Bigtable first and then Spanner, i.e. in the production network, spread over ten locations. There's file level auditing and access control. It would be very dumb of them not to have intrusion and anomaly detection in place.
bryandollery 8y ago
Software architecture isn't like that. Nobody has a list of all the software running in every datacenter -- not in any large corporation anywhere in the world, and an API isn't going to change that. And, you can't quit all application servers at the same time. These are distributed, self-healing, and highly-redundant -- meaning that there are thousands of copies of each service, and the system will bring up new copies to replace the ones you kill, without loosing service (though you could, potentially, affect quality of service). Even the control mechanisms that allow all this magic to happen are run on the same platform, meaning that you can't affect the managers in this way either. The sort of things you can do are: affect billing and reporting, find sensitive company data, and, potentially, execute code remotely (though that will probably be in a container, and not have access to much else). Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. The code is huge, so downloading it will take forever, and then you'd have to read it. Finally, it's written in dozens of programming languages, some, like Golang, are unreadable to anyone but experts.
- ponyfleisch 8y ago
  > Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. Google is using a monorepo.

z3t4 8y ago

Google API Auth/Access is extremely tedious, so I guess someone just didn't bother. Basically you have to get an token, format it, make a couple of http posts, and finally you will have a token to make an access-token-token ... Now image all the steps you would have to make to create a new type of access to some internal API, that should not have public access anyway. Probably saved six months work. And the libraries are probably hiding all the obscurity so no-one did notice until this guy started digging.

eyeareque 8y ago
Sounds plausible. Now I’m curious how much extra work this finding has caused teams at Google. There are probably many other similar insecure paths that need to be cleaned up.