16 ms·

Binary Authorization for Borg

102 points by mayakacz 7y ago

panarky 7y ago
> We want to have confidence that the administrators who run the systems that access user data cannot abuse their powers. So "Binary Authorization for Borg" is a defense against getting Snowdened.
- shadowgovt 7y ago
  It's more a defense against getting NSA'd (via the specific threat model of an attacker secretly replacing a security service with an implementation that looks very similar but is much easier to crack).
  - skybrian 7y ago
    More generally you might say it supports rule of law. If something happens according to procedure then it's ok. You might not think that's much of a guarantee, but it beats the alternative where things happen due to shadow processes.
    - mayakacz 7y ago
      I think that's right. I would strengthen that statement slightly - it's about ensuring that no actor - whether an insider, or someone who has stolen their credentials, or otherwise compromised them - can perform an action that single handedly accesses user data, without it being known to another actor - via access logs, via approvals, etc. In terms of the upstream introduction of a new vulnerability, Binary Authorization for Borg can only verify that the code was in fact merged. See the section on third party code, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)." Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.
tptacek 7y ago
Has anyone outside of Google implemented something similar in spirit to this for K8s or ECS? What was the threat model you were considering when you built it? Was it worth it?
- ImJasonH 7y ago
  Kritis[0] is a K8s implementation of this that intends to block deployments of images that haven't been properly vetted beforehand, or has critical vulnerabilities, etc. Whitepaper: https://github.com/grafeas/kritis/blob/master/docs/binary-authorization.md https://github.com/grafeas/kritis/blob/master/docs/binary-au... [0] https://github.com/grafeas/kritis https://github.com/grafeas/kritis
- mayakacz 7y ago
  Yes, there's a few listed in this blog post: https://cloud.google.com/blog/products/identity-security/beyondprod-whitepaper-discusses-cloud-native-security-at-google https://cloud.google.com/blog/products/identity-security/bey... - Kubernetes admission controllers, OSS part of Kubernetes: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ https://kubernetes.io/docs/reference/access-authn-authz/admi... - Kritis, OSS: https://opensource.google/projects/kritis https://opensource.google/projects/kritis - OPA Gatekeeper, OSS: https://github.com/open-policy-agent/gatekeeper https://github.com/open-policy-agent/gatekeeper - Binary Authorization on GKE/Anthos: https://cloud.google.com/binary-authorization/ https://cloud.google.com/binary-authorization/ They don't all do all the pieces. The hardest part is going to be integrating whatever enforcement solution you choose with your upstream CI/CD pipeline. Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.
- deleted 7y ago
  [deleted]
rsync 7y ago
In case anyone else wonders what 'borg' is: "Our infrastructure is containerized, using a cluster management system called Borg." I was hoping they had some predictable, indexed build for borg backup[1]. [1] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/
- kevincox 7y ago
  Borg is basically the internal predecessor to Kubernetes.
  - KaiserPro 7y ago
    I know this is pedantic, but I'd argue is an inspiration for, in its present state kubernetes is unable to scale to a datacenter, let alone globally at google scale.
    - kevincox 7y ago
      I will agree that it was the inspiration. And I think there is slow movement towards scaling kubernetes. Note that Borg isn't a global services. There are many instances in different locations. Borg also doesn't scale globally.
      - p_l 7y ago
        Isn't Borg usually scaled to few rows of racks? i.e. not even spanning whole building, yet alone datacenter.
  - mehrdada 7y ago
    Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).
    - 7y ago
justicezyx 7y ago
I led the portion of this project on Borg itself. Security team did most of the security infrastructure, and coordination among almost every large infrastructure system team inside TI. I'll be waiting for them to answer any questions. :)
- agv123 7y ago
  I've seen references to gVisor being used 'in production' for google app engine && cloud run and so forth. Scanning through recent commits && the github repo this is clearly not the case - there are way too many outstanding issues and outright missing support for various things. Is this another project where it was written in a different language or something and then ported out? Can you clarify?
  - justicezyx 7y ago
    I cannot say anything about internal use of gVisor. Sorry. As a bystander from outside, I generally don't like VM type of mechanism as security mechanism. Unless it's actually a VM hypervisor. That way hardware can be utilized to define a relatively simper and more robust security model. (Of cuz, not saying hardware is always superior please don't chase me on this direction). On the contrary, true software sandbox like ebpf and webassembly with limited capabilities in its building blocks and clearly defined application scenarios, are better ways to do security in software.
  - prattmic 7y ago
    I work on gVisor, I can answer this! gVisor is not a rewritten version of an internal tool. The code you see really does run in production for App Engine and Cloud Run. While there are some internal modifications to better integrate with internal infrastructure, the vast majority of the code is identical to open source, critically including all of the system call handling, filesystem, and memory management code. While browsing through our issues will show that we still have plenty to work on, the vast majority of applications work well inside gVisor.
- ASinclair 7y ago
  A comment, not a question: Though I think it was worth the cost, I'd say this was one of the most painful mandates/rollouts I've had to endure. The cost to developer productivity was pretty significant. I would have liked to have seen that impact discussed.
trishankdatadog 7y ago
On a related note, we have built an E2E-verified, tamper-evident CI/CD pipeline for the Datadog Agent integrations [1]: the Agent will trust and install only integrations that correspond to source code that have signed by our developers. If there is an attack anywhere between our developers and end-users, it will be caught. Unlike Binary Authorization for Borg, our security guarantees are publicly verifiable. [1] https://www.datadoghq.com/blog/engineering/secure-publication-of-datadog-agent-integrations-with-tuf-and-in-toto/ https://www.datadoghq.com/blog/engineering/secure-publicatio...
- tptacek 7y ago
  I saw this before and meant to post about it, because it's really neat.
  - trishankdatadog 7y ago
    Thanks, Thomas!
- packetslave 7y ago
  That's a bit of disingenuous reply... Binary Authorization for Borg is for verifying binaries running inside Google, not code installed on end-user machines. Having the authorization be "publicly verifiable" makes no sense.
  - ithkuil 7y ago
    Publicly verifiable is a (stronger) superset of privately verifiable
  - mayakacz 7y ago
    Agreed with this statement. It's a best practice generally to verify all software updates originate from a particular source before applying them in your environment. Most over the wire updates do this. What's different with Binary Authorization for Borg is that within Google, that last verification step means more than just "came from Google", but "came from Google and went through all previous necessary checks", because of the way the CI/CD system works together. Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.
alexellisuk 7y ago
Fascinating, the part we may get the most out of is "Adopting similar controls in your organization" - not just Google For Everyone Else.
seriesf 7y ago
One thing that really squicked me out when I left Google is how other companies, even large and sophisticated ones, are using all kinds of garbage that comes from canonical or red hat or percona, and they have NO IDEA what's in there. Say what you want about google's NIH culture, but in regards to code provenance and verifiable builds they are doing the right thing and many others are not.
- GuyPostington 7y ago
  Can you give an example of this garbage?
  - seriesf 7y ago
    Literally anything that comes from a vendor in a package? Percona server/toolkit? Every binary package in Ubuntu? The Linux kernel as built and distributed by Red Hat?
    - pjc50 7y ago
      Yeah, that's not a definition of garbage that's going to be taken seriously.
      - seriesf 7y ago
        You're right; Google is the only big tech that takes insider risk seriously.
  - mayakacz 7y ago
    I think 'garbage' is a strong word, but I believe what the original poster is trying to say is that there are a lot of binaries, packages, and libraries that most organizations will consume from upstream, and not verify directly. This requires either trust on a third party (often many third parties - in the case of open source), or more intense validation of those components and any changes to those components. Binary Authorization for Borg performs verification for pieces that come out of Google's CI/CD pipeline. For third party code, see in the doc, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)." Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.
sterlind 7y ago
At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?
- als0 7y ago
  Once configured, the IMA appraisal subsystem in the Linux kernel can ensure that only signed code executes.
- mdip 7y ago
  Thanks for that -- I love MS security practices/stories; interesting. Yes, Linux does have ELF signing. I'm guessing you are speaking more generally about ensuring that "only signed code is allowed to execute", rather than just "making sure ELF binaries are signed" based on the remaining context of your comment. Similar to Windows, making sure that exe files are signed isn't enough (PowerShell, drivers, kernel, firmware, etc) -- there's PowerShell scripts, etc, and "block execution of unsigned code" or even "block privileged execution of unsigned code". Assuming that "signed code execution", as I frequently discover when I go looking for "how to do Foo Linux", there's more than one[0] way, depending on what the need/device/system is. Windows is in a lot of places you don't expect -- ATMs, IoT devices, etc -- Linux is ... it'd be easier to come up with a list of device types that haven't had a Linux kernel running on them. LWN had a write up in 2017 -- I know I've read more current, but theirs was a good summary and answers your question. The Linux Integrity Measurement Architecture (IMA)[2] is a more complete approach. Those are the more "general-use options" that I was aware of. [0] Often at least 8; and there's usually a few of them arguing with eachother over something that's between the extremes of "who's dad would actually win in a boxing match" and ... religion. /s [1] https://lwn.net/Articles/733431/ https://lwn.net/Articles/733431/ [2] Wiki Link- https://sourceforge.net/p/linux-ima/wiki/Home/ https://sourceforge.net/p/linux-ima/wiki/Home/ Good write-up https://lwn.net/Articles/488906/ https://lwn.net/Articles/488906/
- theamk 7y ago
  How does this work with scripts? Python or Ruby script can do a lot of damage. Or does MS prohibit all scripting languages on production systems?
philsnow 7y ago
> Adopting similar controls in your organization > Figure out how to manage third party code. > Many of the CI/CD controls we describe in this paper are placed where your code is developed, reviewed, and maintained by one organization. If you are in this situation, consider how you will include third party code as part of your policy requirements. For example, you could initially exempt the code, while you move towards an ideal state of keeping a repository of all third party code used, and regularly vet that code against your security requirements. I don't know how much third party code is in use at Google these days, but I'd be curious to know if there is a formal effort at cataloging most-often used / most-sensitive third party code and prioritizing reviews of it. I've thought about the problem of vetting programming language packages (pypi, npm, rubygems, whatever) off and on. It seems like the only two tenable strategies are "don't pin anything / always use tip of master" and "freeze deps, vet transitive deps at that frozen point, vendor the corresponding deps, and if you ever need to update a requirement for a feature or bugfix, go through the process again". The latter seems like it could be out-sourced to a certain degree, if you trusted other organizations to "vet transitive deps".
- cbhl 7y ago
  For one thing, all third-party code is checked into one top-level folder in the monorepo. This is publicly documented here: https://opensource.google/docs/thirdparty/ https://opensource.google/docs/thirdparty/ This means that dependencies on a third-party library can be found simply by looking at deps lines in BUILD rules. That can then inform which projects you want to run (for example) fuzzers on: https://opensource.googleblog.com/2016/12/announcing-oss-fuzz-continuous-fuzzing.html https://opensource.googleblog.com/2016/12/announcing-oss-fuz... https://google.github.io/oss-fuzz/ https://google.github.io/oss-fuzz/
HocusLocus 7y ago
I read the CIO-level summary! Woo hoo hoo! Im portant.

102 points by mayakacz 7y ago

> We want to have confidence that the administrators who run the systems that access user data cannot abuse their powers. So "Binary Authorization for Borg" is a defense against getting Snowdened.

shadowgovt 7y ago
It's more a defense against getting NSA'd (via the specific threat model of an attacker secretly replacing a security service with an implementation that looks very similar but is much easier to crack).
- skybrian 7y ago
  More generally you might say it supports rule of law. If something happens according to procedure then it's ok. You might not think that's much of a guarantee, but it beats the alternative where things happen due to shadow processes.
  - mayakacz 7y ago
    I think that's right. I would strengthen that statement slightly - it's about ensuring that no actor - whether an insider, or someone who has stolen their credentials, or otherwise compromised them - can perform an action that single handedly accesses user data, without it being known to another actor - via access logs, via approvals, etc. In terms of the upstream introduction of a new vulnerability, Binary Authorization for Borg can only verify that the code was in fact merged. See the section on third party code, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)." Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

tptacek 7y ago

Has anyone outside of Google implemented something similar in spirit to this for K8s or ECS? What was the threat model you were considering when you built it? Was it worth it?

ImJasonH 7y ago
Kritis[0] is a K8s implementation of this that intends to block deployments of images that haven't been properly vetted beforehand, or has critical vulnerabilities, etc. Whitepaper: https://github.com/grafeas/kritis/blob/master/docs/binary-authorization.md https://github.com/grafeas/kritis/blob/master/docs/binary-au... [0] https://github.com/grafeas/kritis https://github.com/grafeas/kritis
mayakacz 7y ago
Yes, there's a few listed in this blog post: https://cloud.google.com/blog/products/identity-security/beyondprod-whitepaper-discusses-cloud-native-security-at-google https://cloud.google.com/blog/products/identity-security/bey... - Kubernetes admission controllers, OSS part of Kubernetes: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ https://kubernetes.io/docs/reference/access-authn-authz/admi... - Kritis, OSS: https://opensource.google/projects/kritis https://opensource.google/projects/kritis - OPA Gatekeeper, OSS: https://github.com/open-policy-agent/gatekeeper https://github.com/open-policy-agent/gatekeeper - Binary Authorization on GKE/Anthos: https://cloud.google.com/binary-authorization/ https://cloud.google.com/binary-authorization/ They don't all do all the pieces. The hardest part is going to be integrating whatever enforcement solution you choose with your upstream CI/CD pipeline. Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.
deleted 7y ago
[deleted]

rsync 7y ago

In case anyone else wonders what 'borg' is: "Our infrastructure is containerized, using a cluster management system called Borg." I was hoping they had some predictable, indexed build for borg backup[1]. [1] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/

kevincox 7y ago
Borg is basically the internal predecessor to Kubernetes.
- KaiserPro 7y ago
  I know this is pedantic, but I'd argue is an inspiration for, in its present state kubernetes is unable to scale to a datacenter, let alone globally at google scale.
  - kevincox 7y ago
    I will agree that it was the inspiration. And I think there is slow movement towards scaling kubernetes. Note that Borg isn't a global services. There are many instances in different locations. Borg also doesn't scale globally.
    - p_l 7y ago
      Isn't Borg usually scaled to few rows of racks? i.e. not even spanning whole building, yet alone datacenter.
- mehrdada 7y ago
  Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).
  - 7y ago

justicezyx 7y ago

I led the portion of this project on Borg itself. Security team did most of the security infrastructure, and coordination among almost every large infrastructure system team inside TI. I'll be waiting for them to answer any questions. :)

agv123 7y ago
I've seen references to gVisor being used 'in production' for google app engine && cloud run and so forth. Scanning through recent commits && the github repo this is clearly not the case - there are way too many outstanding issues and outright missing support for various things. Is this another project where it was written in a different language or something and then ported out? Can you clarify?
- justicezyx 7y ago
  I cannot say anything about internal use of gVisor. Sorry. As a bystander from outside, I generally don't like VM type of mechanism as security mechanism. Unless it's actually a VM hypervisor. That way hardware can be utilized to define a relatively simper and more robust security model. (Of cuz, not saying hardware is always superior please don't chase me on this direction). On the contrary, true software sandbox like ebpf and webassembly with limited capabilities in its building blocks and clearly defined application scenarios, are better ways to do security in software.
- prattmic 7y ago
  I work on gVisor, I can answer this! gVisor is not a rewritten version of an internal tool. The code you see really does run in production for App Engine and Cloud Run. While there are some internal modifications to better integrate with internal infrastructure, the vast majority of the code is identical to open source, critically including all of the system call handling, filesystem, and memory management code. While browsing through our issues will show that we still have plenty to work on, the vast majority of applications work well inside gVisor.
ASinclair 7y ago
A comment, not a question: Though I think it was worth the cost, I'd say this was one of the most painful mandates/rollouts I've had to endure. The cost to developer productivity was pretty significant. I would have liked to have seen that impact discussed.

trishankdatadog 7y ago

On a related note, we have built an E2E-verified, tamper-evident CI/CD pipeline for the Datadog Agent integrations [1]: the Agent will trust and install only integrations that correspond to source code that have signed by our developers. If there is an attack anywhere between our developers and end-users, it will be caught. Unlike Binary Authorization for Borg, our security guarantees are publicly verifiable. [1] https://www.datadoghq.com/blog/engineering/secure-publication-of-datadog-agent-integrations-with-tuf-and-in-toto/ https://www.datadoghq.com/blog/engineering/secure-publicatio...

tptacek 7y ago
I saw this before and meant to post about it, because it's really neat.
- trishankdatadog 7y ago
  Thanks, Thomas!
packetslave 7y ago
That's a bit of disingenuous reply... Binary Authorization for Borg is for verifying binaries running inside Google, not code installed on end-user machines. Having the authorization be "publicly verifiable" makes no sense.
- ithkuil 7y ago
  Publicly verifiable is a (stronger) superset of privately verifiable
- mayakacz 7y ago
  Agreed with this statement. It's a best practice generally to verify all software updates originate from a particular source before applying them in your environment. Most over the wire updates do this. What's different with Binary Authorization for Borg is that within Google, that last verification step means more than just "came from Google", but "came from Google and went through all previous necessary checks", because of the way the CI/CD system works together. Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

alexellisuk 7y ago

Fascinating, the part we may get the most out of is "Adopting similar controls in your organization" - not just Google For Everyone Else.

seriesf 7y ago

One thing that really squicked me out when I left Google is how other companies, even large and sophisticated ones, are using all kinds of garbage that comes from canonical or red hat or percona, and they have NO IDEA what's in there. Say what you want about google's NIH culture, but in regards to code provenance and verifiable builds they are doing the right thing and many others are not.

GuyPostington 7y ago
Can you give an example of this garbage?
- seriesf 7y ago
  Literally anything that comes from a vendor in a package? Percona server/toolkit? Every binary package in Ubuntu? The Linux kernel as built and distributed by Red Hat?
  - pjc50 7y ago
    Yeah, that's not a definition of garbage that's going to be taken seriously.
    - seriesf 7y ago
      You're right; Google is the only big tech that takes insider risk seriously.
- mayakacz 7y ago
  I think 'garbage' is a strong word, but I believe what the original poster is trying to say is that there are a lot of binaries, packages, and libraries that most organizations will consume from upstream, and not verify directly. This requires either trust on a third party (often many third parties - in the case of open source), or more intense validation of those components and any changes to those components. Binary Authorization for Borg performs verification for pieces that come out of Google's CI/CD pipeline. For third party code, see in the doc, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)." Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

sterlind 7y ago

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

als0 7y ago
Once configured, the IMA appraisal subsystem in the Linux kernel can ensure that only signed code executes.
mdip 7y ago
Thanks for that -- I love MS security practices/stories; interesting. Yes, Linux does have ELF signing. I'm guessing you are speaking more generally about ensuring that "only signed code is allowed to execute", rather than just "making sure ELF binaries are signed" based on the remaining context of your comment. Similar to Windows, making sure that exe files are signed isn't enough (PowerShell, drivers, kernel, firmware, etc) -- there's PowerShell scripts, etc, and "block execution of unsigned code" or even "block privileged execution of unsigned code". Assuming that "signed code execution", as I frequently discover when I go looking for "how to do Foo Linux", there's more than one[0] way, depending on what the need/device/system is. Windows is in a lot of places you don't expect -- ATMs, IoT devices, etc -- Linux is ... it'd be easier to come up with a list of device types that haven't had a Linux kernel running on them. LWN had a write up in 2017 -- I know I've read more current, but theirs was a good summary and answers your question. The Linux Integrity Measurement Architecture (IMA)[2] is a more complete approach. Those are the more "general-use options" that I was aware of. [0] Often at least 8; and there's usually a few of them arguing with eachother over something that's between the extremes of "who's dad would actually win in a boxing match" and ... religion. /s [1] https://lwn.net/Articles/733431/ https://lwn.net/Articles/733431/ [2] Wiki Link- https://sourceforge.net/p/linux-ima/wiki/Home/ https://sourceforge.net/p/linux-ima/wiki/Home/ Good write-up https://lwn.net/Articles/488906/ https://lwn.net/Articles/488906/
theamk 7y ago
How does this work with scripts? Python or Ruby script can do a lot of damage. Or does MS prohibit all scripting languages on production systems?

philsnow 7y ago

> Adopting similar controls in your organization > Figure out how to manage third party code. > Many of the CI/CD controls we describe in this paper are placed where your code is developed, reviewed, and maintained by one organization. If you are in this situation, consider how you will include third party code as part of your policy requirements. For example, you could initially exempt the code, while you move towards an ideal state of keeping a repository of all third party code used, and regularly vet that code against your security requirements. I don't know how much third party code is in use at Google these days, but I'd be curious to know if there is a formal effort at cataloging most-often used / most-sensitive third party code and prioritizing reviews of it. I've thought about the problem of vetting programming language packages (pypi, npm, rubygems, whatever) off and on. It seems like the only two tenable strategies are "don't pin anything / always use tip of master" and "freeze deps, vet transitive deps at that frozen point, vendor the corresponding deps, and if you ever need to update a requirement for a feature or bugfix, go through the process again". The latter seems like it could be out-sourced to a certain degree, if you trusted other organizations to "vet transitive deps".

cbhl 7y ago
For one thing, all third-party code is checked into one top-level folder in the monorepo. This is publicly documented here: https://opensource.google/docs/thirdparty/ https://opensource.google/docs/thirdparty/ This means that dependencies on a third-party library can be found simply by looking at deps lines in BUILD rules. That can then inform which projects you want to run (for example) fuzzers on: https://opensource.googleblog.com/2016/12/announcing-oss-fuzz-continuous-fuzzing.html https://opensource.googleblog.com/2016/12/announcing-oss-fuz... https://google.github.io/oss-fuzz/ https://google.github.io/oss-fuzz/

HocusLocus 7y ago

I read the CIO-level summary! Woo hoo hoo! Im portant.