7 ms·

Kubernetes ingress-nginx controller vulnerability

51 points by jkaftzan 5y ago

jeffbee 5y ago
Are they for real with that curl? Normalize better security practices!
- doesnotexist 5y ago
  Yeah I think everyone feels icky about 'curl https:// https://.. | sh' but to what degree is doing so any more insecure than other "normalized" software distribution practices? See also: https://twitter.com/moyix/status/1451318133021675520 https://twitter.com/moyix/status/1451318133021675520
  - jeffbee 5y ago
    You can check for this vulnerability using the existing tooling any k8s admin already has on their systems by necessity. It's always foolish to install unknown software and security professionals should never advise that.
    - tbrownaw 5y ago
      > unknown software Looking at the blog URL and header bar, and the script URL, this is pretty clearly a company blog recommending to use the company's own product. I hardly think that context counts as "unknown".
      - justinclift 5y ago
        Lets say (for example) that was published on a wordpress site, and the admins for whatever reason didn't secure it properly. The article in question which _today_ looks all legit and points to a nice working script, might tomorrow be pointing to someone else's script that's a lot less legit. And yes, in this imaginary scenario that wordpress install leads to a host of other problems for the company. No need to make them your problems too though. ;)
- tbrownaw 5y ago
  I think it's contrary to currently-recommended ops practice (clearly imperative rather than nominally declarative), but how is it bad security practice? If you don't trust the origin, it's not like saving to a temporary file first is going to help you. Unless you're suggesting that everyone should always do their own code review and then compile from source...
  - justinclift 5y ago
    > If you don't trust the origin, it's not like saving to a temporary file first is going to help you. Huh? The whole point is to review the the code before you run it. If it seems like many people _don't_, well frankly, that's their problem. Encouraging bad practise is not the right direction.
  - jl6 5y ago
    Saving to a file first means you can at least run it in a test environment first, then be confident you’re running the same thing when moving into production.
  - asguy 5y ago
    At minimum you could download software that is at least signed by someone you trust. Rather than streaming arbitrary commands from a source that the extent of the authentication is that their CDN's TLS certs matches their domain.
  - junon 5y ago
    > but how is it bad security practice? Companies / organizations / team members go rogue sometimes. We've seen this even recently with e.g. kicad, freenode, the timezone database, etc. Just because you trust the source doesn't mean you should trust all of the scripts they tell you to run. Even if it's a good-faith script, you have no idea if it's making assumptions about your system that are not true and opening you up to sidechannel attacks and the like. Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
    - k8sToGo 5y ago
- 015a 5y ago
  I will argue until the day I die: curl | bash is actually more secure than most traditional software delivery mechanisms. Slightly, but meaningfully. With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run. If its on Github or a similar site, I can see in plain language the exact organization who published it (I wouldn't implicitly trust whoever "armosec" is, but I would trust, for example, "kubernetes", "microsoft", or "google" e.g.). Alternatives? - For something like this, to generate a report on whether you're affected? Maybe just a copy-paste bit of code? Isn't that the same thing, but less convenient? - apt-get install something. Who published it? Does apt have security scanning (no lol)? Where's the source code for that package (scattered to the wind no doubt)? Does it have reproducible builds so I can be certain the code I eventually find is what I install (lol no. but bash scripts do!) - Let's talk about `snap` for a second; Did you know that there's a `snap install aws-cli` package available [1], published by "Amazon Web Services", with a verified checkmark. Sounds great. Except, for years, no one on the actual aws-cli team had any idea where it came from [2]. They didn't authorize it, and they don't maintain it. It turns out, it was probably the result of a hackathon project by one AWS engineer. "Better security practices" for sure! The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content. Legit risky, good to educate about; don't curl | bash from sites you don't recognize. But this is a GitHub link! I would agree that package managers are superior from a security angle, if most package managers had strong security review processes, on both the package content and the publisher. The problem is: they don't. They're all lacking in this regard, from apt to flatpak to the Apple App Store. The most sophisticated is probably the Play Store, but binary command line distribution channels are nowhere close to having reproducible builds, verified Real ID developers, static analysis on distributed binaries, etc. Package managers are literally just "curl | bash" with some steps added. By and large, anyone can publish anything under any name; educating people that they're "safe" is actually a massive disservice. [1] https://snapcraft.io/aws-cli https://snapcraft.io/aws-cli [2] https://github.com/aws/aws-cli/issues/5142 https://github.com/aws/aws-cli/issues/5142
  - deleted 5y ago
    [deleted]
- chmod-777 5y ago
  I see you can also download it from Git and try it out, probably just tried to make it easier for people.
blakesterz 5y ago
"... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster." I'm not a Kubernetes guy, so I'm curious, how often is there a user with only those permissions? Is it common to have a user that can create/update and then doesn't already have some kind of other access to everything else? I don't know much about Kubernetes permissions. (I also just learned that the word Kubernetes is in the default Chrome spell checker thing, which was sure helpful in writing this comment!)
- cassianoleal 5y ago
  A user can have many permissions in his/her own namespace without compromising the other tenants. This sounds like it would allow any user with Ingress create/update access in any namespace to compromise secrets across all namespaces.
  - cassianoleal 5y ago
    It is unclear whether this is a problem if you're running namespaced ingress-controllers though. This comment [0] in the bug report says: there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved). I can't see how list permissions would allow retrieval of the secret value though. You'd need get permissions for that. [0] https://github.com/kubernetes/ingress-nginx/issues/7837#issuecomment-949688321 https://github.com/kubernetes/ingress-nginx/issues/7837#issu...
    - xnxn 5y ago
      > You'd need get permissions for that. I'm afraid not. HTTP GET on a collection endpoint (which is the operation represented by the list verb) returns the full object content. https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb https://kubernetes.io/docs/reference/access-authn-authz/auth...
      - 5y ago
invisible 5y ago
An issue with discussion: https://github.com/kubernetes/ingress-nginx/issues/7837 https://github.com/kubernetes/ingress-nginx/issues/7837 The only fix so far is removing the feature. >According to the publication, multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue. Ouch! I guess this means lots of secret/password rotations for k8s admins coming soon. I've always wondered if there are public Kubernetes multi-tenant setups in the wild rather than just ones with admins and developer roles/namespaces.
- deathanatos 5y ago
  What's the actual vulnerability? The issue links to itself for "more details"… (Also, yeah, it'd be nice if the URL of the OP had just been the Github issue. It is just as informative (or uninformative), and isn't an advertisement…)
- erulabs 5y ago
  For whatever it’s worth my startup began life as a multi tenant kubernetes provider. We have a ton of custom controls - and for example do not allow this “custom snippet” directive at all. We spent a very long time locking down as much of the api surface as possible. I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!
asguy 5y ago
> Hey Everyone, here's a high severity vulnerability report, to check for it, run the folliowing: curl -shttps://raw.githubusercontent.com/noideawhatyourerunning/install.sh https://raw.githubusercontent.com/noideawhatyourerunning/ins... |/bin/bash .... oh man
- justinclift 5y ago
  Don't forget to add a "sudo" in there for extra goodness. :D
  - deleted 5y ago
    [deleted]
  - corobo 5y ago
    It does that in the script for you :)
    - capableweb 5y ago
      Not only that, they don't do any checksum checks either, and seem to not even publish checksums.
      - lsandler 5y ago
        While I understand the concern with curl | bash, this method is used in many different open source product installations. The sh file is coming from github - pretty trustful source. You can always watch it in the browser (github will not trick you into different version) but you can also curl it or just download the source from the repository (just half a page above). You can also always review and rebuild the entire tool - another beauty of the open source. But most people just want to use the tool as fast and as simple as it can be. I guess there are options for any possible taste.
- hn_throwaway_99 5y ago
  Honestly, this kind of made my blood boil, and just points out how difficult security can be. Here is presumably a security company, in their own blog post outlining a high-severity vulnerability alert, advocating that you run untrusted code from some rando GitHub repo. It also outlines my fear about dealing with some "security consultants", because on the one hand they outline all of the things you need to "box check" for some security audit, but at the same time leave you less secure because they've just opened another giant gaping attack surface for your company - this is exactly what happened with the SolarWinds attack.
captn3m0 5y ago
Can we fix the title to include ingress-nginx instead? The upstream URL seems more clear as well: https://github.com/kubernetes/ingress-nginx/issues/7837 https://github.com/kubernetes/ingress-nginx/issues/7837
- waynesonfire 5y ago
  why? does ingress-nginx run outside of k8s?
  - captn3m0 5y ago
    You can run a kubernetes cluster without the nginx-ingress.
    - waynesonfire 5y ago
      calling it "Kubernetes ingress-nginx controller" seems reasonable
      - dang 5y ago
        Ok, we've put that in the title above. Thanks!
- jamescostian 5y ago
  Agreed, I'd prefer a GitHub issue over an advertisement. The HN user who posted this works for the company that wrote this post, and that person's title is "VP Marketing & Business Development". The issue contains a comment where they're also advertising their service, so I think that using the GitHub issue would be better, and would be a good compromise for them since they'll still have their ad.
  - lsandler 5y ago
    The problem is in intersection of nginx-ingress and kubernetes. Since ingress controller has access to secrets from all the namespaces (which is kubernetes side of the story) the nginx implementation with snippets added by users (the nginx contribution) may expose these secrets. The post also points to an open source tool that helps people to check if they are vulnerable, whether they want to get to the bottom of the issue or not.
zzyzxd 5y ago
Although this particular vulnerability is all about ingress-nginx and has nothing to do with Kubernetes itself, the current design of the ingress API sort of encouraged it. The reason that an ingress controller need permission to read all secrets across all namespaces in the first place, is because ingress objects can specify their own certs (stored in namespaced secret objects) within their namespaces, hence the ingress controller needs permission to read those secrets and load the certs into the reverse proxy (in case of ingress-nginx, it is nginx). This is such a scary permission to grant that cluster admins have to hand over all the secrets in the cluster to the ingress controller and hope it does the right thing. Now, ingress-nginx actually allows you to specify a default secret to be used by all ingresses it manages, and that would be the only secret it needs access to. I am not sure how many users out there are willing to dig into thousands lines of its manifest yaml files to tweak its rbac settings.
amirkaush 5y ago
You can use Kubescape (open source) to check if you are vulnerable. https://github.com/armosec/kubescape https://github.com/armosec/kubescape

51 points by jkaftzan 5y ago

jeffbee 5y ago

Are they for real with that curl? Normalize better security practices!

doesnotexist 5y ago
Yeah I think everyone feels icky about 'curl https:// https://.. | sh' but to what degree is doing so any more insecure than other "normalized" software distribution practices? See also: https://twitter.com/moyix/status/1451318133021675520 https://twitter.com/moyix/status/1451318133021675520
- jeffbee 5y ago
  You can check for this vulnerability using the existing tooling any k8s admin already has on their systems by necessity. It's always foolish to install unknown software and security professionals should never advise that.
  - tbrownaw 5y ago
    > unknown software Looking at the blog URL and header bar, and the script URL, this is pretty clearly a company blog recommending to use the company's own product. I hardly think that context counts as "unknown".
    - justinclift 5y ago
      Lets say (for example) that was published on a wordpress site, and the admins for whatever reason didn't secure it properly. The article in question which _today_ looks all legit and points to a nice working script, might tomorrow be pointing to someone else's script that's a lot less legit. And yes, in this imaginary scenario that wordpress install leads to a host of other problems for the company. No need to make them your problems too though. ;)
tbrownaw 5y ago
I think it's contrary to currently-recommended ops practice (clearly imperative rather than nominally declarative), but how is it bad security practice? If you don't trust the origin, it's not like saving to a temporary file first is going to help you. Unless you're suggesting that everyone should always do their own code review and then compile from source...
- justinclift 5y ago
  > If you don't trust the origin, it's not like saving to a temporary file first is going to help you. Huh? The whole point is to review the the code before you run it. If it seems like many people _don't_, well frankly, that's their problem. Encouraging bad practise is not the right direction.
- jl6 5y ago
  Saving to a file first means you can at least run it in a test environment first, then be confident you’re running the same thing when moving into production.
- asguy 5y ago
  At minimum you could download software that is at least signed by someone you trust. Rather than streaming arbitrary commands from a source that the extent of the authentication is that their CDN's TLS certs matches their domain.
- junon 5y ago
  > but how is it bad security practice? Companies / organizations / team members go rogue sometimes. We've seen this even recently with e.g. kicad, freenode, the timezone database, etc. Just because you trust the source doesn't mean you should trust all of the scripts they tell you to run. Even if it's a good-faith script, you have no idea if it's making assumptions about your system that are not true and opening you up to sidechannel attacks and the like. Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
  - k8sToGo 5y ago
015a 5y ago
I will argue until the day I die: curl | bash is actually more secure than most traditional software delivery mechanisms. Slightly, but meaningfully. With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run. If its on Github or a similar site, I can see in plain language the exact organization who published it (I wouldn't implicitly trust whoever "armosec" is, but I would trust, for example, "kubernetes", "microsoft", or "google" e.g.). Alternatives? - For something like this, to generate a report on whether you're affected? Maybe just a copy-paste bit of code? Isn't that the same thing, but less convenient? - apt-get install something. Who published it? Does apt have security scanning (no lol)? Where's the source code for that package (scattered to the wind no doubt)? Does it have reproducible builds so I can be certain the code I eventually find is what I install (lol no. but bash scripts do!) - Let's talk about `snap` for a second; Did you know that there's a `snap install aws-cli` package available [1], published by "Amazon Web Services", with a verified checkmark. Sounds great. Except, for years, no one on the actual aws-cli team had any idea where it came from [2]. They didn't authorize it, and they don't maintain it. It turns out, it was probably the result of a hackathon project by one AWS engineer. "Better security practices" for sure! The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content. Legit risky, good to educate about; don't curl | bash from sites you don't recognize. But this is a GitHub link! I would agree that package managers are superior from a security angle, if most package managers had strong security review processes, on both the package content and the publisher. The problem is: they don't. They're all lacking in this regard, from apt to flatpak to the Apple App Store. The most sophisticated is probably the Play Store, but binary command line distribution channels are nowhere close to having reproducible builds, verified Real ID developers, static analysis on distributed binaries, etc. Package managers are literally just "curl | bash" with some steps added. By and large, anyone can publish anything under any name; educating people that they're "safe" is actually a massive disservice. [1] https://snapcraft.io/aws-cli https://snapcraft.io/aws-cli [2] https://github.com/aws/aws-cli/issues/5142 https://github.com/aws/aws-cli/issues/5142
- deleted 5y ago
  [deleted]
chmod-777 5y ago
I see you can also download it from Git and try it out, probably just tried to make it easier for people.

blakesterz 5y ago

"... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster." I'm not a Kubernetes guy, so I'm curious, how often is there a user with only those permissions? Is it common to have a user that can create/update and then doesn't already have some kind of other access to everything else? I don't know much about Kubernetes permissions. (I also just learned that the word Kubernetes is in the default Chrome spell checker thing, which was sure helpful in writing this comment!)

cassianoleal 5y ago
A user can have many permissions in his/her own namespace without compromising the other tenants. This sounds like it would allow any user with Ingress create/update access in any namespace to compromise secrets across all namespaces.
- cassianoleal 5y ago
  It is unclear whether this is a problem if you're running namespaced ingress-controllers though. This comment [0] in the bug report says: there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved). I can't see how list permissions would allow retrieval of the secret value though. You'd need get permissions for that. [0] https://github.com/kubernetes/ingress-nginx/issues/7837#issuecomment-949688321 https://github.com/kubernetes/ingress-nginx/issues/7837#issu...
  - xnxn 5y ago
    > You'd need get permissions for that. I'm afraid not. HTTP GET on a collection endpoint (which is the operation represented by the list verb) returns the full object content. https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb https://kubernetes.io/docs/reference/access-authn-authz/auth...
    - 5y ago

invisible 5y ago

An issue with discussion: https://github.com/kubernetes/ingress-nginx/issues/7837 https://github.com/kubernetes/ingress-nginx/issues/7837 The only fix so far is removing the feature. >According to the publication, multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue. Ouch! I guess this means lots of secret/password rotations for k8s admins coming soon. I've always wondered if there are public Kubernetes multi-tenant setups in the wild rather than just ones with admins and developer roles/namespaces.

deathanatos 5y ago
What's the actual vulnerability? The issue links to itself for "more details"… (Also, yeah, it'd be nice if the URL of the OP had just been the Github issue. It is just as informative (or uninformative), and isn't an advertisement…)
erulabs 5y ago
For whatever it’s worth my startup began life as a multi tenant kubernetes provider. We have a ton of custom controls - and for example do not allow this “custom snippet” directive at all. We spent a very long time locking down as much of the api surface as possible. I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!

asguy 5y ago

> Hey Everyone, here's a high severity vulnerability report, to check for it, run the folliowing: curl -shttps://raw.githubusercontent.com/noideawhatyourerunning/install.sh https://raw.githubusercontent.com/noideawhatyourerunning/ins... |/bin/bash .... oh man

justinclift 5y ago
Don't forget to add a "sudo" in there for extra goodness. :D
- deleted 5y ago
  [deleted]
- corobo 5y ago
  It does that in the script for you :)
  - capableweb 5y ago
    Not only that, they don't do any checksum checks either, and seem to not even publish checksums.
    - lsandler 5y ago
      While I understand the concern with curl | bash, this method is used in many different open source product installations. The sh file is coming from github - pretty trustful source. You can always watch it in the browser (github will not trick you into different version) but you can also curl it or just download the source from the repository (just half a page above). You can also always review and rebuild the entire tool - another beauty of the open source. But most people just want to use the tool as fast and as simple as it can be. I guess there are options for any possible taste.
hn_throwaway_99 5y ago
Honestly, this kind of made my blood boil, and just points out how difficult security can be. Here is presumably a security company, in their own blog post outlining a high-severity vulnerability alert, advocating that you run untrusted code from some rando GitHub repo. It also outlines my fear about dealing with some "security consultants", because on the one hand they outline all of the things you need to "box check" for some security audit, but at the same time leave you less secure because they've just opened another giant gaping attack surface for your company - this is exactly what happened with the SolarWinds attack.

captn3m0 5y ago

Can we fix the title to include ingress-nginx instead? The upstream URL seems more clear as well: https://github.com/kubernetes/ingress-nginx/issues/7837 https://github.com/kubernetes/ingress-nginx/issues/7837

waynesonfire 5y ago
why? does ingress-nginx run outside of k8s?
- captn3m0 5y ago
  You can run a kubernetes cluster without the nginx-ingress.
  - waynesonfire 5y ago
    calling it "Kubernetes ingress-nginx controller" seems reasonable
    - dang 5y ago
      Ok, we've put that in the title above. Thanks!
jamescostian 5y ago
Agreed, I'd prefer a GitHub issue over an advertisement. The HN user who posted this works for the company that wrote this post, and that person's title is "VP Marketing & Business Development". The issue contains a comment where they're also advertising their service, so I think that using the GitHub issue would be better, and would be a good compromise for them since they'll still have their ad.
- lsandler 5y ago
  The problem is in intersection of nginx-ingress and kubernetes. Since ingress controller has access to secrets from all the namespaces (which is kubernetes side of the story) the nginx implementation with snippets added by users (the nginx contribution) may expose these secrets. The post also points to an open source tool that helps people to check if they are vulnerable, whether they want to get to the bottom of the issue or not.

zzyzxd 5y ago

Although this particular vulnerability is all about ingress-nginx and has nothing to do with Kubernetes itself, the current design of the ingress API sort of encouraged it. The reason that an ingress controller need permission to read all secrets across all namespaces in the first place, is because ingress objects can specify their own certs (stored in namespaced secret objects) within their namespaces, hence the ingress controller needs permission to read those secrets and load the certs into the reverse proxy (in case of ingress-nginx, it is nginx). This is such a scary permission to grant that cluster admins have to hand over all the secrets in the cluster to the ingress controller and hope it does the right thing. Now, ingress-nginx actually allows you to specify a default secret to be used by all ingresses it manages, and that would be the only secret it needs access to. I am not sure how many users out there are willing to dig into thousands lines of its manifest yaml files to tweak its rbac settings.

amirkaush 5y ago

You can use Kubescape (open source) to check if you are vulnerable. https://github.com/armosec/kubescape https://github.com/armosec/kubescape