4 ms·

Has there been any word of how long this has been known about? I recall a certain group of people talking confidently, open but vaguely about a serious k8s secu

Has there been any word of how long this has been known about? I recall a certain group of people talking confidently, open but vaguely about a serious k8s security bug at least a few months ago. Given the number of clusters that undoubtedly had to be upgraded for this, it wouldn't surprise me if this was found a while ago. Also, my heart goes out to the [big company] engineering manager asking in the GitHub thread if 1.7.x clusters are affected.

aberoham 8y ago
The chap who discovered & reported the flaw wrote up his story here: https://rancher.com/blog/2018/2018-12-04-k8s-cve/ https://rancher.com/blog/2018/2018-12-04-k8s-cve/
darren0 8y ago
I'm fairly confident this issue has existed since pretty much the beginning of k8s. It is possible it doesn't go back that far because SPDY used to be used and not websockets. The basic Upgrade logic still applies so I assume it was there. This flaw is actually in quite a few open source projects, it's a common approach and mistake. For most projects it results in a functional bug, not a security issue. I personally found this issue and highly doubt anybody previously exploited it. This bug was discovered from the perspective of a functional bug and only later realized it had a security impact. So it wasn't like somebody was first hacked, and also a security researcher didn't find it either. Basically this issue is very nuanced and I don't think many would be looking for it (although people will now). Although plenty of people are saying the sky is failing, this is mostly a privileged escalation issue which means you first need valid access to do something harmful. So it's not a fabulous attack vector because hard multitenant clusters are extremely rare. Searching for anonymous auth on kubelets is a better use of your time.
- colemickens 8y ago
  Thanks Darren, I wish I'd read your post earlier to have realized the timeline. I agree with your conclusions re: the sky. Thank you for the write-up and replying here.
spiffxp 8y ago
I was part of the Kubernetes v1.13 release team, though not part of the team that produced this fix. The project's Product Security Team adhered to the timeline indicated in the project's security release process: https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#patch-release-and-public-communication https://github.com/kubernetes/sig-release/blob/master/securi... tl;dr a fix is (edit: optionally) sent out to a private distributors list under embargo within 2 weeks of disclosure, and public disclosure (with new releases) happens within 3 weeks of disclosure (with some discretion for timing to make sure it's not buried in a weekend or off-hours) I can't speak to who knew about it when outside of the project, but I know the project acted expediently once the vulnerability was disclosed.
- colemickens 8y ago
  I should've known that there was an official timeline as part of the process, but forgot. I wouldn't have asked about that otherwise; I didn't mean to cast any aspersions. Whatever I'm thinking of was from folks I respect, presumably about another issue or I'm simply mis-remembering things. Thank you. (Also, aside, props to everyone that got this rolled out so fast at the major providers.)

aberoham 8y ago

The chap who discovered & reported the flaw wrote up his story here: https://rancher.com/blog/2018/2018-12-04-k8s-cve/ https://rancher.com/blog/2018/2018-12-04-k8s-cve/

darren0 8y ago

I'm fairly confident this issue has existed since pretty much the beginning of k8s. It is possible it doesn't go back that far because SPDY used to be used and not websockets. The basic Upgrade logic still applies so I assume it was there. This flaw is actually in quite a few open source projects, it's a common approach and mistake. For most projects it results in a functional bug, not a security issue. I personally found this issue and highly doubt anybody previously exploited it. This bug was discovered from the perspective of a functional bug and only later realized it had a security impact. So it wasn't like somebody was first hacked, and also a security researcher didn't find it either. Basically this issue is very nuanced and I don't think many would be looking for it (although people will now). Although plenty of people are saying the sky is failing, this is mostly a privileged escalation issue which means you first need valid access to do something harmful. So it's not a fabulous attack vector because hard multitenant clusters are extremely rare. Searching for anonymous auth on kubelets is a better use of your time.

colemickens 8y ago
Thanks Darren, I wish I'd read your post earlier to have realized the timeline. I agree with your conclusions re: the sky. Thank you for the write-up and replying here.

spiffxp 8y ago

I was part of the Kubernetes v1.13 release team, though not part of the team that produced this fix. The project's Product Security Team adhered to the timeline indicated in the project's security release process: https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#patch-release-and-public-communication https://github.com/kubernetes/sig-release/blob/master/securi... tl;dr a fix is (edit: optionally) sent out to a private distributors list under embargo within 2 weeks of disclosure, and public disclosure (with new releases) happens within 3 weeks of disclosure (with some discretion for timing to make sure it's not buried in a weekend or off-hours) I can't speak to who knew about it when outside of the project, but I know the project acted expediently once the vulnerability was disclosed.

colemickens 8y ago
I should've known that there was an official timeline as part of the process, but forgot. I wouldn't have asked about that otherwise; I didn't mean to cast any aspersions. Whatever I'm thinking of was from folks I respect, presumably about another issue or I'm simply mis-remembering things. Thank you. (Also, aside, props to everyone that got this rolled out so fast at the major providers.)