3 ms·
Most container images in production have serious vulnerabilities
- theamk 4y ago> An alarming 87% of container images running in production have critical or high-severity vulnerabilities, [...] Yet only 15% of those unpatched critical and high-severity vulnerabilities are in packages in use at runtime when patches are available. [...] Further, only 2% of the vulnerabilities are exploitable. And this is why the numbers are so high -- because all of those "vulnerability reports" are complete junk most of the time. SSHD warnings for systems which do not run sshd. ImageMagick vulnerabilities for internal code docs generator. A high severity sudo vulnerability which requires a specific very uncommon configuration. A great idea on paper, but it is achieving an opposite effect, teaching people that most of automated vulnerability detections are junk.
- pyuser583 4y agoI’ve worked doing data analysis on vulnerabilities, and it’s really really bizarre. Some times the most secure systems are running insanely old software. Sometimes software that’s a bit out of date correlates to fewer attacks. When people are attacked, they upgrade their software. So a company that has gone a long time with no updates, followed by frantically updating everything could be a warning sign.
- threatactor 4y agoMaybe some weirdness is cleared up by noting that besides being an indication that upgrades haven't been necessary for a while, really old software often gains additional resilience to attacks by becoming more obscure with age. Targeted efforts are warded off somewhat by the unfamiliar attack surface and mass scanners typically won't be looking for ancient services. This is only tangentially related, but Im curious if you found anything about the typical length of time between systems being compromised to when intrusions are discovered as part of your work - if you would like to and are at liberty to share :)
- gladiatr72 4y agoThe local university maintained DECNet routing to support systems still in use in the physics department until the early 2000's (when they also ceased inter-building IPX routing). Guess which systems were never exploited for distributing Warez and porn? All exploited systems before then were either windows nt or Linux. Commercial Unix (Dec, sgi, sun) rarely came up and generally only when there were accounts using identical credentials with a more exploitable platform.
- aflag 4y agoVery old software may gain an edge. However, that takes a while and depends on the number of people using the old software as well. For instance, Windows XP was a big target for decades after it had been superseded by newer versions of windows. If your software stack is from the 90s, I'd say you might just getting obscure enough by this point. Scanners are frequently not looking for the breakthrough 0-day vulnerabilities, but just for the low hanging fruit of old and well known vulnerabilities.
- touisteur 4y agoWhat I'm also seeing is a knee-jerk reaction to COTS/OSS software, and people using these scores to justify building software in-house. Can't be vulnerable if you're not in vulnerability scanners, right? Which seems a bit weird, as they wouldn't have paid a cent before for security auditing OSS codebases.
- koblas 4y agoI cannot empasis how true this is. The classic UNIX problem was that the LPT printer daemon has an issue (it had lots and lots). But, none of your systems were running LPT, but you still had to patch 1000+s of systems just to maintain a security policy. What's different between full on UNIX systems and Docker, the possibility of deploying code based on scratch images. Imagine a system which only had the pieces necessary to run in production, your security exception reports would go to zero.
- goalieca 4y agoIt’s almost free to rebuild and redeploy from a Dockerfile if you have a good devops culture. This would replace the traditional unattended upgrader with scheduled reboots.
- yucky 4y agoThis drives me nuts. And I always remind myself not to fall into this behavior but can't help it with the sheer volume of alerts.
- Proven 4y agoSure. And 99.99% of them aren't exploitable. I have a Python-based pod that polls an internal API endpoint. Why should I care if the Python container has a critical vulnerability completely unrelated to what it does?
- pella 4y agoPRO: 1.) https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model 2.) Our own blind spots + unknown unknowns CON: 1.) There are many false alarms or not critical issues : see: https://github.com/search?q=org%3Adocker-library+CVE&type=issues https://github.com/search?q=org%3Adocker-library+CVE&type=is...
- c7DJTLrn 4y agoNo they don't. Container scanning is a load of shit, sorry. Just because something in the container contains a vulnerability, doesn't mean the application itself is vulnerable. It is not the same thing. Instead of doing the right thing, which would be static and dynamic analysis of the entrypoint to determine the runtime dependencies, the industry has just resorted to saying anything you bundle with your app needs to be 100% invulnerable at all times. It's stupid and practically unachievable if your container is big. Container scanners and dependency bots like Dependabot are snake oil solutions sold to people too lazy to actually secure their application.
- sclarisse 4y agoOh, but I am going to defend Dependabot. It’s not there to secure your app. It’s there so your smallcorp team can get one or two PRs a day against your codebases, updating your software dependencies one at a time — rather than an unmanageable mess or slow rot on old systems. That’s literally the original design goal. Any security implications are indirect.
- jjav 4y ago> Container scanners and dependency bots like Dependabot are snake oil solutions sold to people too lazy to actually secure their application. Many years ago I'd have agreed with this so hard I'd hurt myself. These days I've moved on to a much more nuanced "maybe". Also many years ago the code might've had half a dozen dependencies and the developers knew quite well how they were used and what code paths were relevant now or in the near future. So it was fairly tractable to say: Nope, that doesn't impact us, ignore. These days there are thousands of dependencies and if I grab a random developer from the team and ask them "How is libfoo being used?" the response is most likely along the lines of "libfoo? never heard of it" because it was brought in as a third-degree dependency. So is the vulnerable functionality conceivably relevant or not? Nobody in the house knows. In such a world, it is quite often easiest to have a blanket policy of update/patch everything if at all possible. Only if some upgrade causes trouble, then spend human cycles to review and analyze what's up. I've seen teams that try to analyze everything fall further and further behind because vulnerabilities come in faster than they can analyze.
- 4y ago
- RcouF1uZ4gsC 4y agoContainers are often used to get around the fact that a lot of languages and environments make it hard or impossible to generate a single statically linked binary. Containers are at least as problematic for security updates as a statically linked binary and more inefficient.
- heelix 4y agoMan, I'm torn on this one. I work at a reasonably large shop where we are required to take these CVE scores seriously. I do see a bunch that are really not applicable every single day. (The dude from Mitre has earned a special place in hell for https://nvd.nist.gov/vuln/detail/cve-2016-1000027 https://nvd.nist.gov/vuln/detail/cve-2016-1000027) When I look at images where there are thousands of issues... it almost always is something end of life that is not supported. Our folks should not be using it. General rule of thumb, if you see an issue that you can patch - do so. There should be enough automation/testing so that minor version ++ is not a scary event. The real challenge is getting the business side to understand they need to allow time for updating. A SpringBoot 1.2 app might work just fine, from a functionality perspective. We try to use the CVE cleanup as a tech debt indicator to make sure folks are finding the time to upgrade from Java 5. Crazy, right? Most folks just use a docker image. Few actually build them or know enough to maintain someone elses. Surprisingly rare skill.
- pixl97 4y agoSeemingly all businesses of any size are using applications in production that are no longer being developed. Yea, I can take that application and upgrade that library... Will it work? Might crash. Might work fine. Or in the worst case silently corrupt data in an almost unnoticeable way that we don't notice for till 10 million transactions down the line.
- bravetraveler 4y agoI understand the dread people have for these scans, more noise than signal With that said, if your application doesn't need these things, why are they in there? Time and effort, usually. So there's a choice; continue chasing the vulns or you can improve your builds. You get out what you put in, the process can be valuable. It's not because cargo-culting.
- LinuxBender 4y agoHow many of these containers were pulled down from the web vs. being built in-house and slimmed down or built slim to begin with, meaning they only have the binary files absolutely required by the application?
- vidarh 4y agoUsing builder images and copying the bare essentials is usually what I aim to do. Not just for security, but because it's so much nicer to work with tiny images, no matter how much caching you do.
- mikedelago 4y agoIn my experience, the servers that hosted an application before it was in a container was much more insecure. An unfortunate amount of companies don't patch servers or upgrade the OS over time. In the past year, I've worked with Solaris from 2005 and CentOS 5, and migrated them to either Kubernetes or modern CentOS versions.