4 ms·

If you forget about a PHP container for a few years it will /also/ have 40 new vulnerabilities. Actually, containers are worse because OS updates of core shared

by shocks 3y ago

If you forget about a PHP container for a few years it will /also/ have 40 new vulnerabilities. Actually, containers are worse because OS updates of core shared libraries do nothing. You have to rebuild every damn container. Setting up monitoring for your docker containers is also a whole thing. :) I think you’re taking my example a little too literally. My point is not that docker/k8s/whatever is bad; just that the ‘new’ adds features at the cost of simplicity.

makeitdouble 3y ago
The biggest thing with containers is, these 40 vulnerabilities won't matter as much if they're about erasing your directories or killing your machine. It will get rebooted in a pristine state and an attacker would need to stick there killing it at every reboot to have lasting effect. Which is also the point of monitoring a container, which is fairly reliable nowadays, compared to managing your own health check service to ping your box to check if it's alive. All in all, I think k8s is way too complicated for what it does for most people, but administrating servers was also a complex task to begin with, and the things sys admins were dealing with looked nightmarish to me. Heck, there was a time companies would have their own SMTP in house...
- dumpsterdiver 3y ago
  > won't matter as much if they're about erasing your directories or killing your machine Those sort of attacks have tapered off though, right? Unless you're engaged in a specific feud, or have caught the ire of a social justice warrior, these days we're mostly looking at data exfiltration as the primary goal. That being said, there are a lot of ongoing feuds and active social justice warriors.
  - dns_snek 3y ago
    If we're talking about common, automated attacks, I think the game plan for the past few years has been to probe for systems vulnerable to RCE, install a crypto miner, join the server to a botnet and see how long it takes its owner to notice.
    - dumpsterdiver 3y ago
      Aha, how could I forget the crypto mining botnets. I agree.
- BirAdam 3y ago
  Layering more abstractions on top of what was already complicated doesn’t fix the complications below, it just hides them. You then have to hope that there’s not a bug somewhere in between those layers. Additionally, each new layer has a cost in performance. More layers means more compute, and eventually mean more money. If you’re Netflix, it’s worth it, if you’re not… what admins dealt with before wasn’t so bad. Now that that job has largely been killed and renamed 20 times, it’s far worse. Admins don’t just have to deal with a server, they have to deal with a server and 60 containers which are just little baby servers.
TeMPOraL 3y ago
I think what people are missing is that sure, code "rots", at the very least because of security patches. But since this all happens to everything simultaneously, the more distinct layers and support tools you have in your stack, the more often you have to deal with something breaking in a nontrivial way. In short: the more moving parts you have, the less time you have between major malfunctions. (Manufacturing and hardware world understands it well, which is a big part in why they like integrating things so much.) So e.g. over in the backend-land where I live, it used to be that I had to occasionally update the compiler or one of the few third-party dependencies that I used. Today, I have many more libraries (to the point there's something to update for security reasons roughly once a month, on average), and on top of that, I have CI/CD introducing its own mess, Conan updates which occasionally get messed up, or make some existing recipes incompatible, CMake updates which are done unexpectedly and break stuff, now also Docker is adding more of its own problems, etc. So I have to deal with some kind of tooling breakage every other week now. And always, always, when I think it's all finally OK and I can get on with my actual job, some forgotten or hidden component craps itself out of the blue. Like that time our git precommit hooks broke for me, because someone changed them in a way that doesn't work with my setup. And then me wasting a day on trying to fix it, eventually giving up and degrading my setup to unblock myself. Or another day where, for no apparent reason, some automation that made automated commits to some git repos started losing Change-ID headers in commit messages, making Gerrit very sad, leading to several people wasting a total of several person-days trying to fix it. Etc. There's always something breaking, the frequency of such breakages seems to be increasing, and it's a major source of frustration for me in this job. Which is why I too am often thinking back to "good old days", and am increasingly in favor of keeping the amount of dependencies - both libraries and tooling - to a minimum.
- shocks 3y ago
  Yea, it’s definitely about trading features for simplicity.
  - TeMPOraL 3y ago
    Right, and it's easy to think about all the winnings when things go right - it's harder to think about increased frequency and cost of failure due to increase in complexity and number of independently moving parts.
mrweasel 3y ago
> You have to rebuild every damn container. I can say with great certainty: Almost no one rebuilds their damn container anywhere near as often as the the gray-beard in the basement updates the Debian packages on the the server that runs the container. I had the debate with a client, we did monthly security update, unless something horrible happened. The client was rather upset that we didn't patch more frequently, like weekly or daily. My argumentation is that it doesn't really matter if the Linux kernel or bash is patched, when the only thing running is a container with a beta version of Tomcat that hasn't had security updates applied in three years. Even worse are the people who just pull things from Docker Hub, with no plan as to how and when they'll pull newer versions. But fine, let's just keep running KeyCloak from 2017, and that old Postgresql image which the developer never configured to do backups, I'm sure it's fine.
- mrighele 3y ago
  > I can say with great certainty: Almost no one rebuilds their damn container anywhere near as often as the the gray-beard in the basement updates the Debian packages on the the server that runs the container. Most probably the gray-beard simply enabled "unattended-upgrades". You can do something similar with a container (track the security fixes for the packages used, force rebuild and deploy when needed), but it is a bit of work and I don't know of any ready solution.

by shocks 3y ago

makeitdouble 3y ago

The biggest thing with containers is, these 40 vulnerabilities won't matter as much if they're about erasing your directories or killing your machine. It will get rebooted in a pristine state and an attacker would need to stick there killing it at every reboot to have lasting effect. Which is also the point of monitoring a container, which is fairly reliable nowadays, compared to managing your own health check service to ping your box to check if it's alive. All in all, I think k8s is way too complicated for what it does for most people, but administrating servers was also a complex task to begin with, and the things sys admins were dealing with looked nightmarish to me. Heck, there was a time companies would have their own SMTP in house...

dumpsterdiver 3y ago
> won't matter as much if they're about erasing your directories or killing your machine Those sort of attacks have tapered off though, right? Unless you're engaged in a specific feud, or have caught the ire of a social justice warrior, these days we're mostly looking at data exfiltration as the primary goal. That being said, there are a lot of ongoing feuds and active social justice warriors.
- dns_snek 3y ago
  If we're talking about common, automated attacks, I think the game plan for the past few years has been to probe for systems vulnerable to RCE, install a crypto miner, join the server to a botnet and see how long it takes its owner to notice.
  - dumpsterdiver 3y ago
    Aha, how could I forget the crypto mining botnets. I agree.
BirAdam 3y ago
Layering more abstractions on top of what was already complicated doesn’t fix the complications below, it just hides them. You then have to hope that there’s not a bug somewhere in between those layers. Additionally, each new layer has a cost in performance. More layers means more compute, and eventually mean more money. If you’re Netflix, it’s worth it, if you’re not… what admins dealt with before wasn’t so bad. Now that that job has largely been killed and renamed 20 times, it’s far worse. Admins don’t just have to deal with a server, they have to deal with a server and 60 containers which are just little baby servers.

TeMPOraL 3y ago

I think what people are missing is that sure, code "rots", at the very least because of security patches. But since this all happens to everything simultaneously, the more distinct layers and support tools you have in your stack, the more often you have to deal with something breaking in a nontrivial way. In short: the more moving parts you have, the less time you have between major malfunctions. (Manufacturing and hardware world understands it well, which is a big part in why they like integrating things so much.) So e.g. over in the backend-land where I live, it used to be that I had to occasionally update the compiler or one of the few third-party dependencies that I used. Today, I have many more libraries (to the point there's something to update for security reasons roughly once a month, on average), and on top of that, I have CI/CD introducing its own mess, Conan updates which occasionally get messed up, or make some existing recipes incompatible, CMake updates which are done unexpectedly and break stuff, now also Docker is adding more of its own problems, etc. So I have to deal with some kind of tooling breakage every other week now. And always, always, when I think it's all finally OK and I can get on with my actual job, some forgotten or hidden component craps itself out of the blue. Like that time our git precommit hooks broke for me, because someone changed them in a way that doesn't work with my setup. And then me wasting a day on trying to fix it, eventually giving up and degrading my setup to unblock myself. Or another day where, for no apparent reason, some automation that made automated commits to some git repos started losing Change-ID headers in commit messages, making Gerrit very sad, leading to several people wasting a total of several person-days trying to fix it. Etc. There's always something breaking, the frequency of such breakages seems to be increasing, and it's a major source of frustration for me in this job. Which is why I too am often thinking back to "good old days", and am increasingly in favor of keeping the amount of dependencies - both libraries and tooling - to a minimum.

shocks 3y ago
Yea, it’s definitely about trading features for simplicity.
- TeMPOraL 3y ago
  Right, and it's easy to think about all the winnings when things go right - it's harder to think about increased frequency and cost of failure due to increase in complexity and number of independently moving parts.

mrweasel 3y ago

> You have to rebuild every damn container. I can say with great certainty: Almost no one rebuilds their damn container anywhere near as often as the the gray-beard in the basement updates the Debian packages on the the server that runs the container. I had the debate with a client, we did monthly security update, unless something horrible happened. The client was rather upset that we didn't patch more frequently, like weekly or daily. My argumentation is that it doesn't really matter if the Linux kernel or bash is patched, when the only thing running is a container with a beta version of Tomcat that hasn't had security updates applied in three years. Even worse are the people who just pull things from Docker Hub, with no plan as to how and when they'll pull newer versions. But fine, let's just keep running KeyCloak from 2017, and that old Postgresql image which the developer never configured to do backups, I'm sure it's fine.

mrighele 3y ago
> I can say with great certainty: Almost no one rebuilds their damn container anywhere near as often as the the gray-beard in the basement updates the Debian packages on the the server that runs the container. Most probably the gray-beard simply enabled "unattended-upgrades". You can do something similar with a container (track the security fixes for the packages used, force rebuild and deploy when needed), but it is a bit of work and I don't know of any ready solution.