3 ms·

"Simple", then "apt upgrade" and "reboot" inside the same statement makes me believe that you're thinking small, like "my own server with my own self-written PH

"Simple", then "apt upgrade" and "reboot" inside the same statement makes me believe that you're thinking small, like "my own server with my own self-written PHP application". Which is not wrong, but that's not how tech scales. How would you apply that procedure to 1,000s of servers, in the right order, to prevent downtime? How much would it take? Won't it be far better to create a new container image, run all the testing on it, and deliver it everywhere with the already-existing upgrade procedure for your cluster?

orev 4y ago
This sounds like container kool-aid to me. One can implement exactly the same processes using regular OS management tools as with container tools. They both still need the same information about order, testing, etc. If you’ve only ever put the effort towards doing that in a container instead of at the OS level, then obviously your container procedure will be better because you already built it. Someone who already built it at the OS level will also have similar ease. (And those containers still need to run somewhere, and that somewhere invariably has an OS involved, so you always need the OS update process no matter what your container strategy looks like). But you’re also thinking “small” in the sense that you’re only focused on your own app. I find it striking how so many developers these days have simply forgotten that they’re not the only ones who actually do tech stuff, and how limited their views are. Running IT at a company involves hundreds of applications that are not built by you. Just because they’re not the main thing you sell doesn’t mean you can ignore that they exist.
- alanfranz 4y ago
  "Entering" a server and running "apt dist-upgrade" probably means you've got a stateful fleet, because some package upgrade can fail, it can be performed at different time from one server to another, et cetera. Your servers should be cattle, not pet. So, you should instead create a sort of manifest/snapshot/checkpoint with all your deps pinned, and distribute that to your servers. You can do the same without containerization - you could use a fleet of machines and keep their versions pinned with a configuration management tool, but you'd still not want to manually perform an apt upgrade on your server. And manually "check that everything is fine and maybe reboot" to verify your new shared lib was loaded - you should already have a precise, reproducible, automated testing and upgrade procedure and just apply that. So, containerization is not the point here. Automation of deployment and testing procedures is. But the OP was specifically attacking containerization (as if it were the problem), and I was answering him. But, containerization is generally useful here, because the "container image" is, generally speaking, a good, well-defined artifact. So the OP's example was especially bad - telling "machine X is vulnerable to SSL bug Y" is hard, requires specific machine instance inspection: what happens if the process/server wasn't restarted and, even though package is updated, it's still using an old ssl version? On the contrary, "this is using container image version X with openssl version Y which is not vulnerable" is a very precise True/False statement, easy to inspect and verify at scale.
xena 4y ago
OP here, most of my advice later in the article is aimed at people running smaller services on smaller scales. I'm (probably poorly) assuming that people at larger scales know what they are doing.
- alanfranz 4y ago
  For OP I meant franga2000 btw. Your article is good IMHO.
franga2000 4y ago
I'm not thinking 1000s of servers, because that is extremely rare, but this definitely doesn't apply to only non-critical private stuff. Many orgs have up to a dozen servers, each running multiple services. If you have multiple replicas of certain servers, you can reboot them (or just restart the impacted services) one at a time with no disruption. The thing is, you usually aren't building everything from scratch. A pretty typical containerized web app deployment might consist of postgres, redis, elasticsearch, nginx, task queue and app server containers. Each of these needs to be updated and you're probably only building two of them from Dockerfiles. Instead of just your host distro, you're now waiting on the maintainers of all of these images to push updates that you can pull. Big distros get updates early so they can push them out the instant the embargo lifts, but I doubt all of these Docker packagers and statically-built software maintainers do. You're either stuck waiting or you have to build all of them from scratch.
- alanfranz 4y ago
  If you’re a small org with just a dozen servers and you’re overcomplicating your life with hundreds of containers, then I agree with you. But you can’t attack containerization if it’s used by proper orgs with the right scale.

orev 4y ago

This sounds like container kool-aid to me. One can implement exactly the same processes using regular OS management tools as with container tools. They both still need the same information about order, testing, etc. If you’ve only ever put the effort towards doing that in a container instead of at the OS level, then obviously your container procedure will be better because you already built it. Someone who already built it at the OS level will also have similar ease. (And those containers still need to run somewhere, and that somewhere invariably has an OS involved, so you always need the OS update process no matter what your container strategy looks like). But you’re also thinking “small” in the sense that you’re only focused on your own app. I find it striking how so many developers these days have simply forgotten that they’re not the only ones who actually do tech stuff, and how limited their views are. Running IT at a company involves hundreds of applications that are not built by you. Just because they’re not the main thing you sell doesn’t mean you can ignore that they exist.

alanfranz 4y ago
"Entering" a server and running "apt dist-upgrade" probably means you've got a stateful fleet, because some package upgrade can fail, it can be performed at different time from one server to another, et cetera. Your servers should be cattle, not pet. So, you should instead create a sort of manifest/snapshot/checkpoint with all your deps pinned, and distribute that to your servers. You can do the same without containerization - you could use a fleet of machines and keep their versions pinned with a configuration management tool, but you'd still not want to manually perform an apt upgrade on your server. And manually "check that everything is fine and maybe reboot" to verify your new shared lib was loaded - you should already have a precise, reproducible, automated testing and upgrade procedure and just apply that. So, containerization is not the point here. Automation of deployment and testing procedures is. But the OP was specifically attacking containerization (as if it were the problem), and I was answering him. But, containerization is generally useful here, because the "container image" is, generally speaking, a good, well-defined artifact. So the OP's example was especially bad - telling "machine X is vulnerable to SSL bug Y" is hard, requires specific machine instance inspection: what happens if the process/server wasn't restarted and, even though package is updated, it's still using an old ssl version? On the contrary, "this is using container image version X with openssl version Y which is not vulnerable" is a very precise True/False statement, easy to inspect and verify at scale.

xena 4y ago

OP here, most of my advice later in the article is aimed at people running smaller services on smaller scales. I'm (probably poorly) assuming that people at larger scales know what they are doing.

alanfranz 4y ago
For OP I meant franga2000 btw. Your article is good IMHO.

franga2000 4y ago

I'm not thinking 1000s of servers, because that is extremely rare, but this definitely doesn't apply to only non-critical private stuff. Many orgs have up to a dozen servers, each running multiple services. If you have multiple replicas of certain servers, you can reboot them (or just restart the impacted services) one at a time with no disruption. The thing is, you usually aren't building everything from scratch. A pretty typical containerized web app deployment might consist of postgres, redis, elasticsearch, nginx, task queue and app server containers. Each of these needs to be updated and you're probably only building two of them from Dockerfiles. Instead of just your host distro, you're now waiting on the maintainers of all of these images to push updates that you can pull. Big distros get updates early so they can push them out the instant the embargo lifts, but I doubt all of these Docker packagers and statically-built software maintainers do. You're either stuck waiting or you have to build all of them from scratch.

alanfranz 4y ago
If you’re a small org with just a dozen servers and you’re overcomplicating your life with hundreds of containers, then I agree with you. But you can’t attack containerization if it’s used by proper orgs with the right scale.