3 ms·

This is an insightful post. Deploying production applications in containers with an auto-updating kernel underneath still has to be proven in the real world. I

This is an insightful post. Deploying production applications in containers with an auto-updating kernel underneath still has to be proven in the real world. I do want to quibble with one point, however: "On top of that, by using such a specialized system to run your apps, you lose all the flexibility of having a full linux OS to troubleshoot and debug from. You now have to rely on them building on all the components that already exist in regular Linux world, like debuggers, tracers, sniffers, profilers, etc. You'll have to slip all that into your application deploy to troubleshoot a weird one-off bug." CoreOS is a full fledged linux, and since applications are running in containers, there is no reason you couldn't use debugging tools on the host. Most production services have strict controls anyway, so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away.

peterwwillis 13y ago
The complete lack of any detail about how the system actually works may have confused me. I assumed they packaged an application with its dependencies and deployed it as one big piece, similar to (or using) LXC. From what I understand about LXC, you have to create a chroot environment for your service to run in. This means installing applications in the chroot environment in order to use them from the application. For various types of debugging/troubleshooting, this may be necessary, as the resources of the environment and its behavior may be (read: are guaranteed to be) different from that of the host OS. I don't know what kind of environments you work in, but "strict controls" go out the window when the production site is randomly going down and you're losing millions of dollars in revenue. When all hell breaks loose, you dig in your heels and debug the app server while it is crashing, with a developer sitting next to you and three fuming managers behind your chairs. In this scenario i'd rather have a plain old fat-ass Linux distro than a clunky "minimal" container manager.
spudlyo 13y ago
... so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away. This may happen more often than you think. There is a technique popularized by some MySQL hackers at Facebook called "poor man's query profiling" that uses gdb to (among other things) dump the the stack traces of every MySQL thread. Some awk normalizes, aggregates and sorts the traces. I often do this when I encounter a badly flailing or completely wedged MySQL daemon. It's a good way to see inside MySQL, and if a bunch of threads are all blocked on a mutex or something it's pretty obvious.
vidarh 13y ago
> Most production services have strict controls anyway, so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away. You'd be surprised. Outside of large companies with dedicated devops, it is very common to see stuff like that happening on production machines. But that's a use case that stuff like this is ideal for: Clone the container. Install gdb in clone, and try to reproduce the problem.

peterwwillis 13y ago

The complete lack of any detail about how the system actually works may have confused me. I assumed they packaged an application with its dependencies and deployed it as one big piece, similar to (or using) LXC. From what I understand about LXC, you have to create a chroot environment for your service to run in. This means installing applications in the chroot environment in order to use them from the application. For various types of debugging/troubleshooting, this may be necessary, as the resources of the environment and its behavior may be (read: are guaranteed to be) different from that of the host OS. I don't know what kind of environments you work in, but "strict controls" go out the window when the production site is randomly going down and you're losing millions of dollars in revenue. When all hell breaks loose, you dig in your heels and debug the app server while it is crashing, with a developer sitting next to you and three fuming managers behind your chairs. In this scenario i'd rather have a plain old fat-ass Linux distro than a clunky "minimal" container manager.

spudlyo 13y ago

... so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away. This may happen more often than you think. There is a technique popularized by some MySQL hackers at Facebook called "poor man's query profiling" that uses gdb to (among other things) dump the the stack traces of every MySQL thread. Some awk normalizes, aggregates and sorts the traces. I often do this when I encounter a badly flailing or completely wedged MySQL daemon. It's a good way to see inside MySQL, and if a bunch of threads are all blocked on a mutex or something it's pretty obvious.

vidarh 13y ago

> Most production services have strict controls anyway, so it isn't like it is common practice to log in to a production database server and do apt-get/yum install gdb and start banging away. You'd be surprised. Outside of large companies with dedicated devops, it is very common to see stuff like that happening on production machines. But that's a use case that stuff like this is ideal for: Clone the container. Install gdb in clone, and try to reproduce the problem.