4 ms·

Yet another list of potentially extremely useful concepts that I'll never actually have the time to fully verify. I wonder where all this complexity ends. If a

Yet another list of potentially extremely useful concepts that I'll never actually have the time to fully verify. I wonder where all this complexity ends. If a human can't fully grok the systems we work on, then there is no way we can hope to not be misled and taken advantage of. Does anyone else share these concerns?

ButterWashed 6y ago
I feel this way every time I read an OpenSCAP or CIS lockdown policy. They're hundreds of pages of "recommendations" that I think I can trust but don't fully understand. I don't know if I should worry about my lack of understanding or not.
- blowski 6y ago
  My Grandad was an illiterate farmer. He didn’t understand the technology he used either, but it didn’t matter as long as it worked or he could call someone to fix it. Somebody who lives 100% this stuff will abstract it and solve the problem for you, just as others did then. But do consider the list when some junior says they can get the company billing system running on Docker on prod.
3np 6y ago
As a docker user, most of this aligns with my understanding, it’s mostly straight forward stuff and not inherently complex. Try SELinux and see if you think this is still complicated ;) ADD vs COPY was new to me, though. ...More in general, I think it’s not the systems that are complex but our minds/brains that are limited to deal with it. Just look at the complexity in nature.
the8472 6y ago
As always, it depends? If you're only using docker for development to bundle things in a different OS base image then there's not that much need for paranoia. You probably would trust the other OS vendor with your host system too. Running docker without security may be perfectly fine in that case. For random scripts from github running them in docker is mostly a precaution so they don't screw up your host system or some malicious dependency makes a low-effort attempt at exfiltrating your ~/.ssh dir or whatever. In that case it makes sense to understand the basics, e.g. not just running docker --privileged willy-nilly just because some README asks you to. If you're running some PaaS container service where users can execute arbitrary code on shared hardware then you better have a full-time security engineer or two who will be paid to understanding all the details involved.
deadbytes 6y ago
A large majority of people in tech seem to have convinced themselves that you solve complexity by adding even more complexity ontop. Yes, linux security is complicated. So why not work on improving this instead? Optimise and simplify what we already have. Docker hasn't made the complexity of linux security go away. It has just added a whole other dimension of potential security issues people now need to manage in addition to the security of the base system. Programmers need to shift their mindset. We already have far too much complexity. Stop thinking about what new things you can create, start thinking about how you can improve and simplify the software that we already have.
- paulryanrogers 6y ago
  Docker solves a few other problems too, like conflicting dependencies in user space. Still, I agree the broader point that one can't solve too much abstraction with more abstraction.
- smw 6y ago
  It has gotten simpler, especially for things like testing safely. If you don't think about it as docker, or containers, the ability to sandbox things with namespaces and cgroups is almost magical. It's effectively instant and much more effective than just chroot. "Run something with no -- or very specific -- network access" was a really annoying problem to solve (LD_PRELOAD?) in the before times.
  - deadbytes 6y ago
    The abstraction that docker provides you is simpler, but the whole system has gotten more complex. >Run something with no, or very specific, network access If this was such a problem in linux then why didn't people focus on improving this instead? We could have solved this problem in linux and made the whole system better for everyone. Instead people left the problem there and just piled more crud ontop. We added an entire new layer of abstraction that everyone now has to spend weeks learning how to use instead of just fixing the original problem. The whole system is now far more complex, and the original problem is still there.
bob1029 6y ago
I absolutely share these concerns. The art of software engineering is about managing complexity. The best way to manage something is to reduce the amount of it you have to worry about. For the rest, paying really close attention, accepting ownership and directly engaging is key. Reducing the number of parties you have to trust is typically a happy side-effect of reducing complexity. I look at containerization as an attempt to hand-wave away critical responsibilities around ownership of complexity in an application. There are tons of examples that illustrate this throughout the ecosystem, but I think this one is most apt - One day you discover your application is a difficult mess to reconstruct from source each time. You have reached a fork in the road because management is complaining that it takes 2 weeks to configure a new QA environment from scratch. Do you either: A) Review fundamental assumptions about the problem domain, technology choices and teamwork. Potentially consider rewriting your application from scratch using fewer tools & computers, and with more focus on driving the actual business value equations. Or, B) Decide that Tom's computer is the new golden image of production and bless it as such. Now let's find a way to manage a whole farm of these things!
- geofft 6y ago
  The actual answer people come up with is neither of those. The process of moving your application to run in containers generally doesn't involve snapshotting Tom's computer - it usually wouldn't even work inside Docker for myriad reasons. And the Dockerfile format is a good way of capturing what it actually takes to reconstruct your application/environment from source in a manageable, version-control-able text file. I do agree that the rush to containerization is generally a rush to draw expansive abstraction boundaries, but this particular story isn't what people are doing. (At least not with containers - it's certainly what people were doing with VM images ten years ago!)
closeparen 6y ago
I think what you should take away from this is that containers are not the abstraction you want for running mutually distrusting workloads, and you can continue to use separate VMs or even separate physical networks of bare metal servers. Some people will geek out about this quixotic undertaking; you don’t have to be one of them. Linux is not very good at security boundaries anyway, just run one thing in each VM and don’t leave anything else there to privilege-escalate into.
eeZah7Ux 6y ago
Yes. Stay away from Docker or anything so complex. systemd-nspawn does the same things and can be sandboxed very effectively by the unit files.
nikisweeting 6y ago
As a sysadmin that that worked on securing linux apps before docker was a thing, imo doing it with docker is a lot less complex than before. No tech is perfect, but docker has dramatically reduced the surface area of tunables that my application devs have to care about in order to get our product shipped. Even if it's not brought it down to zero, it's a step in the right direction in terms of UX complexity exposed to the developer.

ButterWashed 6y ago

I feel this way every time I read an OpenSCAP or CIS lockdown policy. They're hundreds of pages of "recommendations" that I think I can trust but don't fully understand. I don't know if I should worry about my lack of understanding or not.

blowski 6y ago
My Grandad was an illiterate farmer. He didn’t understand the technology he used either, but it didn’t matter as long as it worked or he could call someone to fix it. Somebody who lives 100% this stuff will abstract it and solve the problem for you, just as others did then. But do consider the list when some junior says they can get the company billing system running on Docker on prod.

3np 6y ago

As a docker user, most of this aligns with my understanding, it’s mostly straight forward stuff and not inherently complex. Try SELinux and see if you think this is still complicated ;) ADD vs COPY was new to me, though. ...More in general, I think it’s not the systems that are complex but our minds/brains that are limited to deal with it. Just look at the complexity in nature.

the8472 6y ago

As always, it depends? If you're only using docker for development to bundle things in a different OS base image then there's not that much need for paranoia. You probably would trust the other OS vendor with your host system too. Running docker without security may be perfectly fine in that case. For random scripts from github running them in docker is mostly a precaution so they don't screw up your host system or some malicious dependency makes a low-effort attempt at exfiltrating your ~/.ssh dir or whatever. In that case it makes sense to understand the basics, e.g. not just running docker --privileged willy-nilly just because some README asks you to. If you're running some PaaS container service where users can execute arbitrary code on shared hardware then you better have a full-time security engineer or two who will be paid to understanding all the details involved.

deadbytes 6y ago

A large majority of people in tech seem to have convinced themselves that you solve complexity by adding even more complexity ontop. Yes, linux security is complicated. So why not work on improving this instead? Optimise and simplify what we already have. Docker hasn't made the complexity of linux security go away. It has just added a whole other dimension of potential security issues people now need to manage in addition to the security of the base system. Programmers need to shift their mindset. We already have far too much complexity. Stop thinking about what new things you can create, start thinking about how you can improve and simplify the software that we already have.

paulryanrogers 6y ago
Docker solves a few other problems too, like conflicting dependencies in user space. Still, I agree the broader point that one can't solve too much abstraction with more abstraction.
smw 6y ago
It has gotten simpler, especially for things like testing safely. If you don't think about it as docker, or containers, the ability to sandbox things with namespaces and cgroups is almost magical. It's effectively instant and much more effective than just chroot. "Run something with no -- or very specific -- network access" was a really annoying problem to solve (LD_PRELOAD?) in the before times.
- deadbytes 6y ago
  The abstraction that docker provides you is simpler, but the whole system has gotten more complex. >Run something with no, or very specific, network access If this was such a problem in linux then why didn't people focus on improving this instead? We could have solved this problem in linux and made the whole system better for everyone. Instead people left the problem there and just piled more crud ontop. We added an entire new layer of abstraction that everyone now has to spend weeks learning how to use instead of just fixing the original problem. The whole system is now far more complex, and the original problem is still there.

bob1029 6y ago

I absolutely share these concerns. The art of software engineering is about managing complexity. The best way to manage something is to reduce the amount of it you have to worry about. For the rest, paying really close attention, accepting ownership and directly engaging is key. Reducing the number of parties you have to trust is typically a happy side-effect of reducing complexity. I look at containerization as an attempt to hand-wave away critical responsibilities around ownership of complexity in an application. There are tons of examples that illustrate this throughout the ecosystem, but I think this one is most apt - One day you discover your application is a difficult mess to reconstruct from source each time. You have reached a fork in the road because management is complaining that it takes 2 weeks to configure a new QA environment from scratch. Do you either: A) Review fundamental assumptions about the problem domain, technology choices and teamwork. Potentially consider rewriting your application from scratch using fewer tools & computers, and with more focus on driving the actual business value equations. Or, B) Decide that Tom's computer is the new golden image of production and bless it as such. Now let's find a way to manage a whole farm of these things!

geofft 6y ago
The actual answer people come up with is neither of those. The process of moving your application to run in containers generally doesn't involve snapshotting Tom's computer - it usually wouldn't even work inside Docker for myriad reasons. And the Dockerfile format is a good way of capturing what it actually takes to reconstruct your application/environment from source in a manageable, version-control-able text file. I do agree that the rush to containerization is generally a rush to draw expansive abstraction boundaries, but this particular story isn't what people are doing. (At least not with containers - it's certainly what people were doing with VM images ten years ago!)

closeparen 6y ago

I think what you should take away from this is that containers are not the abstraction you want for running mutually distrusting workloads, and you can continue to use separate VMs or even separate physical networks of bare metal servers. Some people will geek out about this quixotic undertaking; you don’t have to be one of them. Linux is not very good at security boundaries anyway, just run one thing in each VM and don’t leave anything else there to privilege-escalate into.

eeZah7Ux 6y ago

Yes. Stay away from Docker or anything so complex. systemd-nspawn does the same things and can be sandboxed very effectively by the unit files.

nikisweeting 6y ago

As a sysadmin that that worked on securing linux apps before docker was a thing, imo doing it with docker is a lot less complex than before. No tech is perfect, but docker has dramatically reduced the surface area of tunables that my application devs have to care about in order to get our product shipped. Even if it's not brought it down to zero, it's a step in the right direction in terms of UX complexity exposed to the developer.