8 ms·

Two Objects Not Namespaced by the Linux Kernel (2017)

169 points by setra 8y ago

lalaithion 8y ago
Why is this the case? No one has bothered to do it? It would break backwards compatibility? Linus thinks it's a bad idea?
- jchw 8y ago
  Shouldn't break backwards compatibility. More than anything, my guess is that it's just a result of most of Linux's modern day design having been implemented before the era of containers. Afaik, namespaces+cgroups were never meant to support complete isolation.
- emmelaich 8y ago
  Probably merely because it's hard to do and no one has sufficient motivation. I can think of one good use case -- y2k style problems. Also sometimes apps are tied to external events like legislation. It would be good to set the time forward for testing. You can sort of do this with LD_PRELOAD but it can get hairy. Also see @wmf's comment above.
  - etaoins 8y ago
    Another use case is dealing with tokens that assume globally synchronised clocks such as JWTs and Kerberos/Active Directory. Ideally all clocks would be perfectly synchronised but things happen. For example, you might have one container that’s exchanging JWTs with a micro service that should be using AWS’s NTP servers and another that’s joined an Active Directory domain that should be using the AD NTP server. Right now you either need to run them on separate machines or expose yourself to interesting problems if clock skew happens.
  - briffle 8y ago
    If there is one thing Y2K taught us, its to ignore any worry about the 2038 problem until 2036, then make a HUGE deal out of it. https://en.wikipedia.org/wiki/Year_2038_problem https://en.wikipedia.org/wiki/Year_2038_problem
    - cyphar 8y ago
      Linux and glibc have been working on 2038 problems for at least the past decade.
      - pjmlp 8y ago
        There are plenty of other POSIX platforms out there.
- simcop2387 8y ago
  The time namespace is being worked on, it's a very difficult problem because of how pervasive time is in the kernel. Here's a recent in depth LWN article about the topic. https://lwn.net/Articles/766089/ https://lwn.net/Articles/766089/ They keychain stuff I haven't heard about any work being done but I don't know any reason it shouldn't be doable.
wmf 8y ago
Since this was written a time namespace was proposed: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Time-Namespace-RFC https://www.phoronix.com/scan.php?page=news_item&px=Linux-Ti...
- DonHopkins 8y ago
  This proposal implements clock offsets, but does it support continuous time scaling? One clock-handy use case would be to run your programs really slow or fast (or backwards!), for testing purposes. Kaleida Lab's ScriptX (a multimedia programing language kinda like Dylan with classes) had built-in support for hierarchal clocks within the container (in the sense of "window" not "vm") hierarchy. The same way every window or node has a 2D or 3D transformation matrix, each clock has a time scale and offset relative to its parent, so anything that consumes time (like a QuickTime player, or a simulation) runs at the scaled and offset time that it inherits from through its parent containers. And you can move and scale each container around in time as necessary, to pause movies or simulations.) You could even set the scale to a negative number, and it played QuickTime movies backwards! (That was pretty cool in 1995 -- try playing a movie backwards in the web browser today!) http://www.art.net/~hopkins/Don/lang/scriptx/tech-qa.html http://www.art.net/~hopkins/Don/lang/scriptx/tech-qa.html Q: How does the ScriptX core class library compare to class libraries available with other programming languages (e.g. MFC, OWL, MacApp, or TCL)? A: The ScriptX core class library has many similarities to other object oriented frameworks in that it provides many basic services common to all applications built on them. All frameworks provide classes for creating windows, handling keyboard and mouse events, reading and writing files, etc. Where ScriptX is unique is in its focus. The ScriptX core classes are oriented towards interactive, media rich applications. For example, clocks and timing are fundamental in the ScriptX class library; most other frameworks have no concept of timing built in. ScriptX also tightly integrates media data (bitmaps, video, audio) with the class library, and hides the details of storing, retrieving, and presenting media to the user. Q: What are the major sections of the core class library? Clocks, players, and animation. Time is a fundamental element of the core classes. Starting with basic clocks, subclasses extend the capabilities for animation, video, and audio playback. Clocks can be tied to underlying hardware clocks or slaved in a hierarchical fashion to other clock objects. Varying the rate of a master clock, all sub-clocks will stay synchronized to the master clock, permitting the programmer to precisely control time in a title. Clock hierarchies also free the programmer from dealing with differences in performance between slower and faster CPUs. Player classes build upon clocks. These classes allow you to create and play sequences of actions that take place over time. These sequences can be used to create animations as well as control other presentation elements such as video or sound. A special type of clock object, the action list player, can be used to play actions in sequence at specified times. Various action objects are added to an action list specifying the time at which the action is to occur. Action objects are used to move graphic elements on the screen, execute ScriptX code, or modify the action list. Other player classes provide simple ways to play digital video, audio, and MIDI. As with all players, the clocks underlying these players can be sped up, slowed down, or run backwards. Q: Can video be synchronized with other events? A: Yes, internal video players are based on ScriptX clocks and can be slaved together to provide synchronization with animations and other events. For example, buttons can appear in a window at precise times based on video playback.
tyingq 8y ago
Syslog seems to be on the proposal list as well.
derefr 8y ago
I wonder whether namespacing time would also result in those namespaces being able to have separate "clocks" (time backends? time schedulers?) that progress at different rates, or for different reasons. Being able to put a process into a time namespace with a deterministic "clock" would obviate a large benefit of http://www.zerovm.org/ http://www.zerovm.org/. Also, having "clock slew" be a matter of perspective—with processes that can handle leap seconds seeing them happen instantaneously; and processes that can't handle leap-seconds, seeing slewed time—would be nice. Then you could have different system facilities that care about monotonic time, vs. synced to calendar time, vs. one second per second time, all having that kind of time available to them as "the time", rather than through different APIs.
- kbenson 8y ago
  > Also, having "clock slew" be a matter of perspective—with processes that can handle leap seconds seeing them happen instantaneously; and processes that can't handle leap-seconds, seeing slewed time—would be nice. I imagine there might be some really interesting (for meanings of interesting that include shoot me now) and hard to track down bugs as you deal with inconsistent clocks not just across systems within a network, but processes within a single system.
  - chatmasta 8y ago
    IIRC Docker for Mac had a bug like this for a long time where the clocks of containers would become wildly out of date.
    - TheDong 8y ago
      More accurately, the clocks of the linux virtual machine running docker containers would differ from the OSX clock. Those aren't really containers skewing from other processes on the same system as the parent describes, but of clocks skewing on two different systems (which is a totally normal thing we deal with regularly).
  - derefr 8y ago
    > I imagine there might be some really interesting (for meanings of interesting that include shoot me now) and hard to track down bugs as you deal with inconsistent clocks not just across systems within a network, but processes within a single system. I feel like the "safe assumption" that the other end of a given IPC channel (or even inter-thread communication channel) is on the same machine, is responsible for the vast majority of failures we see in e.g. Jepsen testing of databases. After all, in sufficiently-large computers (i.e. HPC clusters that pretend to be one "computer"), you've got NUMA zones that are light-microseconds away from one another, where even threads of the same process can literally end up needing vector clocks to linearize events between themselves. It probably wouldn't be too bad a thing if things like the Linux base-system used only internal IPC mechanisms that exposed this unreliability (like e.g. Erlang does with "unreliable async message passing" as its IPC primitive), forcing each component to deal with the fact that its peers may or may not be netsplit away from it. Even if that scenario will only come up if you're writing code to get your GPS position from a Dyson sphere of 10-mile-deep Matryoska brains.
timeattack 8y ago
You can't change time in container, but it's possible to change timezone files. With generating fake timezones it is possible to change time in container.
- cyphar 8y ago
  This doesn't change what gettimeofday(2) gives you (and actually you can't even use ptrace easily to fake the time of day because gettimeofday(2) isn't a real syscall -- it's actually implemented as a read from the vDSO page the kernel maps into every process).
haberman 8y ago
> The current set of namespaces in the kernel are: mount, pid, uts, ipc, net, user, and cgroup. [...] [Time is] not namespaced. [...] The kernel keyring is another item not namespaced. I've always argued that "everything is a file" is an exaggeration. These moments make the extent of that exaggeration clear. If everything truly was a file, the only thing you would need to namespace is the filesystem. But in reality there are a lot of other kernel objects that are not files at all.
- tyingq 8y ago
  I agree it's unfortunate, but it doesn't really seem in conflict with "everything is a file". Making it a file is separate from making it sensible/usable for containers. Like the /proc filesystem. They are "files", but don't many don't work as expected without something like lxcfs. Like /proc/uptime, for example.
- madhadron 8y ago
  Everything is a file hasn't been true for Unix since almost the beginning. It's kind of like the Unix philosophy of small, independent tools...except for the database where you store all the important data.
- zapita 8y ago
  You are 100% correct. “Everything is a file” was more of an early design insight, which was gradually abandoned as new features were added. There is a movement of “Unix purists” who lament this deviation from founding principles, and advocate for a return to them. The most notable example is Plan 9. In Plan 9, everything actually is a file. And exactly as you said, all resources are namespaced via the filesystem. It’s quite elegant and practical. Sadly Plan 9 has remained a fringe OS, and although it influenced mainstream operating system design in many ways (including the concept of /proc), I wish that influence had been stronger.
  - temac 8y ago
    hm is NT a purer Unix than Unix then? After all, it has all its object in a filesystem-like tree...
vxNsr 8y ago
meta: This is from 2017, Super interesting though, the keyring thing especially seems to have broader implications...
theamk 8y ago
I personally miss core pattern namespacing. I would love to give some of my containers a custom coredump handler, but this is impossible. And in general, a sysctls settings namespace would be really useful. Sure, sometimes it makes no sense to namespace a setting, but net.ipv4.tcp_congestion_control for example? I'd love to be able to change it without modifying the code.
Sharlin 8y ago
I’m not sure that people who think ”containers are just like VMs” should have any business working with containers.

169 points by setra 8y ago

Why is this the case? No one has bothered to do it? It would break backwards compatibility? Linus thinks it's a bad idea?

jchw 8y ago
Shouldn't break backwards compatibility. More than anything, my guess is that it's just a result of most of Linux's modern day design having been implemented before the era of containers. Afaik, namespaces+cgroups were never meant to support complete isolation.
emmelaich 8y ago
Probably merely because it's hard to do and no one has sufficient motivation. I can think of one good use case -- y2k style problems. Also sometimes apps are tied to external events like legislation. It would be good to set the time forward for testing. You can sort of do this with LD_PRELOAD but it can get hairy. Also see @wmf's comment above.
- etaoins 8y ago
  Another use case is dealing with tokens that assume globally synchronised clocks such as JWTs and Kerberos/Active Directory. Ideally all clocks would be perfectly synchronised but things happen. For example, you might have one container that’s exchanging JWTs with a micro service that should be using AWS’s NTP servers and another that’s joined an Active Directory domain that should be using the AD NTP server. Right now you either need to run them on separate machines or expose yourself to interesting problems if clock skew happens.
- briffle 8y ago
  If there is one thing Y2K taught us, its to ignore any worry about the 2038 problem until 2036, then make a HUGE deal out of it. https://en.wikipedia.org/wiki/Year_2038_problem https://en.wikipedia.org/wiki/Year_2038_problem
  - cyphar 8y ago
    Linux and glibc have been working on 2038 problems for at least the past decade.
    - pjmlp 8y ago
      There are plenty of other POSIX platforms out there.
simcop2387 8y ago
The time namespace is being worked on, it's a very difficult problem because of how pervasive time is in the kernel. Here's a recent in depth LWN article about the topic. https://lwn.net/Articles/766089/ https://lwn.net/Articles/766089/ They keychain stuff I haven't heard about any work being done but I don't know any reason it shouldn't be doable.

wmf 8y ago

Since this was written a time namespace was proposed: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Time-Namespace-RFC https://www.phoronix.com/scan.php?page=news_item&px=Linux-Ti...

DonHopkins 8y ago
This proposal implements clock offsets, but does it support continuous time scaling? One clock-handy use case would be to run your programs really slow or fast (or backwards!), for testing purposes. Kaleida Lab's ScriptX (a multimedia programing language kinda like Dylan with classes) had built-in support for hierarchal clocks within the container (in the sense of "window" not "vm") hierarchy. The same way every window or node has a 2D or 3D transformation matrix, each clock has a time scale and offset relative to its parent, so anything that consumes time (like a QuickTime player, or a simulation) runs at the scaled and offset time that it inherits from through its parent containers. And you can move and scale each container around in time as necessary, to pause movies or simulations.) You could even set the scale to a negative number, and it played QuickTime movies backwards! (That was pretty cool in 1995 -- try playing a movie backwards in the web browser today!) http://www.art.net/~hopkins/Don/lang/scriptx/tech-qa.html http://www.art.net/~hopkins/Don/lang/scriptx/tech-qa.html Q: How does the ScriptX core class library compare to class libraries available with other programming languages (e.g. MFC, OWL, MacApp, or TCL)? A: The ScriptX core class library has many similarities to other object oriented frameworks in that it provides many basic services common to all applications built on them. All frameworks provide classes for creating windows, handling keyboard and mouse events, reading and writing files, etc. Where ScriptX is unique is in its focus. The ScriptX core classes are oriented towards interactive, media rich applications. For example, clocks and timing are fundamental in the ScriptX class library; most other frameworks have no concept of timing built in. ScriptX also tightly integrates media data (bitmaps, video, audio) with the class library, and hides the details of storing, retrieving, and presenting media to the user. Q: What are the major sections of the core class library? Clocks, players, and animation. Time is a fundamental element of the core classes. Starting with basic clocks, subclasses extend the capabilities for animation, video, and audio playback. Clocks can be tied to underlying hardware clocks or slaved in a hierarchical fashion to other clock objects. Varying the rate of a master clock, all sub-clocks will stay synchronized to the master clock, permitting the programmer to precisely control time in a title. Clock hierarchies also free the programmer from dealing with differences in performance between slower and faster CPUs. Player classes build upon clocks. These classes allow you to create and play sequences of actions that take place over time. These sequences can be used to create animations as well as control other presentation elements such as video or sound. A special type of clock object, the action list player, can be used to play actions in sequence at specified times. Various action objects are added to an action list specifying the time at which the action is to occur. Action objects are used to move graphic elements on the screen, execute ScriptX code, or modify the action list. Other player classes provide simple ways to play digital video, audio, and MIDI. As with all players, the clocks underlying these players can be sped up, slowed down, or run backwards. Q: Can video be synchronized with other events? A: Yes, internal video players are based on ScriptX clocks and can be slaved together to provide synchronization with animations and other events. For example, buttons can appear in a window at precise times based on video playback.

tyingq 8y ago

Syslog seems to be on the proposal list as well.

derefr 8y ago

I wonder whether namespacing time would also result in those namespaces being able to have separate "clocks" (time backends? time schedulers?) that progress at different rates, or for different reasons. Being able to put a process into a time namespace with a deterministic "clock" would obviate a large benefit of http://www.zerovm.org/ http://www.zerovm.org/. Also, having "clock slew" be a matter of perspective—with processes that can handle leap seconds seeing them happen instantaneously; and processes that can't handle leap-seconds, seeing slewed time—would be nice. Then you could have different system facilities that care about monotonic time, vs. synced to calendar time, vs. one second per second time, all having that kind of time available to them as "the time", rather than through different APIs.

kbenson 8y ago
> Also, having "clock slew" be a matter of perspective—with processes that can handle leap seconds seeing them happen instantaneously; and processes that can't handle leap-seconds, seeing slewed time—would be nice. I imagine there might be some really interesting (for meanings of interesting that include shoot me now) and hard to track down bugs as you deal with inconsistent clocks not just across systems within a network, but processes within a single system.
- chatmasta 8y ago
  IIRC Docker for Mac had a bug like this for a long time where the clocks of containers would become wildly out of date.
  - TheDong 8y ago
    More accurately, the clocks of the linux virtual machine running docker containers would differ from the OSX clock. Those aren't really containers skewing from other processes on the same system as the parent describes, but of clocks skewing on two different systems (which is a totally normal thing we deal with regularly).
- derefr 8y ago
  > I imagine there might be some really interesting (for meanings of interesting that include shoot me now) and hard to track down bugs as you deal with inconsistent clocks not just across systems within a network, but processes within a single system. I feel like the "safe assumption" that the other end of a given IPC channel (or even inter-thread communication channel) is on the same machine, is responsible for the vast majority of failures we see in e.g. Jepsen testing of databases. After all, in sufficiently-large computers (i.e. HPC clusters that pretend to be one "computer"), you've got NUMA zones that are light-microseconds away from one another, where even threads of the same process can literally end up needing vector clocks to linearize events between themselves. It probably wouldn't be too bad a thing if things like the Linux base-system used only internal IPC mechanisms that exposed this unreliability (like e.g. Erlang does with "unreliable async message passing" as its IPC primitive), forcing each component to deal with the fact that its peers may or may not be netsplit away from it. Even if that scenario will only come up if you're writing code to get your GPS position from a Dyson sphere of 10-mile-deep Matryoska brains.

timeattack 8y ago

You can't change time in container, but it's possible to change timezone files. With generating fake timezones it is possible to change time in container.

cyphar 8y ago
This doesn't change what gettimeofday(2) gives you (and actually you can't even use ptrace easily to fake the time of day because gettimeofday(2) isn't a real syscall -- it's actually implemented as a read from the vDSO page the kernel maps into every process).

haberman 8y ago

> The current set of namespaces in the kernel are: mount, pid, uts, ipc, net, user, and cgroup. [...] [Time is] not namespaced. [...] The kernel keyring is another item not namespaced. I've always argued that "everything is a file" is an exaggeration. These moments make the extent of that exaggeration clear. If everything truly was a file, the only thing you would need to namespace is the filesystem. But in reality there are a lot of other kernel objects that are not files at all.

tyingq 8y ago
I agree it's unfortunate, but it doesn't really seem in conflict with "everything is a file". Making it a file is separate from making it sensible/usable for containers. Like the /proc filesystem. They are "files", but don't many don't work as expected without something like lxcfs. Like /proc/uptime, for example.
madhadron 8y ago
Everything is a file hasn't been true for Unix since almost the beginning. It's kind of like the Unix philosophy of small, independent tools...except for the database where you store all the important data.
zapita 8y ago
You are 100% correct. “Everything is a file” was more of an early design insight, which was gradually abandoned as new features were added. There is a movement of “Unix purists” who lament this deviation from founding principles, and advocate for a return to them. The most notable example is Plan 9. In Plan 9, everything actually is a file. And exactly as you said, all resources are namespaced via the filesystem. It’s quite elegant and practical. Sadly Plan 9 has remained a fringe OS, and although it influenced mainstream operating system design in many ways (including the concept of /proc), I wish that influence had been stronger.
- temac 8y ago
  hm is NT a purer Unix than Unix then? After all, it has all its object in a filesystem-like tree...

vxNsr 8y ago

meta: This is from 2017, Super interesting though, the keyring thing especially seems to have broader implications...

theamk 8y ago

I personally miss core pattern namespacing. I would love to give some of my containers a custom coredump handler, but this is impossible. And in general, a sysctls settings namespace would be really useful. Sure, sometimes it makes no sense to namespace a setting, but net.ipv4.tcp_congestion_control for example? I'd love to be able to change it without modifying the code.

Sharlin 8y ago

I’m not sure that people who think ”containers are just like VMs” should have any business working with containers.