9 ms·

How to Use User Mode Linux

269 points by xena 7y ago

lordleft 7y ago
I had no idea this was possible. Excellent article - thank you for opening my eyes to yet another insanely cool aspect of Linux.
EvanAnderson 7y ago
The next logical thing to do for more fun is to run User Mode Linux under the Windows Subsystem for Linux. After that, it would make sense to try WINE.
- userbinator 7y ago
  A Windows app running on WINE inside UML inside WSL of a Win10 which is itself a VM running on e.g. macOS would be funny to see.
  - sigjuice 7y ago
    Does WSL support ptrace now? UML can't run without it.
  - hdfbdtbcdg 7y ago
    Especially if the app is Electron based.
- kybernetikos 7y ago
  There used to be a version of uml that ran on windows. It was very like the modern windows subsystem for Linux. http://colinux.org/ http://colinux.org/
- ddalex 7y ago
  I remember that at some point there was wine port to Windows, so you could run windows while running windows...
  - 20after4 7y ago
    yo dawg.
  - lmm 7y ago
    I remember that being actually useful for me! XP 64-bit couldn't run 16-bit windows executables, but you could build wine against SFU/SUA and then use that to run them.
badrabbit 7y ago
Very nice. Now I'm tempted to try and run X11 and firefox in UML somehow.
vmlinuz 7y ago
Many many years ago, when I was working at a large 3-letter-starting-with-S Unix company (no, not the graphics guys, nor the suing-Linux-for-Microsoft guys), we were planning a Linux compatibility layer... At the time, there were a couple of options out there - lxrun and uml. Anyone remember lxrun? I tested them both and did a little write-up, basically saying that neither of them ran reliably or fast enough to be worth adopting What we decided was to go with option 3 - we implemented a Linux system call interface within our own kernel to give us 'native' support. This, of course, is exactly what the first version of WSL did, many years later. Plus ça change...
- meruru 7y ago
  >at a large 3-letter-starting-with-S Unix company (no, not the graphics guys, nor the suing-Linux-for-Microsoft guys) Is it Sun? Why use more than 20 words to (probably) mean "for Sun"?
  - ghostpepper 7y ago
    Presumably they think this is a defense that will hold up in court as plausible deniability if they are ever sued for defamation relating to this comment.
    - meruru 7y ago
      Sun doesn't exist anymore though. Is getting sued even a possibility?
      - bencollier49 7y ago
        In the hypothetical scenario that this commenter is implying that Solaris contains (or contained) a load of IP which is non-clean-roomed Linux copyright (is that the implication??), then the current owners of Solaris would be the ones getting in trouble with - the FSF? That would be Oracle. https://www.oracle.com/solaris/solaris11/ https://www.oracle.com/solaris/solaris11/ And it would be Oracle chasing the commenter for an NDA breach, I suppose.
nneonneo 7y ago
UML has been around for a very, very long time. I remember using it back in undergrad for one of my courses, but now I forget which one. It’s got great documentation. See the kernel.org documentation here: https://www.kernel.org/doc/Documentation/virtual/uml/UserModeLinux-HOWTO.txt https://www.kernel.org/doc/Documentation/virtual/uml/UserMod... and a write up from the creator Jeff Dike (which includes a lot of technical detail on the inner workings) here: https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf (from 2001!) slirp is the only _purely usermode_ way to do networking, but UML also supports TUN/TAP for something a little more sane. UML does a bunch of pretty crazy things in order to work, including a dedicated thread whose job it is to ptrace every other process for syscall and signal emulation (and, as a bonus, it even ptraces external debuggers to make them think they’ve successfully ptraced a process that’s already being traced!), and a full emulation of physical/virtual memory translation with paging (done by giving every virtual process under UML its own virtual address space). All in all, a super impressive project, and definitely one worth trying out.
- svd4anything 7y ago
  I’ve recently been using proot [1] as a way to have a fully controlled rootfs for jobs on a HPC cluster/grid service where I am a very regular unprivileged user. Is it theoretically possible I could run an entire linux kernel instead with this? Is there any setup on the host system that is required which needs administrative privileges? [1] https://proot-me.github.io/ https://proot-me.github.io/
  - CameronNemo 7y ago
    I do not have much info on UML, but you may be interested in https://rootlesscontaine.rs/ https://rootlesscontaine.rs/
    - gnufx 7y ago
      Unless you're interested in OCI (?) containers, you're probably better off with straight proot, or charlecloud if you're allowed user namespaces. https://github.com/hpc/charliecloud https://github.com/hpc/charliecloud
wmeddie 7y ago
Brings back memories. My first VPS back in 2003 used UML and it worked great. You could also run it inside Windows as well. Could even get GUI apps running by running Xming on your Windows desktop using CoLinux (http://www.colinux.org/?section=screenshots http://www.colinux.org/?section=screenshots).
- zoobab 7y ago
  Colinux was freaking great, I could run an OpenSSH server on port 22 and sometimes ending up on a Linux box, sometimes on a Windows box. Weird.
meruru 7y ago
How does this compare to other methods of isolation / running untrusted code? What are the pros and cons?
- nisa 7y ago
  I've used for developing on a community mesh network - you can boot OpenWrt with a fake wifi driver module and you can plug in in networking as you will and run multiple processes. It's far simpler to configure and does not need compatibily fixes for running in containers.
- stevekemp 7y ago
  I worked for a hosting company for 10ish years, and they started out their hosting by providing people with UML-based virtual machines for £10/month. In terms of isolation it is "pretty good". The guest has their own kernel, and you can run it as an ordinary user. Add a chroot jail and you can have 50-150 different UML instances running on a single host, each under their own UID. They can't talk to each other, and if they do manage to exploit the kernel they're only gonna get the access to the host which you restrict with chroot, and standard things. It's less of a risk to run isolated things than this than have lots of containers share the same kernel. An escape there means container-A can view/poke container-B AND the host. Of course there is overhead, and most people moved on from UML to Xen/KVM when they became available - both have better performance thanks to virtio, and both have more flexibility for networking, block devices & etc.
- dspillett 7y ago
  https://www.linode.com/ https://www.linode.com/ used to run off UML, though they switched quite a number of years ago. It might be worth hunting out their documentation and/or forum archives from around the time of the transition. I recall a lot of detailed discussion (useful discussion too: not just people bashing the technology they don't prefer). I used to run it myself, way back when. For isolation, it is going to be better than a container or jail, but not as good as a full VM, though I was using it for testing personal stuff so wasn't really concerned with isolation from a security point of view. One key con is that it can perform pretty badly in some respects, particularly for loads that ned to talk to the kernel much at all, including processes that perform much IO. It "felt" less cumbersome than full VMs (IIRC I was playing with a mix of VirtualBox and VMWare' Player & Server for that sort of thing at the time), in terms of my automation and to some extent in terms of host resource use (though I didn't test that with any scientific rigour, and if I did any good tests back then the results would not be relevant today anyway as they were pre VT-x and other CPU support which UML can't use but a modern hypervisor can and will).
svd4anything 7y ago
https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf In 2001 this mentions SMP support is possible. First thing tomorrow I need to check if it was ever implemented. Perhaps it’s late and I’m tired but somehow this seems to be much better than containers for a number of problems, or maybe the performance hit is too much.
michaelgrafl 7y ago
Off topic, but here it goes: I really like the styling of this website. I shouldn't like it, because I don't like non-proportional fonts being used for prose, but it works great in this case. Also loads super fast, which is an exception these days.
- opan 7y ago
  I like it as well. It looks like one of those generated static sites from hugo or haunt. I've been seeing it more often lately and I think it's a good trend.
  - xena 7y ago
    My site isn't actually a static site. The code is in https://github.com/Xe/site https://github.com/Xe/site
etaioinshrdlu 7y ago
When I tried User Mode Linux I found that there was very little in the way of ready-made, up to date tooling. Very little "just worked". I think there's huge potential there, however, as an architectural building block, just like containers are.
- xena 7y ago
  It's the kind of stuff that drives you insane a bit. You have to hack together things from software last released in 2006. You have to dig through kernel source code to really understand what is going on in the user mode Linux drivers.
bencollier49 7y ago
The article states that processes in User Mode Linux are completely isolated from the host, and cannot be seen. Could someone clarify this for me - there must be some way of pulling data from UML back into the host system?
- nneonneo 7y ago
  Actually, very much on the contrary, the processes running inside UML are completely visible as separate threads within the UML process, and you can go poking their `proc` files at-will. Also, UML uses a read/write disk image as the rootfs, so any writes to the filesystem will be persisted in the disk image.
  - bencollier49 7y ago
    I thought as much! Thanks.
  - svd4anything 7y ago
    > processes running inside UML are completely visible as separate threads within the UML process Does this mean the UML system can utilize many cores well and SMP was eventually implied?
Tepix 7y ago
It would be nice to combine UML with an encrypted root filesystem so a nosy admin on a VPS will have a harder time seeing what you are doing on your VPS. Still I wonder what the performance will be like. Is the entire UML process running single threaded on a single CPU core?
- xena 7y ago
  Each process in the guest has its own host thread, but yeah the way I have it set up there's only one logical "CPU core" being exposed to the guest.
twayback 7y ago
Great work! I am trying to run this on the latest Ubuntu but getting the below error: root@kit:~/uml/furry-happiness# docker run --rm -it xena/docker-uml [WARN tini (7)] Tini is not running as PID 1 and isn't registered as a child subreaper. Zombie processes will not be re-parented to Tini, so zombie reaping won't work. To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1. Core dump limits : soft - NONE hard - NONE Checking that ptrace can change system call numbers...ptrace: Operation not permitted check_ptrace : expected SIGSTOP, got status = 9
- xena 7y ago
  What host OS do you run? I got this working on both Ubuntu and Alpine.
  - twayback 7y ago
    I am running this on ubuntu 18.04 :(
zoobab 7y ago
weberlars/diuid https://github.com/weber-software/diuid https://github.com/weber-software/diuid runs a UML in a docker container, which then runs a Docker daemon of a different version inside. Clever.
linsomniac 7y ago
I have often thought back to how great UML was. I had filed it under "distant history" when I thought about it last week... Back in 2003 or 2004, I threw together a "virtual hosting" platform using UML. It had a web form accepting payment, and a few minutes later you'd have a fully running virtual machine. I put this together in a couple weeks leading up to PyCon, where we launched it. But we didn't really have the business savvy to take it anywhere, we never had more than a few hundred VMs. We eventually moved to other technologies: VMWare, Xen, Proxmox (KVM). But UML was my favorite. (Xen was my least favorite :-)

269 points by xena 7y ago

I had no idea this was possible. Excellent article - thank you for opening my eyes to yet another insanely cool aspect of Linux.

EvanAnderson 7y ago

The next logical thing to do for more fun is to run User Mode Linux under the Windows Subsystem for Linux. After that, it would make sense to try WINE.

userbinator 7y ago
A Windows app running on WINE inside UML inside WSL of a Win10 which is itself a VM running on e.g. macOS would be funny to see.
- sigjuice 7y ago
  Does WSL support ptrace now? UML can't run without it.
- hdfbdtbcdg 7y ago
  Especially if the app is Electron based.
kybernetikos 7y ago
There used to be a version of uml that ran on windows. It was very like the modern windows subsystem for Linux. http://colinux.org/ http://colinux.org/
ddalex 7y ago
I remember that at some point there was wine port to Windows, so you could run windows while running windows...
- 20after4 7y ago
  yo dawg.
- lmm 7y ago
  I remember that being actually useful for me! XP 64-bit couldn't run 16-bit windows executables, but you could build wine against SFU/SUA and then use that to run them.

badrabbit 7y ago

Very nice. Now I'm tempted to try and run X11 and firefox in UML somehow.

vmlinuz 7y ago

Many many years ago, when I was working at a large 3-letter-starting-with-S Unix company (no, not the graphics guys, nor the suing-Linux-for-Microsoft guys), we were planning a Linux compatibility layer... At the time, there were a couple of options out there - lxrun and uml. Anyone remember lxrun? I tested them both and did a little write-up, basically saying that neither of them ran reliably or fast enough to be worth adopting What we decided was to go with option 3 - we implemented a Linux system call interface within our own kernel to give us 'native' support. This, of course, is exactly what the first version of WSL did, many years later. Plus ça change...

meruru 7y ago
>at a large 3-letter-starting-with-S Unix company (no, not the graphics guys, nor the suing-Linux-for-Microsoft guys) Is it Sun? Why use more than 20 words to (probably) mean "for Sun"?
- ghostpepper 7y ago
  Presumably they think this is a defense that will hold up in court as plausible deniability if they are ever sued for defamation relating to this comment.
  - meruru 7y ago
    Sun doesn't exist anymore though. Is getting sued even a possibility?
    - bencollier49 7y ago
      In the hypothetical scenario that this commenter is implying that Solaris contains (or contained) a load of IP which is non-clean-roomed Linux copyright (is that the implication??), then the current owners of Solaris would be the ones getting in trouble with - the FSF? That would be Oracle. https://www.oracle.com/solaris/solaris11/ https://www.oracle.com/solaris/solaris11/ And it would be Oracle chasing the commenter for an NDA breach, I suppose.

nneonneo 7y ago

UML has been around for a very, very long time. I remember using it back in undergrad for one of my courses, but now I forget which one. It’s got great documentation. See the kernel.org documentation here: https://www.kernel.org/doc/Documentation/virtual/uml/UserModeLinux-HOWTO.txt https://www.kernel.org/doc/Documentation/virtual/uml/UserMod... and a write up from the creator Jeff Dike (which includes a lot of technical detail on the inner workings) here: https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf (from 2001!) slirp is the only _purely usermode_ way to do networking, but UML also supports TUN/TAP for something a little more sane. UML does a bunch of pretty crazy things in order to work, including a dedicated thread whose job it is to ptrace every other process for syscall and signal emulation (and, as a bonus, it even ptraces external debuggers to make them think they’ve successfully ptraced a process that’s already being traced!), and a full emulation of physical/virtual memory translation with paging (done by giving every virtual process under UML its own virtual address space). All in all, a super impressive project, and definitely one worth trying out.

svd4anything 7y ago
I’ve recently been using proot [1] as a way to have a fully controlled rootfs for jobs on a HPC cluster/grid service where I am a very regular unprivileged user. Is it theoretically possible I could run an entire linux kernel instead with this? Is there any setup on the host system that is required which needs administrative privileges? [1] https://proot-me.github.io/ https://proot-me.github.io/
- CameronNemo 7y ago
  I do not have much info on UML, but you may be interested in https://rootlesscontaine.rs/ https://rootlesscontaine.rs/
  - gnufx 7y ago
    Unless you're interested in OCI (?) containers, you're probably better off with straight proot, or charlecloud if you're allowed user namespaces. https://github.com/hpc/charliecloud https://github.com/hpc/charliecloud

wmeddie 7y ago

Brings back memories. My first VPS back in 2003 used UML and it worked great. You could also run it inside Windows as well. Could even get GUI apps running by running Xming on your Windows desktop using CoLinux (http://www.colinux.org/?section=screenshots http://www.colinux.org/?section=screenshots).

zoobab 7y ago
Colinux was freaking great, I could run an OpenSSH server on port 22 and sometimes ending up on a Linux box, sometimes on a Windows box. Weird.

meruru 7y ago

How does this compare to other methods of isolation / running untrusted code? What are the pros and cons?

nisa 7y ago
I've used for developing on a community mesh network - you can boot OpenWrt with a fake wifi driver module and you can plug in in networking as you will and run multiple processes. It's far simpler to configure and does not need compatibily fixes for running in containers.
stevekemp 7y ago
I worked for a hosting company for 10ish years, and they started out their hosting by providing people with UML-based virtual machines for £10/month. In terms of isolation it is "pretty good". The guest has their own kernel, and you can run it as an ordinary user. Add a chroot jail and you can have 50-150 different UML instances running on a single host, each under their own UID. They can't talk to each other, and if they do manage to exploit the kernel they're only gonna get the access to the host which you restrict with chroot, and standard things. It's less of a risk to run isolated things than this than have lots of containers share the same kernel. An escape there means container-A can view/poke container-B AND the host. Of course there is overhead, and most people moved on from UML to Xen/KVM when they became available - both have better performance thanks to virtio, and both have more flexibility for networking, block devices & etc.
dspillett 7y ago
https://www.linode.com/ https://www.linode.com/ used to run off UML, though they switched quite a number of years ago. It might be worth hunting out their documentation and/or forum archives from around the time of the transition. I recall a lot of detailed discussion (useful discussion too: not just people bashing the technology they don't prefer). I used to run it myself, way back when. For isolation, it is going to be better than a container or jail, but not as good as a full VM, though I was using it for testing personal stuff so wasn't really concerned with isolation from a security point of view. One key con is that it can perform pretty badly in some respects, particularly for loads that ned to talk to the kernel much at all, including processes that perform much IO. It "felt" less cumbersome than full VMs (IIRC I was playing with a mix of VirtualBox and VMWare' Player & Server for that sort of thing at the time), in terms of my automation and to some extent in terms of host resource use (though I didn't test that with any scientific rigour, and if I did any good tests back then the results would not be relevant today anyway as they were pre VT-x and other CPU support which UML can't use but a modern hypervisor can and will).

svd4anything 7y ago

https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf In 2001 this mentions SMP support is possible. First thing tomorrow I need to check if it was ever implemented. Perhaps it’s late and I’m tired but somehow this seems to be much better than containers for a number of problems, or maybe the performance hit is too much.

michaelgrafl 7y ago

Off topic, but here it goes: I really like the styling of this website. I shouldn't like it, because I don't like non-proportional fonts being used for prose, but it works great in this case. Also loads super fast, which is an exception these days.

opan 7y ago
I like it as well. It looks like one of those generated static sites from hugo or haunt. I've been seeing it more often lately and I think it's a good trend.
- xena 7y ago
  My site isn't actually a static site. The code is in https://github.com/Xe/site https://github.com/Xe/site

etaioinshrdlu 7y ago

When I tried User Mode Linux I found that there was very little in the way of ready-made, up to date tooling. Very little "just worked". I think there's huge potential there, however, as an architectural building block, just like containers are.

xena 7y ago
It's the kind of stuff that drives you insane a bit. You have to hack together things from software last released in 2006. You have to dig through kernel source code to really understand what is going on in the user mode Linux drivers.

bencollier49 7y ago

The article states that processes in User Mode Linux are completely isolated from the host, and cannot be seen. Could someone clarify this for me - there must be some way of pulling data from UML back into the host system?

nneonneo 7y ago
Actually, very much on the contrary, the processes running inside UML are completely visible as separate threads within the UML process, and you can go poking their `proc` files at-will. Also, UML uses a read/write disk image as the rootfs, so any writes to the filesystem will be persisted in the disk image.
- bencollier49 7y ago
  I thought as much! Thanks.
- svd4anything 7y ago
  > processes running inside UML are completely visible as separate threads within the UML process Does this mean the UML system can utilize many cores well and SMP was eventually implied?

Tepix 7y ago

It would be nice to combine UML with an encrypted root filesystem so a nosy admin on a VPS will have a harder time seeing what you are doing on your VPS. Still I wonder what the performance will be like. Is the entire UML process running single threaded on a single CPU core?

xena 7y ago
Each process in the guest has its own host thread, but yeah the way I have it set up there's only one logical "CPU core" being exposed to the guest.

twayback 7y ago

Great work! I am trying to run this on the latest Ubuntu but getting the below error: root@kit:~/uml/furry-happiness# docker run --rm -it xena/docker-uml [WARN tini (7)] Tini is not running as PID 1 and isn't registered as a child subreaper. Zombie processes will not be re-parented to Tini, so zombie reaping won't work. To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1. Core dump limits : soft - NONE hard - NONE Checking that ptrace can change system call numbers...ptrace: Operation not permitted check_ptrace : expected SIGSTOP, got status = 9

xena 7y ago
What host OS do you run? I got this working on both Ubuntu and Alpine.
- twayback 7y ago
  I am running this on ubuntu 18.04 :(

zoobab 7y ago

weberlars/diuid https://github.com/weber-software/diuid https://github.com/weber-software/diuid runs a UML in a docker container, which then runs a Docker daemon of a different version inside. Clever.

linsomniac 7y ago

I have often thought back to how great UML was. I had filed it under "distant history" when I thought about it last week... Back in 2003 or 2004, I threw together a "virtual hosting" platform using UML. It had a web form accepting payment, and a few minutes later you'd have a fully running virtual machine. I put this together in a couple weeks leading up to PyCon, where we launched it. But we didn't really have the business savvy to take it anywhere, we never had more than a few hundred VMs. We eventually moved to other technologies: VMWare, Xen, Proxmox (KVM). But UML was my favorite. (Xen was my least favorite :-)