12 ms·

Improving Docker with Unikernels: Introducing HyperKit, VPNKit and DataKit

292 points by samber 10y ago

kordless 10y ago
Seems like only a year ago Docker changed how it used Virtualbox to boot VMs using machine (and caused me endless amounts of suffering trying to figure out how to fix it). Now it would seem they are getting rid of Virtualbox entirely with their own VM...which needs contributions.
- avita1 10y ago
  My understanding is not great, but it looks like they're bypassing a host VM entirely and using a shim (HyperKit) to talk to whatever virtualization the underlying OS supplies.
- danieldk 10y ago
  getting rid of Virtualbox entirely with their own VM... Not really. This builds heavily upon Hypervisor.framework, which is included in new OS X versions. Hypervisor.framework is pretty cool, because it allows for building VM applications that are sandboxed and don't require kernel modules. As a result, Hypervisor.framework applications can even be distributed via the App Store (see e.g. Veertu).
  - lobster_johnson 10y ago
    I wonder if Vagrant intends to provide a Hypervisor.framework-based Linux backend. That would be cool. Edit: Experimental plugin: https://github.com/oldpatricka/vagrant-xhyve https://github.com/oldpatricka/vagrant-xhyve.
  - fpoling 10y ago
    The drawback is that now VM manager is at mercy of what Hypervisor.framework provides. For example, Veertu does not support redirection of USB devices into VM because the framework does not support it. That makes it rather useless for my usage and I stick with VirtualBox for now. Of cause, this is not an issue for Docker as the framework covers all its needs.
- rhinoceraptor 10y ago
  I'm running the new beta native Docker for Mac, it is really nice. You install from the DMG, enter your password, and you're done. No more running 'eval $(docker-machine env dev)', no more having to delete the VM and rebuild it, etc.
  - knz 10y ago
    Have your encountered any major bugs with Docker for OSX? I've been running docker-machine without any major issues for months but have noticed in the last few weeks that I need to reboot the VM lately due to weird caching issues.
    - toomuchtodo 10y ago
      I have constant connection problems between the Docker CLI and the Docker VM running Docker for OSX :/
      - knz 10y ago
        Thanks for the feedback.
      - justincormack 10y ago
        In recent updates? We have changed how this works in the last few releases, we hope it will be more stable.
        zwischenzug 10y ago
        I had this also; will re-try a more recent version.
        avsm 10y ago
        Please do let us know if it persists, to beta-feedback@docker.com or on the forums.
    - rhinoceraptor 10y ago
- amackera 10y ago
  Docker is still pretty bleeding edge, and was never designed to work with Mac OS X or Windows from the get-go. The fact that the proposed experience for OS X and Windows is so good is a testament to how hard the community and the Docker team have been working! It's natural that there would be some growing pains.
  - jcoffland 10y ago
    So you're saying it's really bad and that's why it's so good.
andrew_wc_brown 10y ago
I guess I just want to know the take away. eg. Will consume less memory on mac.
- amirmc 10y ago
  If you're asking about Docker for Mac/Windows, an earlier blog post has a few details. https://blog.docker.com/2016/03/docker-for-mac-windows-beta/ https://blog.docker.com/2016/03/docker-for-mac-windows-beta/ In essence, it's a much nicer experience using the app and since it's improving all the time (still in beta, fwiw).
pjmlp 10y ago
With lots of OCaml love it seems, from a quick glance through the source repositories.
- danieldk 10y ago
  And Mirage. This release is really awesome: - It puts Mirage in use in real production systems. - It strengthens the ecosystem around Hypervisor.framework. After xhyve and Veertu, this is really nice. So much for the meme that newer OS X versions only integrate iOS features.
  - floatboth 10y ago
    They only took, like, the network stack from Mirage. Their virtualization library is xhyve… well, turned into a library.
    - avsm 10y ago
      Open sourcing our libraries is obviously just the start. We're really keen to hear about any other uses that people might have and move HyperKit in the direction of easy integration with higher-level frameworks. For instance, direct ports of unikernel projects to run against the HyperKit kexec APIs would be really fun. Any takers for MirageOS or HalVM? :-) We've started keeping a list of "help wanted" issues for anyone interested in getting started with hacking in this area: https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22 https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3A...
      - bboreham 10y ago
        What was the motivation to create a new repo with a different name rather than maintaining a fork of xhyve?
        avsm 10y ago
        We're taking it in a fairly different direction from upstream xhyve and bhyve, with the integration with other kits like VPNKit, DataKit (and soon, FSKit). We wanted to be able to diverge the build system in particular, and this makes it difficult to maintain a direct fork. However, we're committed to upstreaming patches to their relevant projects where relevant, and so the HyperKit Git repository is as close as we can make it. If it diverges a lot in the future with file renames, we'll have to figure out the Git-fu to make cherry-picks easier...
deleted 10y ago
[deleted]
tachion 10y ago
I wonder if we'll see a move towards getting Docker working on FreeBSD using either Jails or bhyve finally, since it talks about using bhyve hypervisor... That would be really great.
- floatboth 10y ago
  There is a port of docker that works with jails+ZFS (and the linux.ko syscall translator to run linux images) https://wiki.freebsd.org/Docker https://wiki.freebsd.org/Docker There's also jetpack for the app container spec (rkt) https://github.com/3ofcoins/jetpack https://github.com/3ofcoins/jetpack I'm also experimenting with building a simpler system.
  - avsm 10y ago
    It should be possible to build a simpler port for FreeBSD with the recent Docker 1.11 release. It moved the container execution to containerd (https://github.com/docker/containerd https://github.com/docker/containerd), so that's where a bunch of the Jails logic would go. I'm quite keen to see Docker running on FreeBSD so I can use it for my CI pipelines. I'm less interested in Linux emulation to run Linux images -- I'd like Docker support on FreeBSD to run FreeBSD images first!
    - voltagex_ 10y ago
      What advantages would Docker give you over jails themselves?
      - evgen 10y ago
        Can't speak for anyone else, but while I think jails are "better" for a variety of reasons it seems clear there is far more market momentum behind Docker. More particularly, its toolchain. Given the choice between building myself (or relying upon a small, but talented, pool of devs) and drafting off tens of millions in VC cash and a much larger community I think some would prefer the ability to have a *BSD CI chain that is just a config setting in a Docker setup.
kevinmgranger 10y ago
docker's go-9p now makes for the 3rd implementation of 9p in go: docker/go-9p https://github.com/docker/go-p9p https://github.com/docker/go-p9p rminnich/ninep: https://github.com/rminnich/ninep https://github.com/rminnich/ninep rminnich/go9p: https://github.com/rminnich/go9p https://github.com/rminnich/go9p There's also the Andrey Mirtchovski and Latchesar Ionkov implementation of go9p, but all I can find is a dead Google Code link from here: http://9p.cat-v.org/implementations http://9p.cat-v.org/implementations
- justincormack 10y ago
  We did try the other implementations first, but had a lot of issues doing what we wanted with them.
chuhnk 10y ago
Very interesting work. I find go-9p quite fascinating and think it could really have broader applications. Docker if you see this, I actually think you're on to something for microservice development thats native to the docker world. I've been trying to come up with ways of replicating the unix philosophy around programs that do one thing well and the use of pipes but was always limited in my thinking in terms of http, json, etc, etc. My advice, as a guy who's currently building something in the microservice space, explore this further. Spend some time building fit for purpose apps with this and see where it goes.

292 points by samber 10y ago

Seems like only a year ago Docker changed how it used Virtualbox to boot VMs using machine (and caused me endless amounts of suffering trying to figure out how to fix it). Now it would seem they are getting rid of Virtualbox entirely with their own VM...which needs contributions.

avita1 10y ago
My understanding is not great, but it looks like they're bypassing a host VM entirely and using a shim (HyperKit) to talk to whatever virtualization the underlying OS supplies.
danieldk 10y ago
getting rid of Virtualbox entirely with their own VM... Not really. This builds heavily upon Hypervisor.framework, which is included in new OS X versions. Hypervisor.framework is pretty cool, because it allows for building VM applications that are sandboxed and don't require kernel modules. As a result, Hypervisor.framework applications can even be distributed via the App Store (see e.g. Veertu).
- lobster_johnson 10y ago
  I wonder if Vagrant intends to provide a Hypervisor.framework-based Linux backend. That would be cool. Edit: Experimental plugin: https://github.com/oldpatricka/vagrant-xhyve https://github.com/oldpatricka/vagrant-xhyve.
- fpoling 10y ago
  The drawback is that now VM manager is at mercy of what Hypervisor.framework provides. For example, Veertu does not support redirection of USB devices into VM because the framework does not support it. That makes it rather useless for my usage and I stick with VirtualBox for now. Of cause, this is not an issue for Docker as the framework covers all its needs.
rhinoceraptor 10y ago
I'm running the new beta native Docker for Mac, it is really nice. You install from the DMG, enter your password, and you're done. No more running 'eval $(docker-machine env dev)', no more having to delete the VM and rebuild it, etc.
- knz 10y ago
  Have your encountered any major bugs with Docker for OSX? I've been running docker-machine without any major issues for months but have noticed in the last few weeks that I need to reboot the VM lately due to weird caching issues.
  - toomuchtodo 10y ago
    I have constant connection problems between the Docker CLI and the Docker VM running Docker for OSX :/
    - knz 10y ago
      Thanks for the feedback.
    - justincormack 10y ago
      In recent updates? We have changed how this works in the last few releases, we hope it will be more stable.
      - zwischenzug 10y ago
        I had this also; will re-try a more recent version.
        avsm 10y ago
        Please do let us know if it persists, to beta-feedback@docker.com or on the forums.
  - rhinoceraptor 10y ago
amackera 10y ago
Docker is still pretty bleeding edge, and was never designed to work with Mac OS X or Windows from the get-go. The fact that the proposed experience for OS X and Windows is so good is a testament to how hard the community and the Docker team have been working! It's natural that there would be some growing pains.
- jcoffland 10y ago
  So you're saying it's really bad and that's why it's so good.

andrew_wc_brown 10y ago

I guess I just want to know the take away. eg. Will consume less memory on mac.

amirmc 10y ago
If you're asking about Docker for Mac/Windows, an earlier blog post has a few details. https://blog.docker.com/2016/03/docker-for-mac-windows-beta/ https://blog.docker.com/2016/03/docker-for-mac-windows-beta/ In essence, it's a much nicer experience using the app and since it's improving all the time (still in beta, fwiw).

pjmlp 10y ago

With lots of OCaml love it seems, from a quick glance through the source repositories.

danieldk 10y ago
And Mirage. This release is really awesome: - It puts Mirage in use in real production systems. - It strengthens the ecosystem around Hypervisor.framework. After xhyve and Veertu, this is really nice. So much for the meme that newer OS X versions only integrate iOS features.
- floatboth 10y ago
  They only took, like, the network stack from Mirage. Their virtualization library is xhyve… well, turned into a library.
  - avsm 10y ago
    Open sourcing our libraries is obviously just the start. We're really keen to hear about any other uses that people might have and move HyperKit in the direction of easy integration with higher-level frameworks. For instance, direct ports of unikernel projects to run against the HyperKit kexec APIs would be really fun. Any takers for MirageOS or HalVM? :-) We've started keeping a list of "help wanted" issues for anyone interested in getting started with hacking in this area: https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22 https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3A...
    - bboreham 10y ago
      What was the motivation to create a new repo with a different name rather than maintaining a fork of xhyve?
      - avsm 10y ago
        We're taking it in a fairly different direction from upstream xhyve and bhyve, with the integration with other kits like VPNKit, DataKit (and soon, FSKit). We wanted to be able to diverge the build system in particular, and this makes it difficult to maintain a direct fork. However, we're committed to upstreaming patches to their relevant projects where relevant, and so the HyperKit Git repository is as close as we can make it. If it diverges a lot in the future with file renames, we'll have to figure out the Git-fu to make cherry-picks easier...

deleted 10y ago

[deleted]

tachion 10y ago

I wonder if we'll see a move towards getting Docker working on FreeBSD using either Jails or bhyve finally, since it talks about using bhyve hypervisor... That would be really great.

floatboth 10y ago
There is a port of docker that works with jails+ZFS (and the linux.ko syscall translator to run linux images) https://wiki.freebsd.org/Docker https://wiki.freebsd.org/Docker There's also jetpack for the app container spec (rkt) https://github.com/3ofcoins/jetpack https://github.com/3ofcoins/jetpack I'm also experimenting with building a simpler system.
- avsm 10y ago
  It should be possible to build a simpler port for FreeBSD with the recent Docker 1.11 release. It moved the container execution to containerd (https://github.com/docker/containerd https://github.com/docker/containerd), so that's where a bunch of the Jails logic would go. I'm quite keen to see Docker running on FreeBSD so I can use it for my CI pipelines. I'm less interested in Linux emulation to run Linux images -- I'd like Docker support on FreeBSD to run FreeBSD images first!
  - voltagex_ 10y ago
    What advantages would Docker give you over jails themselves?
    - evgen 10y ago
      Can't speak for anyone else, but while I think jails are "better" for a variety of reasons it seems clear there is far more market momentum behind Docker. More particularly, its toolchain. Given the choice between building myself (or relying upon a small, but talented, pool of devs) and drafting off tens of millions in VC cash and a much larger community I think some would prefer the ability to have a *BSD CI chain that is just a config setting in a Docker setup.

kevinmgranger 10y ago

docker's go-9p now makes for the 3rd implementation of 9p in go: docker/go-9p https://github.com/docker/go-p9p https://github.com/docker/go-p9p rminnich/ninep: https://github.com/rminnich/ninep https://github.com/rminnich/ninep rminnich/go9p: https://github.com/rminnich/go9p https://github.com/rminnich/go9p There's also the Andrey Mirtchovski and Latchesar Ionkov implementation of go9p, but all I can find is a dead Google Code link from here: http://9p.cat-v.org/implementations http://9p.cat-v.org/implementations

justincormack 10y ago
We did try the other implementations first, but had a lot of issues doing what we wanted with them.

chuhnk 10y ago

Very interesting work. I find go-9p quite fascinating and think it could really have broader applications. Docker if you see this, I actually think you're on to something for microservice development thats native to the docker world. I've been trying to come up with ways of replicating the unix philosophy around programs that do one thing well and the use of pipes but was always limited in my thinking in terms of http, json, etc, etc. My advice, as a guy who's currently building something in the microservice space, explore this further. Spend some time building fit for purpose apps with this and see where it goes.