5 ms·

I'm quite interested in seeing where slim VM's go. Personally I don't use Kubernetes, it just doesn't fit my client work which is nearly all single-server and i

I'm quite interested in seeing where slim VM's go. Personally I don't use Kubernetes, it just doesn't fit my client work which is nearly all single-server and it makes more sense to just run podman systemd units or docker-compose setups. So from that perspective, when I've peeked at firecracker, kata containers, etc, the "small dev dx" isn't quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn't supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn't quite there yet (but very close). Honestly I don't really care about containers vs VMs except the VM alleges better security which is nice, and I guess I like poking at things but they're were a little too rough for weekend poking. Is anyone doing "small scale" lightweight vm deployments - maybe just in your homelab or toy projects? Have you found the experience better than containers?

xani_ 4y ago
They went to trash because containers are more convenient to use and saving few MBs of disk/memory is not what most users care. The whole idea was pretty much either use custom kernel (which inevitably have way less info on how to debug anything in it), and re-do all of the network and storage plumbing containers already do via the OS they are running one. OR just very slim linux one which at least people know how to use but STILL is more complexity than "just a blob with some namespaces in it" and STILL requires a bunch of config and data juggling between hypervisor and VM just to share some host files to the guest. Either way to get to the level of "just a slim layer of code between hypervisor and your code" you need to do a quite a lot of deep plumbing and when anything goes wrong debugging is harder. All to get some perceived security and no better performance than just... running the binary in a container. It did percolate into "slim containers" idea where the container is just statically compiled binary + few configs and while it does have same problems with debuggability, you can just attach sidecart to it I guess next big hype will be "VM bUt YoU RuN WebAsSeMbLy In CuStOm KeRnEl"
- evol262 4y ago
  Virtualization is not just "perceived" security over containerization. From CPU rings on down, it offers dramatically more isolation for security than containerization does. This isn't about 'what most users care' about either. Most users don't really care about 99% of what container orchestration platforms offer. The providers do absolutely care that malicious users cannot punch out to get a shell on an Azure AKS controller or go digging around inside /proc to figure out what other tenants are doing unless the provider is on top of their configuration and regularly updates to match CVEs. "most users" will end up using one of the frameworks written by a "big boy" for their stuff, and they'll end up using what's convenient for cloud providers. The goal of microvms is ultimately to remove everything you're talking about from the equation. Kata and other microvm frameworks aim to be basically jsut another CRI which removes the "deep plumbing" you're talking about. The onus is on them to make this work, but there's an enormous financial payoff, and you'll end up with this whether you think it's worthwhile or not.
  - convolvatron 4y ago
    in a related vein, most of the distinctions that are being brought up around containers vs vms (pricing, debugability, tooling, overhead) are nothing fundamental at all. they are both executable formats that cut at different layers, and there is really no reason why features of one can't be easily brought to the other. operating above these abstractions can save us time, but please stop confusing the artifacts of implementation with some kind of fundamental truth. its really hindering our progress.
    - evol262 4y ago
      Bringing the features of one to the other is exactly what microvms means.
    - galangalalgol 4y ago
      How would you compare the security of running in wasmer vs the other two options. I know it is a bit apples and oranges. Just curious if it would be harder to break out of a wasm sandbox, or a vm.
  - 4y ago
- pjmlp 4y ago
  Except stuff like Kata Containers exist exactly because containers alone is not enough. WebAssembly on the cloud is only re-inventing what other bytecode based platforms have been doing for decades, but VC need ideas to invest into apparently.
discardedrefuse 4y ago
> So from that perspective, when I've peeked at firecracker, kata containers, etc, the "small dev dx" isn't quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn't supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn't quite there yet (but very close). This is pretty much me and my homelab. I haven't visited it in a while, but Weave Ignite might be of interest here. https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite
NorwegianDude 4y ago
I've been using containers since 2007 for isolating workloads. I don't really like Docker for production either because of the network overhead with the "docker-way" of doing things. LXD is definetly my favorite container tool.
- pojzon 4y ago
  How differently LXD manages isolation in comparison to docker ? I suppose both create netns, bridge, ifs ?
  - lstodd 4y ago
    It's the same stuff - namespaces, etc. But it doesn't shove greasy fingers into network config like docker. More a tooling question/approach than tech.
    - zonethundery 4y ago
      Have similar feelings about docker. LXD containers through a bridged interface fit my mental model/use case.
  - antod 4y ago
    LXC/LXD use the same kernel isolation/security features Docker does - namespaces, cgroups, capabilities etc. After all, it is the kernel functionality lets you run something as a container. Docker and LXC/LXD are different management / FS packaging layers on top of that.
    - staticassertion 4y ago
      I assume it's not using seccomp, which Docker uses, although seccomp is not Docker specific and you can go grab their policy.
- indigodaddy 4y ago
  LXD can now do VM as well. https://ubuntu.com/blog/lxd-virtual-machines-an-overview https://ubuntu.com/blog/lxd-virtual-machines-an-overview
stavros 4y ago
I was in the same boat as you and built something simple that I really like: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster It'll just pull some repos, make sure the containers are up, and make your configuration simple and discoverable. It really works great at that.
gaocegege 4y ago
I do not think Docker is the end game. Firecracker/kata containers neither. https://github.com/tensorchord/envd https://github.com/tensorchord/envd is my favorite container tool. It provides the build language based on Python and the build is optimized for this scenario. There may be more runtime and image build tools for both containers and VMs. And different tools may be designed for different purposes. That's the future I believe.

xani_ 4y ago

They went to trash because containers are more convenient to use and saving few MBs of disk/memory is not what most users care. The whole idea was pretty much either use custom kernel (which inevitably have way less info on how to debug anything in it), and re-do all of the network and storage plumbing containers already do via the OS they are running one. OR just very slim linux one which at least people know how to use but STILL is more complexity than "just a blob with some namespaces in it" and STILL requires a bunch of config and data juggling between hypervisor and VM just to share some host files to the guest. Either way to get to the level of "just a slim layer of code between hypervisor and your code" you need to do a quite a lot of deep plumbing and when anything goes wrong debugging is harder. All to get some perceived security and no better performance than just... running the binary in a container. It did percolate into "slim containers" idea where the container is just statically compiled binary + few configs and while it does have same problems with debuggability, you can just attach sidecart to it I guess next big hype will be "VM bUt YoU RuN WebAsSeMbLy In CuStOm KeRnEl"

evol262 4y ago
Virtualization is not just "perceived" security over containerization. From CPU rings on down, it offers dramatically more isolation for security than containerization does. This isn't about 'what most users care' about either. Most users don't really care about 99% of what container orchestration platforms offer. The providers do absolutely care that malicious users cannot punch out to get a shell on an Azure AKS controller or go digging around inside /proc to figure out what other tenants are doing unless the provider is on top of their configuration and regularly updates to match CVEs. "most users" will end up using one of the frameworks written by a "big boy" for their stuff, and they'll end up using what's convenient for cloud providers. The goal of microvms is ultimately to remove everything you're talking about from the equation. Kata and other microvm frameworks aim to be basically jsut another CRI which removes the "deep plumbing" you're talking about. The onus is on them to make this work, but there's an enormous financial payoff, and you'll end up with this whether you think it's worthwhile or not.
- convolvatron 4y ago
  in a related vein, most of the distinctions that are being brought up around containers vs vms (pricing, debugability, tooling, overhead) are nothing fundamental at all. they are both executable formats that cut at different layers, and there is really no reason why features of one can't be easily brought to the other. operating above these abstractions can save us time, but please stop confusing the artifacts of implementation with some kind of fundamental truth. its really hindering our progress.
  - evol262 4y ago
    Bringing the features of one to the other is exactly what microvms means.
  - galangalalgol 4y ago
    How would you compare the security of running in wasmer vs the other two options. I know it is a bit apples and oranges. Just curious if it would be harder to break out of a wasm sandbox, or a vm.
- 4y ago
pjmlp 4y ago
Except stuff like Kata Containers exist exactly because containers alone is not enough. WebAssembly on the cloud is only re-inventing what other bytecode based platforms have been doing for decades, but VC need ideas to invest into apparently.

discardedrefuse 4y ago

> So from that perspective, when I've peeked at firecracker, kata containers, etc, the "small dev dx" isn't quite there yet, or maybe never will get there since the players target other spaces (aws, fly.io, etc). Stuff like a way to share volumes isn't supported, etc. Personally I find Dockers architecture a bit distasteful and Podmans tooling isn't quite there yet (but very close). This is pretty much me and my homelab. I haven't visited it in a while, but Weave Ignite might be of interest here. https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite

NorwegianDude 4y ago

I've been using containers since 2007 for isolating workloads. I don't really like Docker for production either because of the network overhead with the "docker-way" of doing things. LXD is definetly my favorite container tool.

pojzon 4y ago
How differently LXD manages isolation in comparison to docker ? I suppose both create netns, bridge, ifs ?
- lstodd 4y ago
  It's the same stuff - namespaces, etc. But it doesn't shove greasy fingers into network config like docker. More a tooling question/approach than tech.
  - zonethundery 4y ago
    Have similar feelings about docker. LXD containers through a bridged interface fit my mental model/use case.
- antod 4y ago
  LXC/LXD use the same kernel isolation/security features Docker does - namespaces, cgroups, capabilities etc. After all, it is the kernel functionality lets you run something as a container. Docker and LXC/LXD are different management / FS packaging layers on top of that.
  - staticassertion 4y ago
    I assume it's not using seccomp, which Docker uses, although seccomp is not Docker specific and you can go grab their policy.
indigodaddy 4y ago
LXD can now do VM as well. https://ubuntu.com/blog/lxd-virtual-machines-an-overview https://ubuntu.com/blog/lxd-virtual-machines-an-overview

stavros 4y ago

I was in the same boat as you and built something simple that I really like: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster It'll just pull some repos, make sure the containers are up, and make your configuration simple and discoverable. It really works great at that.

gaocegege 4y ago

I do not think Docker is the end game. Firecracker/kata containers neither. https://github.com/tensorchord/envd https://github.com/tensorchord/envd is my favorite container tool. It provides the build language based on Python and the build is optimized for this scenario. There may be more runtime and image build tools for both containers and VMs. And different tools may be designed for different purposes. That's the future I believe.