4 ms·

I played with Weave Ignite the other day which is a Docker-like CLI for Firecracker. Sure there were some rough edges but the overall experience was pretty good

I played with Weave Ignite the other day which is a Docker-like CLI for Firecracker. Sure there were some rough edges but the overall experience was pretty good. If you are familiar with the Docker CLI you will be able to get some virtual machines up and running very quickly. Two questions in case someone from Weave tunes into the discussion: I got the impression that VMs needed an SSH server to be accessible. Is this correct and if so will it be possible to implement something similar to docker exec so that I won't need an SSH server on every VM? > At the moment ignite and ignited need root privileges on the host to operate due to certain operations (e.g. mount). This will change in the future. Is there a timetable and could you perhaps elaborate a bit as to why it currently requires root? (I don't know anything about virtual machine internals so this isn't a passive-aggressive question from my side. It's genuine curiosity.)

legulere 7y ago
At least crosvm which this is based on offers serial device emulation.
bennyz 7y ago
>Is there a timetable and could you perhaps elaborate a bit as to why it currently requires root? (I don't know anything about virtual machine internals so this isn't a passive-aggressive question from my side. It's genuine curiosity.) Not from Weave, but I might have an idea as I've played with Firecracker a bit. When you start up a Firecracker VM, you need to provide it with a rootfs drive, which is a file containing the root file system to be used for the VM. Ignite uses OCI images, so I guess they are doing something similar to [1] in code, the `mount` part requires sudo, so that would be my guess to why you need root. [1] https://github.com/firecracker-microvm/firecracker/blob/master/docs/rootfs-and-kernel-setup.md#creating-a-rootfs-image https://github.com/firecracker-microvm/firecracker/blob/mast...
imhoguy 7y ago
Containers are provided by host kernel cgroups and namespaces, therefore the kernel implements attach (exec) operation which is practically running a new proces (e.g. bash) in a cgroup (container). Virtual Machines are provided by software or hardware emulation which run separate guest OS with own kernel. There is no standard way for a host to let you run any process and interact with its stdio inside guest OS because the host simply is not aware what you exactly run inside. The solution is to have an agreed connectivity standard both on the guest and the host. The guest can provide SSH server, telnet server serial terminal, irc bot or some other kind of control capability. Then of course host needs a tooling too, e.g. SSH client.
- CameronNemo 7y ago
  Are there any real alternatives to SSH and/or sftp? E.g. a mutual TLS authenticated HTTP server...
  - wmf 7y ago
    virtio-vsock
- Demiurge 7y ago
  Why not use a virtual terminal? That seems like a pretty standard machine interface to use if the machine is virtualized.
yebyen 7y ago
> I got the impression that VMs needed an SSH server to be accessible. Is this correct and if so will it be possible to implement something similar to docker exec so that I won't need an SSH server on every VM? If you want the capability to exec processes from the host into the VM, I think either Docker API or Kube API is the thing for that, as I understand it. If you could kernel exec processes directly into a VM, then it would not be isolated from the host, this seems almost tautological. You can arrange for process execution another way than SSH, Docker, or Kube API but regardless of what shape it takes, it will still be an entry point in similar fashion to any of these, as the MicroVM or VM runs its own kernel on KVM and does not talk to the host in this way. Perhaps someone knows more about KVM and can clue me in further if there is more here than meets the eye and maybe what you said is possible. If you don't need the isolation of a proper VM and were only looking for a roughly VM-shaped system that you CAN "kernel exec" or use nsenter to get processes into, you should look at Footloose[1]. I'm suggesting what you are looking for is actually a container that looks more like a VM or bare-metal machine from the perspective of inits and with the vantage point of the processes running inside. By default, Footloose nodes are running SSH and SystemD, may appear to work similarly to Ignite VMs, but they are Docker containers that may or may not run privileged mode. So, if it suits you, then you could start up Footloose "VMs" as I still call them, strip SSH from them, then nsenter or Docker exec into them as you desire, or run Kubernetes on them and use the Kube API including exec. That is actually a lead in to the next project, known as Firekube[2], kind of a mashup of all these technologies plus one more (wksctl[3]). Firekube integrates both Ignite and Firecracker as well, so you can use it similarly on Linux, (where KVM support is available for Ignite), or MacOS, where Footloose runs container-VMs instead, both behave alike; this suite of projects all put together is a very slick and well integrated package IMHO. It is probably comparable in functionality to Minikube, but with GitOps baked right in. Disclosure: I am not working for Weaveworks, but we are good friends. [1]: https://github.com/weaveworks/footloose https://github.com/weaveworks/footloose [2]: https://github.com/weaveworks/wks-quickstart-firekube/ https://github.com/weaveworks/wks-quickstart-firekube/ [3]: https://github.com/weaveworks/wksctl https://github.com/weaveworks/wksctl
- mercora 7y ago
  i think using an emulated serial connection or similar driver interface could be part of a solution to this.
cpuguy83 7y ago
See https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container...
scarface74 7y ago
I’ve personally never needed to ssh into my “cattle” even when I could (EC2 instances in an autoscaling group, Docker containers run with Fargate) and I haven’t missed it when I couldn’t (lambda). For Fargate/Lambda all console output goes to CloudWatch, for EC2 tasks (legacy Windows), we use Serilog with a CloudWatch sink. But more generically, if you have to log into your cattle for troubleshooting, you probably need a better logging infrastructure. Then again, if you are referring to how to initially install software, wouldn’t you usually just create an image for it to run?
- tofflos 7y ago
  It's for initial installation. I use it to figure out what the image and orchestration settings should be. Packaging an application for containers and container orchestration takes me many, many, attempts to get right. Personally I'm unable to divine the correct combination of settings by reading documentation alone so I try something, enter the container, and look at the outcome.
- xorcist 7y ago
  It's nice to have ptrace once in a while.
  - scarface74 7y ago
    The only time I can think of where copious logging wasn’t good enough for “remote debugging” is when I was writing C code trying to figure out why I was killing the call stack or overwriting memory.
    - xorcist 7y ago
      Having metrics and logging is a requirement to find out that there is a problem, but it doesn't help much to find out what the problem is.
      - scarface74 7y ago
        It depends on the granularity of logging. What can local debugging tell you that copious logging at the “debug” level can’t?

legulere 7y ago

At least crosvm which this is based on offers serial device emulation.

bennyz 7y ago

>Is there a timetable and could you perhaps elaborate a bit as to why it currently requires root? (I don't know anything about virtual machine internals so this isn't a passive-aggressive question from my side. It's genuine curiosity.) Not from Weave, but I might have an idea as I've played with Firecracker a bit. When you start up a Firecracker VM, you need to provide it with a rootfs drive, which is a file containing the root file system to be used for the VM. Ignite uses OCI images, so I guess they are doing something similar to [1] in code, the `mount` part requires sudo, so that would be my guess to why you need root. [1] https://github.com/firecracker-microvm/firecracker/blob/master/docs/rootfs-and-kernel-setup.md#creating-a-rootfs-image https://github.com/firecracker-microvm/firecracker/blob/mast...

imhoguy 7y ago

Containers are provided by host kernel cgroups and namespaces, therefore the kernel implements attach (exec) operation which is practically running a new proces (e.g. bash) in a cgroup (container). Virtual Machines are provided by software or hardware emulation which run separate guest OS with own kernel. There is no standard way for a host to let you run any process and interact with its stdio inside guest OS because the host simply is not aware what you exactly run inside. The solution is to have an agreed connectivity standard both on the guest and the host. The guest can provide SSH server, telnet server serial terminal, irc bot or some other kind of control capability. Then of course host needs a tooling too, e.g. SSH client.

CameronNemo 7y ago
Are there any real alternatives to SSH and/or sftp? E.g. a mutual TLS authenticated HTTP server...
- wmf 7y ago
  virtio-vsock
Demiurge 7y ago
Why not use a virtual terminal? That seems like a pretty standard machine interface to use if the machine is virtualized.

yebyen 7y ago

> I got the impression that VMs needed an SSH server to be accessible. Is this correct and if so will it be possible to implement something similar to docker exec so that I won't need an SSH server on every VM? If you want the capability to exec processes from the host into the VM, I think either Docker API or Kube API is the thing for that, as I understand it. If you could kernel exec processes directly into a VM, then it would not be isolated from the host, this seems almost tautological. You can arrange for process execution another way than SSH, Docker, or Kube API but regardless of what shape it takes, it will still be an entry point in similar fashion to any of these, as the MicroVM or VM runs its own kernel on KVM and does not talk to the host in this way. Perhaps someone knows more about KVM and can clue me in further if there is more here than meets the eye and maybe what you said is possible. If you don't need the isolation of a proper VM and were only looking for a roughly VM-shaped system that you CAN "kernel exec" or use nsenter to get processes into, you should look at Footloose[1]. I'm suggesting what you are looking for is actually a container that looks more like a VM or bare-metal machine from the perspective of inits and with the vantage point of the processes running inside. By default, Footloose nodes are running SSH and SystemD, may appear to work similarly to Ignite VMs, but they are Docker containers that may or may not run privileged mode. So, if it suits you, then you could start up Footloose "VMs" as I still call them, strip SSH from them, then nsenter or Docker exec into them as you desire, or run Kubernetes on them and use the Kube API including exec. That is actually a lead in to the next project, known as Firekube[2], kind of a mashup of all these technologies plus one more (wksctl[3]). Firekube integrates both Ignite and Firecracker as well, so you can use it similarly on Linux, (where KVM support is available for Ignite), or MacOS, where Footloose runs container-VMs instead, both behave alike; this suite of projects all put together is a very slick and well integrated package IMHO. It is probably comparable in functionality to Minikube, but with GitOps baked right in. Disclosure: I am not working for Weaveworks, but we are good friends. [1]: https://github.com/weaveworks/footloose https://github.com/weaveworks/footloose [2]: https://github.com/weaveworks/wks-quickstart-firekube/ https://github.com/weaveworks/wks-quickstart-firekube/ [3]: https://github.com/weaveworks/wksctl https://github.com/weaveworks/wksctl

mercora 7y ago
i think using an emulated serial connection or similar driver interface could be part of a solution to this.

cpuguy83 7y ago

See https://github.com/firecracker-microvm/firecracker-containerd https://github.com/firecracker-microvm/firecracker-container...

scarface74 7y ago

I’ve personally never needed to ssh into my “cattle” even when I could (EC2 instances in an autoscaling group, Docker containers run with Fargate) and I haven’t missed it when I couldn’t (lambda). For Fargate/Lambda all console output goes to CloudWatch, for EC2 tasks (legacy Windows), we use Serilog with a CloudWatch sink. But more generically, if you have to log into your cattle for troubleshooting, you probably need a better logging infrastructure. Then again, if you are referring to how to initially install software, wouldn’t you usually just create an image for it to run?

tofflos 7y ago
It's for initial installation. I use it to figure out what the image and orchestration settings should be. Packaging an application for containers and container orchestration takes me many, many, attempts to get right. Personally I'm unable to divine the correct combination of settings by reading documentation alone so I try something, enter the container, and look at the outcome.
xorcist 7y ago
It's nice to have ptrace once in a while.
- scarface74 7y ago
  The only time I can think of where copious logging wasn’t good enough for “remote debugging” is when I was writing C code trying to figure out why I was killing the call stack or overwriting memory.
  - xorcist 7y ago
    Having metrics and logging is a requirement to find out that there is a problem, but it doesn't help much to find out what the problem is.
    - scarface74 7y ago
      It depends on the granularity of logging. What can local debugging tell you that copious logging at the “debug” level can’t?