4 ms·

i love the convenience and ease-of-use but worry about the security compared to full-blown vm

by coderatlarge 1y ago

unsnap_biceps 1y ago
Kata containers is a nice compromise. Each container is run as a microvm
- coderatlarge 1y ago
  thanks! so they achieve the convenience of docker with the added security of full-blown kvm? trading some perf and resource-use? https://katacontainers.io/ https://katacontainers.io/
  - unsnap_biceps 1y ago
    Yes. Microvms are stripped down to the basic hardware needed (AWS' Firecracker for example), so they 'boot' really fast, in the tenths of seconds for my containers, but you do have the extra resource overhead of running a second kernel and the performance reduction of the VM context switches. That said, it's minor enough that I feel the security tradeoff is well worth it.
- rickydroll 1y ago
  On the surface, Kata appears to be a variation of LXC/LXD.
  - coderatlarge 1y ago
    meaning not a full-blown vm security boundary?
    - unsnap_biceps 1y ago
      It is
      - coderatlarge 1y ago
        thank you for posting the qemu details above
  - unsnap_biceps 1y ago
    LXC/LXD still shares the same host kernel. Kata runs a full instance of qemu including its own kernel. FWIW, here's an slightly redacted example qemu instance launched by kata /opt/kata-3.18.0/bin/qemu-system-x86_64 -name sandbox-FOO -uuid FOO -machine q35,accel=kvm,nvdimm=on -cpu host,pmu=off -qmp unix:fd=3,server=on,wait=off -m 2048M,slots=10,maxmem=1032958M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=off,addr=2,io-reserve=4k,mem-reserve=1m,pref64-reserve=1m -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/FOO/console.sock,server=on,wait=off -device nvdimm,id=nv0,memdev=mem0,unarmed=on -object memory-backend-file,id=mem0,mem-path=/opt/kata-3.18.0/share/kata-containers/kata-ubuntu-noble.image,size=268435456,readonly=on -device virtio-scsi-pci,id=scsi0,disable-modern=false -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=4,id=vsock-FOO,guest-cid=FOO -chardev socket,id=char-FOO,path=/run/vc/vm/FOO/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-FOO,tag=kataShared,queue-size=1024 -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /opt/kata-3.18.0/share/kata-containers/vmlinux-6.12.28-157 -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 console=hvc0 console=hvc1 quiet systemd.show_status=false panic=1 nr_cpus=88 selinux=0 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 -pidfile /run/vc/vm/FOO/pid -smp 1,cores=1,threads=1,sockets=88,maxcpus=88

i love the convenience and ease-of-use but worry about the security compared to full-blown vm

unsnap_biceps 1y ago

Kata containers is a nice compromise. Each container is run as a microvm

coderatlarge 1y ago
thanks! so they achieve the convenience of docker with the added security of full-blown kvm? trading some perf and resource-use? https://katacontainers.io/ https://katacontainers.io/
- unsnap_biceps 1y ago
  Yes. Microvms are stripped down to the basic hardware needed (AWS' Firecracker for example), so they 'boot' really fast, in the tenths of seconds for my containers, but you do have the extra resource overhead of running a second kernel and the performance reduction of the VM context switches. That said, it's minor enough that I feel the security tradeoff is well worth it.
rickydroll 1y ago
On the surface, Kata appears to be a variation of LXC/LXD.
- coderatlarge 1y ago
  meaning not a full-blown vm security boundary?
  - unsnap_biceps 1y ago
    It is
    - coderatlarge 1y ago
      thank you for posting the qemu details above
- unsnap_biceps 1y ago
  LXC/LXD still shares the same host kernel. Kata runs a full instance of qemu including its own kernel. FWIW, here's an slightly redacted example qemu instance launched by kata /opt/kata-3.18.0/bin/qemu-system-x86_64 -name sandbox-FOO -uuid FOO -machine q35,accel=kvm,nvdimm=on -cpu host,pmu=off -qmp unix:fd=3,server=on,wait=off -m 2048M,slots=10,maxmem=1032958M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=off,addr=2,io-reserve=4k,mem-reserve=1m,pref64-reserve=1m -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/FOO/console.sock,server=on,wait=off -device nvdimm,id=nv0,memdev=mem0,unarmed=on -object memory-backend-file,id=mem0,mem-path=/opt/kata-3.18.0/share/kata-containers/kata-ubuntu-noble.image,size=268435456,readonly=on -device virtio-scsi-pci,id=scsi0,disable-modern=false -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=4,id=vsock-FOO,guest-cid=FOO -chardev socket,id=char-FOO,path=/run/vc/vm/FOO/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-FOO,tag=kataShared,queue-size=1024 -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /opt/kata-3.18.0/share/kata-containers/vmlinux-6.12.28-157 -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 console=hvc0 console=hvc1 quiet systemd.show_status=false panic=1 nr_cpus=88 selinux=0 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 -pidfile /run/vc/vm/FOO/pid -smp 1,cores=1,threads=1,sockets=88,maxcpus=88