10 ms·

Fakecracker: NetBSD as a Function Based MicroVM

166 points by yankcrime 6y ago

hultner 6y ago
Intersting, always fun to read some BSD content. One thing that bothered me though is that no bootup times are specified anywhere in the article which after all focuses on exactly this, it would be very nice to see boot time of Fakecracker vs Docker vs NetBSD amd64 w. bootloader (and unmodified), and compared to the minimal Linux kernel menitoned in the begining.
- sigjuice 6y ago
  Docker is not exactly "booted" in the traditional meaning of starting a (real or virtual) computer.
  - hultner 6y ago
    No but the process is still started and I suppose there's still some sort of setup time for networking, context group, etc.
- flatiron 6y ago
  the animation at the bottom shows netbsd + nginx starting in 1 second
winrid 6y ago
NetBSD is so fun to play with. Highly recommended.
- lbhdc 6y ago
  I don't really know much about NetBSD. Where is a good place to jump in?
  - bch 6y ago
    Start at the beginning, continue until the end, then stop. NetBSD-9 was just released; grab appropriate install img[0] and dive in! When you need help or community, #netbsd on freenode[1] is just a few keystrokes away. [0] http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/ http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/ [1] irc.freenode.org w your favourite IRC client
  - inetsee 6y ago
    The NetBSD Project has its own collection of tutorials here: https://wiki.netbsd.org/tutorials/ https://wiki.netbsd.org/tutorials/
yjftsjthsd-h 6y ago
One limitation of this system as it currently exists seems to be that you need a working i386 NetBSD host to built the filesystem. It might not be worth the effort once pkgin gets involved, but in theory I would expect this to be perfectly possibly to get around; after all, NetBSD's own build.sh is perfectly happy to build a full system image for ex. i386 NetBSD from source on ex. an amd64 Linux host. Or, if you didn't want to deal with cross-compiling, perhaps it would make things easier to build a Fakecracker VM that itself builds root filesystems, analogously to building docker images from a container using dind.
- pwdisswordfish2 6y ago
  "One limitation of this system as it currently exists seems to be that you need a working i386 NetBSD host to build the filesystem." One does not need a NetBSD host. This is not the most time/space efficient method I can think of, but it's the easiest to explain... On Linux host: 1. Download for x in gnusrc sharesrc syssrc src;do ftp ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/source/sets/$x.tgz&& tar xzf $x -C /;done 2. Compile toolchain mkdir /usr/obj PATH=/bin:/sbin:/usr/bin:/usr/sbin ./build.sh -m i386 tools 3. Make filesystem Using makefs instead of vnconfig + mount + newfs has the advantage that makefs is part of the toolchain while vnconfig is not makefs root.img fakecracker
  - anthk 6y ago
    Wouldn't that crap out /usr/include?
    - deleted 6y ago
      [deleted]
    - yjftsjthsd-h 6y ago
      Nope; all the netbsd source tarballs extract to [/]usr/src
  - yjftsjthsd-h 6y ago
    Yep, with a tiny bit of fiddling this does indeed work:) Thanks!
api 6y ago
I'd like to see a small lean kernel like NetBSD ported to WASM. You could run services in the browser or in a relatively safe and portable local WASM VM. Would be an interesting way to distribute certain kinds of software. Performance wouldn't be up to par with native, but it wouldn't be too bad.
- gen3 6y ago
  I was thinking the same thing. I didn’t see the size of the image after his optimizations anywhere. I found a project that emulates an x86 processor, and they already have two BSD images built. https://copy.sh/v86/ https://copy.sh/v86/
  - aspenmayer 6y ago
    That’s amazing. I wish they could fix the bugs on mobile where scrolling the viewport also moves the mouse, which almost makes it entirely unusable. That it works at all on mobile is amazing, all the same.
- greenshackle2 6y ago
  And then let's ship desktop apps in electron running NetBSD. And we will finally have come full circle :)
  - gcbw3 6y ago
    Literally in this case. netBSD -> browser -> jsBSD -> browser -> jsBSD ->
  - api 6y ago
    Can you build NetBSD to run as a user process like user mode Linux? A cross platform user mode "kernel" would also be a pretty interesting runtime.
- rvp-x 6y ago
  Someone compiled NetBSD (rump) to JavaScript before WASM was a thing :-) https://blog.netbsd.org/tnf/entry/kernel_drivers_compiled_to_javascript https://blog.netbsd.org/tnf/entry/kernel_drivers_compiled_to... The link seems to be down, I found this as a replacement http://197.155.77.5/NetBSD/misc/pooka/rump.js/ http://197.155.77.5/NetBSD/misc/pooka/rump.js/
MintelIE 6y ago
Think of the overhead which could be saved if services could be "spun up" in chrooted environments or even just as members of traditional Unix-style user-private groups! I know, I know, this is a newish concept which seems crazy. But it could work.
- anthk 6y ago
  Ahem. Chroot has lots of ways to escape.
  - MintelIE 6y ago
    The article mentions this but links to a Google search as "proof." Even VM's have lots of ways to escape these days. Chroot vulnerabilities have been discovered and fixed over time, as any security issue in an operating system is. It's not accurate to say that they are insecure in a blanket fashion especially these days. My opinion is that we should be using Unix as it is meant to be used, as on operating system, and using its time worn facilities meant for purposes such as security and sandboxing. They are very well tested and the solutions are baked-in and generally pretty small in terms of both code and overhead when compared to spinning up a VM, for instance. There are arguments for using VMs for security and scaling but they don't always win over just one modest local server in either domain. We're not all serving Google search after all.
    - lsofzz 6y ago
      > Chroot vulnerabilities have been discovered and fixed over time, as any security issue in an operating system is. It's not accurate to say that they are insecure in a blanket fashion especially these days. My opinion is that we should be using Unix as it is meant to be used, as on operating system, and using its time worn facilities meant for purposes such as I think you are right in general terms but I do not think an attacker will play by the rule and use `chroot` in UNIX-like OS as it is "meant to be used". They will use whatever means necessary - whether it be 0day or other un-patched vulnerability to get a break out.
      - 6y ago
lsofzz 6y ago
I assume this is a play on the word "firecracker"? :D https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker
- jlgaddis 6y ago
  No need to assume anything. The article starts out: > In November 2018 AWS published an Open Source tool called Firecracker, ...
  - lsofzz 6y ago
    > No need to assume anything. The article starts out: Aha. Reading now.
not_kidding 6y ago
If "blacklist" and "master" are considered racist, how is this not 1,000 times more beligerent. "Fake Cracker" is absolutely beyond racist, and if you think otherwise, maybe you don't understand how racist you really are.
shodan666 6y ago
The NetBSD 9.0 release provides supported binary distributions for the following systems: NetBSD/aarch64 Arm 64-bit NetBSD/acorn32 Acorn RiscPC/A7000, VLSI RC7500 NetBSD/algor Algorithmics, Ltd. MIPS evaluation boards NetBSD/alpha Digital/Compaq Alpha (64-bit) NetBSD/amd64 AMD family processors like Opteron, Athlon64, and Intel CPUs with EM64T extension NetBSD/amiga Commodore Amiga and MacroSystem DraCo NetBSD/amigappc PowerPC-based Amiga boards. NetBSD/arc MIPS-based machines following the Advanced RISC Computing spec NetBSD/atari Atari TT030, Falcon, Hades NetBSD/bebox Be Inc's BeBox NetBSD/cats Chalice Technology's CATS and Intel's EBSA-285 evaluation boards NetBSD/cesfic CES FIC8234 VME processor board NetBSD/cobalt Cobalt Networks' MIPS-based Microservers NetBSD/dreamcast Sega Dreamcast game console NetBSD/emips The Extensible MIPS architecture from Microsoft Research NetBSD/epoc32 Psion EPOC PDAs NetBSD/evbarm Various Arm-based evaluation boards and appliances NetBSD/evbmips Various MIPS-based evaluation boards and appliances NetBSD/evbppc Various PowerPC-based evaluation boards and appliances NetBSD/evbsh3 Various Hitachi Super-H SH3 and SH4-based evaluation boards and appliances NetBSD/ews4800mips NEC's MIPS-based EWS4800 workstation NetBSD/hp300 Hewlett-Packard 9000/300 and 400 series NetBSD/hpcarm StrongArm based Windows CE PDA machines NetBSD/hpcmips MIPS-based Windows CE PDA machines NetBSD/hpcsh Hitachi Super-H based Windows CE PDA machines NetBSD/hppa Hewlett-Packard 9000 Series 700 workstations NetBSD/i386 IBM PCs and PC clones with i486-family processors and up NetBSD/ibmnws IBM Network Station 1000 NetBSD/iyonix Castle Technology's Iyonix Arm based PCs NetBSD/landisk SH4 processor based NAS appliances NetBSD/luna68k OMRON Tateisi Electric's LUNA series NetBSD/mac68k Apple Macintosh with Motorola 68k CPU NetBSD/macppc Apple PowerPC-based Macintosh and clones NetBSD/mipsco MIPS Computer Systems Inc. family of workstations and servers NetBSD/mmeye Brains mmEye multimedia server NetBSD/mvme68k Motorola MVME 68k Single Board Computers NetBSD/mvmeppc Motorola PowerPC VME Single Board Computers NetBSD/netwinder StrongArm based NetWinder machines NetBSD/news68k Sony's 68k-based “NET WORK STATION” series NetBSD/newsmips Sony's MIPS-based “NET WORK STATION” series NetBSD/next68k NeXT 68k “black” hardware NetBSD/ofppc OpenFirmware PowerPC machines NetBSD/pmax Digital MIPS-based DECstations and DECsystems NetBSD/prep PReP (PowerPC Reference Platform) and CHRP machines NetBSD/rs6000 IBM RS/6000 MCA-based PowerPC machines. NetBSD/sandpoint Motorola Sandpoint reference platform, including many PPC-based NAS boxes NetBSD/sgimips Silicon Graphics' MIPS-based workstations NetBSD/shark Digital DNARD (“shark”) NetBSD/sparc Sun SPARC (32-bit) and UltraSPARC (in 32-bit mode) NetBSD/sparc64 Sun UltraSPARC (in native 64-bit mode) NetBSD/sun2 Sun Microsystems Sun 2 machines with Motorola 68010 CPU NetBSD/sun3 Motorola 68020 and 030 based Sun 3 and 3x machines NetBSD/vax Digital VAX NetBSD/x68k Sharp X680x0 series NetBSD/xen The Xen virtual machine monitor NetBSD/zaurus Sharp Arm PDAs
gdamjan1 6y ago
Does NetBSD support initramfs (the name of the Linux feature)? That would avoid having to create a disk image by running a bare system from a ramfs/tmpfs. I experimented with something like that with Linux and cloud-hypervisor (a firecracker fork): https://gist.github.com/gdamjan/1f260b58eb9fb1ba62d2234958582405 https://gist.github.com/gdamjan/1f260b58eb9fb1ba62d223495858...

166 points by yankcrime 6y ago

Intersting, always fun to read some BSD content. One thing that bothered me though is that no bootup times are specified anywhere in the article which after all focuses on exactly this, it would be very nice to see boot time of Fakecracker vs Docker vs NetBSD amd64 w. bootloader (and unmodified), and compared to the minimal Linux kernel menitoned in the begining.

sigjuice 6y ago
Docker is not exactly "booted" in the traditional meaning of starting a (real or virtual) computer.
- hultner 6y ago
  No but the process is still started and I suppose there's still some sort of setup time for networking, context group, etc.
flatiron 6y ago
the animation at the bottom shows netbsd + nginx starting in 1 second

winrid 6y ago

NetBSD is so fun to play with. Highly recommended.

lbhdc 6y ago
I don't really know much about NetBSD. Where is a good place to jump in?
- bch 6y ago
  Start at the beginning, continue until the end, then stop. NetBSD-9 was just released; grab appropriate install img[0] and dive in! When you need help or community, #netbsd on freenode[1] is just a few keystrokes away. [0] http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/ http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/ [1] irc.freenode.org w your favourite IRC client
- inetsee 6y ago
  The NetBSD Project has its own collection of tutorials here: https://wiki.netbsd.org/tutorials/ https://wiki.netbsd.org/tutorials/

yjftsjthsd-h 6y ago

One limitation of this system as it currently exists seems to be that you need a working i386 NetBSD host to built the filesystem. It might not be worth the effort once pkgin gets involved, but in theory I would expect this to be perfectly possibly to get around; after all, NetBSD's own build.sh is perfectly happy to build a full system image for ex. i386 NetBSD from source on ex. an amd64 Linux host. Or, if you didn't want to deal with cross-compiling, perhaps it would make things easier to build a Fakecracker VM that itself builds root filesystems, analogously to building docker images from a container using dind.

pwdisswordfish2 6y ago
"One limitation of this system as it currently exists seems to be that you need a working i386 NetBSD host to build the filesystem." One does not need a NetBSD host. This is not the most time/space efficient method I can think of, but it's the easiest to explain... On Linux host: 1. Download for x in gnusrc sharesrc syssrc src;do ftp ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-9.0/source/sets/$x.tgz&& tar xzf $x -C /;done 2. Compile toolchain mkdir /usr/obj PATH=/bin:/sbin:/usr/bin:/usr/sbin ./build.sh -m i386 tools 3. Make filesystem Using makefs instead of vnconfig + mount + newfs has the advantage that makefs is part of the toolchain while vnconfig is not makefs root.img fakecracker
- anthk 6y ago
  Wouldn't that crap out /usr/include?
  - deleted 6y ago
    [deleted]
  - yjftsjthsd-h 6y ago
    Nope; all the netbsd source tarballs extract to [/]usr/src
- yjftsjthsd-h 6y ago
  Yep, with a tiny bit of fiddling this does indeed work:) Thanks!

api 6y ago

I'd like to see a small lean kernel like NetBSD ported to WASM. You could run services in the browser or in a relatively safe and portable local WASM VM. Would be an interesting way to distribute certain kinds of software. Performance wouldn't be up to par with native, but it wouldn't be too bad.

gen3 6y ago
I was thinking the same thing. I didn’t see the size of the image after his optimizations anywhere. I found a project that emulates an x86 processor, and they already have two BSD images built. https://copy.sh/v86/ https://copy.sh/v86/
- aspenmayer 6y ago
  That’s amazing. I wish they could fix the bugs on mobile where scrolling the viewport also moves the mouse, which almost makes it entirely unusable. That it works at all on mobile is amazing, all the same.
greenshackle2 6y ago
And then let's ship desktop apps in electron running NetBSD. And we will finally have come full circle :)
- gcbw3 6y ago
  Literally in this case. netBSD -> browser -> jsBSD -> browser -> jsBSD ->
- api 6y ago
  Can you build NetBSD to run as a user process like user mode Linux? A cross platform user mode "kernel" would also be a pretty interesting runtime.
rvp-x 6y ago
Someone compiled NetBSD (rump) to JavaScript before WASM was a thing :-) https://blog.netbsd.org/tnf/entry/kernel_drivers_compiled_to_javascript https://blog.netbsd.org/tnf/entry/kernel_drivers_compiled_to... The link seems to be down, I found this as a replacement http://197.155.77.5/NetBSD/misc/pooka/rump.js/ http://197.155.77.5/NetBSD/misc/pooka/rump.js/

MintelIE 6y ago

Think of the overhead which could be saved if services could be "spun up" in chrooted environments or even just as members of traditional Unix-style user-private groups! I know, I know, this is a newish concept which seems crazy. But it could work.

anthk 6y ago
Ahem. Chroot has lots of ways to escape.
- MintelIE 6y ago
  The article mentions this but links to a Google search as "proof." Even VM's have lots of ways to escape these days. Chroot vulnerabilities have been discovered and fixed over time, as any security issue in an operating system is. It's not accurate to say that they are insecure in a blanket fashion especially these days. My opinion is that we should be using Unix as it is meant to be used, as on operating system, and using its time worn facilities meant for purposes such as security and sandboxing. They are very well tested and the solutions are baked-in and generally pretty small in terms of both code and overhead when compared to spinning up a VM, for instance. There are arguments for using VMs for security and scaling but they don't always win over just one modest local server in either domain. We're not all serving Google search after all.
  - lsofzz 6y ago
    > Chroot vulnerabilities have been discovered and fixed over time, as any security issue in an operating system is. It's not accurate to say that they are insecure in a blanket fashion especially these days. My opinion is that we should be using Unix as it is meant to be used, as on operating system, and using its time worn facilities meant for purposes such as I think you are right in general terms but I do not think an attacker will play by the rule and use `chroot` in UNIX-like OS as it is "meant to be used". They will use whatever means necessary - whether it be 0day or other un-patched vulnerability to get a break out.
    - 6y ago

lsofzz 6y ago

I assume this is a play on the word "firecracker"? :D https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker

jlgaddis 6y ago
No need to assume anything. The article starts out: > In November 2018 AWS published an Open Source tool called Firecracker, ...
- lsofzz 6y ago
  > No need to assume anything. The article starts out: Aha. Reading now.

not_kidding 6y ago

If "blacklist" and "master" are considered racist, how is this not 1,000 times more beligerent. "Fake Cracker" is absolutely beyond racist, and if you think otherwise, maybe you don't understand how racist you really are.

shodan666 6y ago

The NetBSD 9.0 release provides supported binary distributions for the following systems: NetBSD/aarch64 Arm 64-bit NetBSD/acorn32 Acorn RiscPC/A7000, VLSI RC7500 NetBSD/algor Algorithmics, Ltd. MIPS evaluation boards NetBSD/alpha Digital/Compaq Alpha (64-bit) NetBSD/amd64 AMD family processors like Opteron, Athlon64, and Intel CPUs with EM64T extension NetBSD/amiga Commodore Amiga and MacroSystem DraCo NetBSD/amigappc PowerPC-based Amiga boards. NetBSD/arc MIPS-based machines following the Advanced RISC Computing spec NetBSD/atari Atari TT030, Falcon, Hades NetBSD/bebox Be Inc's BeBox NetBSD/cats Chalice Technology's CATS and Intel's EBSA-285 evaluation boards NetBSD/cesfic CES FIC8234 VME processor board NetBSD/cobalt Cobalt Networks' MIPS-based Microservers NetBSD/dreamcast Sega Dreamcast game console NetBSD/emips The Extensible MIPS architecture from Microsoft Research NetBSD/epoc32 Psion EPOC PDAs NetBSD/evbarm Various Arm-based evaluation boards and appliances NetBSD/evbmips Various MIPS-based evaluation boards and appliances NetBSD/evbppc Various PowerPC-based evaluation boards and appliances NetBSD/evbsh3 Various Hitachi Super-H SH3 and SH4-based evaluation boards and appliances NetBSD/ews4800mips NEC's MIPS-based EWS4800 workstation NetBSD/hp300 Hewlett-Packard 9000/300 and 400 series NetBSD/hpcarm StrongArm based Windows CE PDA machines NetBSD/hpcmips MIPS-based Windows CE PDA machines NetBSD/hpcsh Hitachi Super-H based Windows CE PDA machines NetBSD/hppa Hewlett-Packard 9000 Series 700 workstations NetBSD/i386 IBM PCs and PC clones with i486-family processors and up NetBSD/ibmnws IBM Network Station 1000 NetBSD/iyonix Castle Technology's Iyonix Arm based PCs NetBSD/landisk SH4 processor based NAS appliances NetBSD/luna68k OMRON Tateisi Electric's LUNA series NetBSD/mac68k Apple Macintosh with Motorola 68k CPU NetBSD/macppc Apple PowerPC-based Macintosh and clones NetBSD/mipsco MIPS Computer Systems Inc. family of workstations and servers NetBSD/mmeye Brains mmEye multimedia server NetBSD/mvme68k Motorola MVME 68k Single Board Computers NetBSD/mvmeppc Motorola PowerPC VME Single Board Computers NetBSD/netwinder StrongArm based NetWinder machines NetBSD/news68k Sony's 68k-based “NET WORK STATION” series NetBSD/newsmips Sony's MIPS-based “NET WORK STATION” series NetBSD/next68k NeXT 68k “black” hardware NetBSD/ofppc OpenFirmware PowerPC machines NetBSD/pmax Digital MIPS-based DECstations and DECsystems NetBSD/prep PReP (PowerPC Reference Platform) and CHRP machines NetBSD/rs6000 IBM RS/6000 MCA-based PowerPC machines. NetBSD/sandpoint Motorola Sandpoint reference platform, including many PPC-based NAS boxes NetBSD/sgimips Silicon Graphics' MIPS-based workstations NetBSD/shark Digital DNARD (“shark”) NetBSD/sparc Sun SPARC (32-bit) and UltraSPARC (in 32-bit mode) NetBSD/sparc64 Sun UltraSPARC (in native 64-bit mode) NetBSD/sun2 Sun Microsystems Sun 2 machines with Motorola 68010 CPU NetBSD/sun3 Motorola 68020 and 030 based Sun 3 and 3x machines NetBSD/vax Digital VAX NetBSD/x68k Sharp X680x0 series NetBSD/xen The Xen virtual machine monitor NetBSD/zaurus Sharp Arm PDAs

gdamjan1 6y ago

Does NetBSD support initramfs (the name of the Linux feature)? That would avoid having to create a disk image by running a bare system from a ramfs/tmpfs. I experimented with something like that with Linux and cloud-hypervisor (a firecracker fork): https://gist.github.com/gdamjan/1f260b58eb9fb1ba62d2234958582405 https://gist.github.com/gdamjan/1f260b58eb9fb1ba62d223495858...