Popcorn Linux main idea started when trying to run Barrelfish OS (from ETHZ) we asked ourselfs "can I apply the same ideas to Linux"? Then we ended up in taking

Popcorn Linux main idea started when trying to run Barrelfish OS (from ETHZ) we asked ourselfs "can I apply the same ideas to Linux"? Then we ended up in taking a somewhat different route in terms of system design for different reasons, some were technical. My personal perspective on the project was to try to understand if I can power up and down heterogeneous cores at any time, scaling indefinitely in the numbers of cores ... I changed that idea a little bit since I started the project (2012). The current version of Popcorn Linux supports aarch64 + x86_64. Popcorn Linux runs only on OS-capable processors (no GPUs, no FPGAs). The kernel is not split up. You run different instances of the kernel on different ISA processors - one kernel per processor island (that is a set of processor cores with homogeneous characteristics). When a kernel boots up, if it is not the first one, it will join the community of Popcorn kernels. Operating system services are distributed, therefore things like memory are globally managed. An applications to run among kernels must be compiled with the Popcorn Linux toolchain, and must be executed inside the Popcorn namespace (just an additional Linux namespace), in which the applications can see all the available memory and CPU cores. When your applications is in the namespace the only thing that it has to do is to call sched_setaffinity to migrate to another core - as easy as POSIX/SMP. You can migrate between different ISA at any point in the code (there is no limitation such as in offloading, that you have to go back as soon as the offloaded function is finished) - this is thanks to the stack rewriter (see below). The current way Popcorn compiles an application to be run on aarch64 + x86_64 is to produce 2 binaries, one for aarch64 and one for x86_64. For now, the compiler toolchain is modified to align all symbols in each binary at the same address, and all data types have the same sizes; all alignments and paddings are forced the same way. For this reason we used musl, and yes, you need to recompile each library (although this is not strictly necessary in some cases). The executable is statically linked. The operating system cannot cross-ISA migrate your application at arbitrary points, the migration should be explicit in the code itself (but we added some OS-suggested migration, at automatically placed migration points). (If you want to cross-ISA migrate with OS control, you need emulation until a valid migration point, you can check the work from Venkat + Tullsen.) Finally, because the executable code of different ISAs sees the address space identically there is no need of state transformation during migration - this enables the developer to migrate a single thread, or a subset of threads of your application. The only thing that Popcorn transforms during cross-ISA migration is the stack, thanks to Rob that wrote the stack transformation routine. We decided to do not implement a single stack layout because we believed that the current stack ABI is optimized for performance, but as far as I remember we never tested the alternative option.