4 ms·

Very interesting! I can't help but wonder why these technologies (the framework, and accompanying observability) aren't an OS-level feature; wondering if at thi

Very interesting! I can't help but wonder why these technologies (the framework, and accompanying observability) aren't an OS-level feature; wondering if at this point we need an OS at all, and why not ship a minimal kernel + the binary of an application built w/ this to run directly on the VM.

billfruit 5y ago
Yes truly, is there something like this for linux threads and processes?
- pkaye 5y ago
  The Linux Perf tool is pretty powerful. I've barely just scratched the surface myself. https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page As as a side note, Go and pretty good profiling tools. You can see a trace of goroutine execution and why a goroutine got scheduled out and when GC kicks in. https://about.sourcegraph.com/go/an-introduction-to-go-tool-trace-rhys-hiltner/ https://about.sourcegraph.com/go/an-introduction-to-go-tool-...
- mycoliza 5y ago
  A lot of the Tokio console UI was inspired by `htop`, which provides a pretty similar overview of processes and threads. It doesn't really have the same ability to inspect things like `pthread_mutex` and timerfds etc in the same way that the Tokio console can inspect the state of `tokio::sync::Mutex` and `tokio::time::Sleep`, though; although I wonder if something like that could be possible with eBPF...
xtagon 5y ago
It's been done, check out https://www.nerves-project.org/ https://www.nerves-project.org/ for an example of running a minimal kernel + Elixir (BEAM VM) on devices. As I understand it, it's powered by Buildroot and should be possible to do for other VMs/languages. Don't think I've seen anything like this for desktop OS's though!
kristjansson 5y ago
Unikernels are a thing e.g. https://mirage.io https://mirage.io. AFIACT uptake has been ... slow. You give up a lot to erase most or all of your OS, and only the most performance sensitive applications realize a benefit
pjmlp 5y ago
With languages with rich runtimes that is exactly the point. Actually in a way, POSIX is the missing C's runtime that wasn't made part of ISO C.
ReactiveJelly 5y ago
They're not OS-level because there isn't wide consensus on the right way to do things yet. Rust, C#, and JS all have similar concepts of async, but they're all slightly different. None of them would be trivial to adapt to other system langs like C and C++ - Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC. (And also compiler support, IIRC) I think there is a proposal to add coroutines in the new C++ standard, but I'm not sure how it would be done in the kernel. And sometimes I see people saying, "async is very bad, just use coroutines." Having only used Lua coroutines, I don't understand what the big difference is supposed to be. But mostly, these runtimes don't need OS-level support. Async is sort of a way to do concurrency without a kernel-level context switch for every task switch, right? If it's working so well in-process, why involve the OS at all? > wondering if at this point we need an OS at all Depends what you mean by OS. If you deploy in a container, of course your OS shares its kernel with the host. But for some user stories, (glares at Android) "OS" means all the software, including a Blink-based web browser, a plethora of GUI programs, and other things that I would rather call a "desktop environment" than an OS.
- paavohtl 5y ago
  > Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC C# actually does the same thing, though it indeed still relies on GC.
pcwalton 5y ago
In Linux there's work being done on a language-agnostic lightweight threading model, User-Mode Concurrency Groups or UMCG: https://lwn.net/Articles/863386/ https://lwn.net/Articles/863386/ One could imagine a similar, runtime-independent console for UMCG. Note, however, that the programming model for such a runtime would be much more similar to 1:1 threading (i.e. blocking I/O with threads) than async/await.
hayley-patton 5y ago
Arguably, they are an OS-level feature, in the form of OS threads and your favourite task manager; but it is more expensive to run a very large number of OS threads, compared to async tasks or green threads. Per Dan Ingalls [1], an operating system is a collection of things that don't fit into a language. There shouldn't be one. [1] Design Principles Behind Smalltalk <https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk....>
- c-cube 5y ago
  What about people who want to use several programs on one computer, possibly even written in different languages? This idea is ridiculous unless you think everyone shares your dream of running everything in a smalltalk image.
  - hayley-patton 5y ago
    I don't think everyone shares _my_ dreams, which mostly don't involve Smalltalk images, but this seems quite feasible. And, from experience, making languages interoperate is a hassle, so "written in different languages" isn't all great. In the particular case of Smalltalk, the operational(?) semantics of which are specified by a virtual machine, one can just compile targeting the virtual machine. c.f compiling Smalltalk and Java to the Self virtual machine, or all the compilers targeting the JVM nowadays.
im_down_w_otp 5y ago
That's basically what we did with https://github.com/auxoncorp/ferros https://github.com/auxoncorp/ferros, Bundle Rust programs together as tasks to run atop the formally verified seL4 microkernel.
erichocean 5y ago
> wondering if at this point we need an OS at all People have written unikernels, but they're not that interesting anymore when you can trivially constrain Linux to a single core, keep the remaining cores for you app (no context switching), and keep all of the Linux-y goodness for admin and debugging (SSH, gdb, etc.). All of the performance, none of the admin and deployment headaches. Basically, unless you truly can't afford that extra core, unikernels are all downside at this point.

billfruit 5y ago

Yes truly, is there something like this for linux threads and processes?

pkaye 5y ago
The Linux Perf tool is pretty powerful. I've barely just scratched the surface myself. https://perf.wiki.kernel.org/index.php/Main_Page https://perf.wiki.kernel.org/index.php/Main_Page As as a side note, Go and pretty good profiling tools. You can see a trace of goroutine execution and why a goroutine got scheduled out and when GC kicks in. https://about.sourcegraph.com/go/an-introduction-to-go-tool-trace-rhys-hiltner/ https://about.sourcegraph.com/go/an-introduction-to-go-tool-...
mycoliza 5y ago
A lot of the Tokio console UI was inspired by `htop`, which provides a pretty similar overview of processes and threads. It doesn't really have the same ability to inspect things like `pthread_mutex` and timerfds etc in the same way that the Tokio console can inspect the state of `tokio::sync::Mutex` and `tokio::time::Sleep`, though; although I wonder if something like that could be possible with eBPF...

xtagon 5y ago

It's been done, check out https://www.nerves-project.org/ https://www.nerves-project.org/ for an example of running a minimal kernel + Elixir (BEAM VM) on devices. As I understand it, it's powered by Buildroot and should be possible to do for other VMs/languages. Don't think I've seen anything like this for desktop OS's though!

kristjansson 5y ago

Unikernels are a thing e.g. https://mirage.io https://mirage.io. AFIACT uptake has been ... slow. You give up a lot to erase most or all of your OS, and only the most performance sensitive applications realize a benefit

pjmlp 5y ago

With languages with rich runtimes that is exactly the point. Actually in a way, POSIX is the missing C's runtime that wasn't made part of ISO C.

ReactiveJelly 5y ago

They're not OS-level because there isn't wide consensus on the right way to do things yet. Rust, C#, and JS all have similar concepts of async, but they're all slightly different. None of them would be trivial to adapt to other system langs like C and C++ - Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC. (And also compiler support, IIRC) I think there is a proposal to add coroutines in the new C++ standard, but I'm not sure how it would be done in the kernel. And sometimes I see people saying, "async is very bad, just use coroutines." Having only used Lua coroutines, I don't understand what the big difference is supposed to be. But mostly, these runtimes don't need OS-level support. Async is sort of a way to do concurrency without a kernel-level context switch for every task switch, right? If it's working so well in-process, why involve the OS at all? > wondering if at this point we need an OS at all Depends what you mean by OS. If you deploy in a container, of course your OS shares its kernel with the host. But for some user stories, (glares at Android) "OS" means all the software, including a Blink-based web browser, a plethora of GUI programs, and other things that I would rather call a "desktop environment" than an OS.

paavohtl 5y ago
> Rust requires compiler support to take apart async functions and put them back together as state machine, and the others lean on their GC C# actually does the same thing, though it indeed still relies on GC.

pcwalton 5y ago

In Linux there's work being done on a language-agnostic lightweight threading model, User-Mode Concurrency Groups or UMCG: https://lwn.net/Articles/863386/ https://lwn.net/Articles/863386/ One could imagine a similar, runtime-independent console for UMCG. Note, however, that the programming model for such a runtime would be much more similar to 1:1 threading (i.e. blocking I/O with threads) than async/await.

hayley-patton 5y ago

Arguably, they are an OS-level feature, in the form of OS threads and your favourite task manager; but it is more expensive to run a very large number of OS threads, compared to async tasks or green threads. Per Dan Ingalls [1], an operating system is a collection of things that don't fit into a language. There shouldn't be one. [1] Design Principles Behind Smalltalk <https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk....>

c-cube 5y ago
What about people who want to use several programs on one computer, possibly even written in different languages? This idea is ridiculous unless you think everyone shares your dream of running everything in a smalltalk image.
- hayley-patton 5y ago
  I don't think everyone shares _my_ dreams, which mostly don't involve Smalltalk images, but this seems quite feasible. And, from experience, making languages interoperate is a hassle, so "written in different languages" isn't all great. In the particular case of Smalltalk, the operational(?) semantics of which are specified by a virtual machine, one can just compile targeting the virtual machine. c.f compiling Smalltalk and Java to the Self virtual machine, or all the compilers targeting the JVM nowadays.

im_down_w_otp 5y ago

That's basically what we did with https://github.com/auxoncorp/ferros https://github.com/auxoncorp/ferros, Bundle Rust programs together as tasks to run atop the formally verified seL4 microkernel.

erichocean 5y ago

> wondering if at this point we need an OS at all People have written unikernels, but they're not that interesting anymore when you can trivially constrain Linux to a single core, keep the remaining cores for you app (no context switching), and keep all of the Linux-y goodness for admin and debugging (SSH, gdb, etc.). All of the performance, none of the admin and deployment headaches. Basically, unless you truly can't afford that extra core, unikernels are all downside at this point.