12 ms·

Biscuit: A monolithic, POSIX-subset operating system kernel in Go

252 points by appwiz 6y ago

azakai 6y ago
Oh nice, the results are pretty good! > On a set of kernel-intensive benchmarks (including NGINX and Redis) the fraction of kernel CPU time Biscuit spends on HLL features (primarily garbage collection and thread stack expansion checks) ranges up to 13%. The longest single GC-related pause suffered by NGINX was 115 microseconds; the longest observed sum of GC delays to a complete NGINX client request was 600 microseconds. In experiments comparing nearly identical system call, page fault, and context switch code paths written in Go and C, the Go version was 5% to 15% slower. 10% slowdown in return for memory safety could be a worthwhile tradeoff in some cases. And GC pauses were almost not an issue (less than 1ms in the worst measured).
- throwaway894345 6y ago
  This is pretty remarkable and surprising that it's only 5-15% slower. The Go compiler isn't very aggressive about optimizing even ignoring GC overhead concerns. I'm also curious to understand better how they bootstrapped the dynamic memory and scheduling facilities since Go obviously relies on an underlying operating system for these things.
  - jolux 6y ago
    I believe the Midori project at Microsoft found similar in trying to write an OS in C#.
    - deleted 6y ago
      [deleted]
    - jen20 6y ago
      Several posts from Joe Duffy, who led that project, are at [1]. [1]: http://joeduffyblog.com/2015/11/03/blogging-about-midori/ http://joeduffyblog.com/2015/11/03/blogging-about-midori/
  - HonshinM 6y ago
    iirc they modified Go runtime
    - pjmlp 6y ago
      Obviously, usually a language runtime expects an underlying OS. When one isn't available, the runtime plays the role of an OS, and much be adapted as such.
  - londons_explore 6y ago
    It's because typically for most applications the OS isn't very much overhead. If the kernel was only using 1% of the CPU time before, and you rewrite it in QBASIC for a 5x slowdown, your application only sees a ~4% slowdown...
    - sitkack 6y ago
      But didn't you just explain why our kernels should be in QBASIC? Or any other safe language? I feel like this is the next version of Christmas not being true. That unsafe languages were a dumb scam.
      - jolux 6y ago
        Most of it comes from a few bad decisions in C in particular that we ended up with stuck for a very long time (nulls, badly designed array type) OSes were written to run on very different hardware from what we use now and over time the hardware was adapted to running specifically C code well. High-level languages like we expect these days hardly existed at all before the mid 90s or so, and they have been gaining in dominance ever since.
        pjmlp 6y ago
        You will notice that Go shares some authors with C. They did the same with C back in the day, versus what was being done in systems programming since the 60's. The language C builds on, B, is a spin off of BCPL, a language designed to Bootstrap CPL. Thanks to UNIX's success we ended up with a language whose original purpose was to bootstrap compilers, not to be a full stack programming language. Here, systems programming in 1961, 10 years before C was born, still being sold by Unisys. https://en.wikipedia.org/wiki/Burroughs_large_systems https://en.wikipedia.org/wiki/Burroughs_large_systems Or at US military, https://en.wikipedia.org/wiki/JOVIAL https://en.wikipedia.org/wiki/JOVIAL Or during the 70's, https://en.wikipedia.org/wiki/PL/8 https://en.wikipedia.org/wiki/PL/8 https://www.computerhistory.org/revolution/input-output/14/347 https://www.computerhistory.org/revolution/input-output/14/3... https://en.wikipedia.org/wiki/Interlisp https://en.wikipedia.org/wiki/Interlisp https://en.wikipedia.org/wiki/BLISS https://en.wikipedia.org/wiki/BLISS https://en.wikipedia.org/wiki/Modula-2 https://en.wikipedia.org/wiki/Modula-2 https://en.wikipedia.org/wiki/Modula-2%2B https://en.wikipedia.org/wiki/Modula-2%2B Remember that project Bell Labs stepped away from, which gave C's authors plenty of free time? https://en.wikipedia.org/wiki/PL/I https://en.wikipedia.org/wiki/PL/I https://multicians.org/myths.html https://multicians.org/myths.html "Thirty Years Later: Lessons from the Multics Security Evaluation" https://hack.org/mc/texts/classic-multics.pdf https://hack.org/mc/texts/classic-multics.pdf
- kevin_thibedeau 6y ago
  > 10% slowdown in return for memory safety could be a worthwhile tradeoff in some cases. And GC pauses were almost not an issue (less than 1ms in the worst measured). Implementing Nim's bounded time GC could cut that overhead down and defer work for idle periods. That should be first priority when implementing an OS in managed code.
- darksaints 6y ago
  Seeing these results touted as being pretty good makes me wonder why we haven't fully adopted microkernels yet. Their singular disadvantage, which is the need for two IPC calls vs one for a monolithic kernel, is completely irrelevant in comparison to these numbers. The average IPC cost of an L4 IPC call is about 500 cycles, which is about half a microsecond on a 1ghz processor. Cache locality is a big deal, and L4, which can fit entirely within a core's l2 cache, is king when it comes to cache locality. And you don't have to worry about memory safety either. Memory safety has been formally proven as part of the SeL4 spec.
  - pa7ch 6y ago
    I've also wondered why we haven't seen more industrial effort around using sel4. Seems like it'd be perfect for mission critical firmware in cars and such too. For replacing linux, I guess there is still a lot of effort that would be required, but it seems worth it from my admittedly surface viewpoint.
  - Matthias247 6y ago
    > Seeing these results touted as being pretty good makes me wonder why we haven't fully adopted microkernels yet. Their singular disadvantage, which is the need for two IPC calls vs one for a monolithic kernel, is completely irrelevant in comparison to these numbers. And that's even the worst case of synchronous/blocking calls. With different API design - e.g. asynchronous interactions as done with io_uring or IOCP - the latency becomes less important as long as the overall efficiency and thereby throughput is high.
  - vlovich123 6y ago
    Depends on who "we" is. Apple's OS is largely a microkernel design & that ships on billions of devices. Build services for that OS at Apple from a safety/security/performance/battery life perspective is a dream compared to other OSes. Granted outside of that it's a bit rarer.
    - ksec 6y ago
      >Apple's OS is largely a microkernel design They are hybrid design, or some would even refer them to monolithic. Definitely not microkernel, although they are moving in that direction with DriverKit and other kit initiative.
- soamv 6y ago
  Yes, the throughput results are pretty great, as is this work in general. It's worth noting the memory usage, though -- "Go’s garbage collector needs a factor of 2 to 3 of heap headroom to run efficiently (see section 8.6)".
  - sitkack 6y ago
    Amortize it over the life the machine. Go is no gold standard, but what does it mean in terms of total energy cost? Manufacturing cost? A twice larger cache and half the frequency ram bus? I dunno what the answer is, but I think being able to run a safe language for the OS gets us to an entirely different regime.
    - bitwize 6y ago
      Then use Rust.
      - pjmlp 6y ago
        Then improve compile times and borrow checker usability.
        creata 6y ago
        If you don't mind, what are your grievances with Rust's borrow checker, and do you have any ideas on how it could be improved, or any alternative implementations of the idea of "ownership" and "borrowing" that do some things better?
        pjmlp 6y ago
        Here is one example, https://github.com/rust-lang/rust/issues/63818 https://github.com/rust-lang/rust/issues/63818 Alternative ideas, here are four, combining the productivity of automatic memory management with ownership, https://swift.org/blog/swift-5-exclusivity/ https://swift.org/blog/swift-5-exclusivity/ https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md https://github.com/apple/swift/blob/master/docs/OwnershipMan... https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/ https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in... "Linear Haskell", https://arxiv.org/abs/1710.09756 https://arxiv.org/abs/1710.09756 In the context of Rust, maybe showing lifetimes graphically in an IDE would help productivity versus languages that offer automatic memory management alongside ownership.
- sitkack 6y ago
  > The longest single GC-related pause suffered by NGINX was 115 microseconds; Wow! It is funny that we have these capabilities within arms (fingers) reach but it takes someone daring enough to put symbols in a certain order to show that a different world is possible.
  - pjmlp 6y ago
    It is impossible to fight cargo cult with facts, only a generation change helps. Joe Duffy mentioned in one of his talks (Rust Keynote) that even with Midori running in front of them (it even powered part of Bing for a while), there were people on the Windows team quite sceptical that it was possible at all. Anti-GC cargo cult goes a long way.
- otabdeveloper4 6y ago
  We've been discussing the "10% slowdown" for two decades now, since at least the time of Java's first release. Microsoft even tried to rewrite Windows in C# on the premise. Meanwhile in 2020 Java programs still suck ass to use. (Sorry for the crude language, but it's true.) Microsoft memoryholed the whole C# thing and went back to promoting C++.
  - bitwize 6y ago
    Yes, because GC is a woefully inefficient memory management strategy. Rust is not just the future of systems programming -- it's the future of all programming. You may not program directly in Rust the language, but borrow-checked static memory management is coming to an application language near you.
    - pjmlp 6y ago
      What is inefficient are programmers that don't learn to use the features for value types and deterministic memory management that many GC languages offer them.
    - pulse7 6y ago
      It is faster to write an app with GC than to fight the borrow checker. For OS kernel development Rust may be worth the fight, but for userland a language with GC is better in most cases (much easier to learn, use and maintain for relatively small speed and memory penalty).
  - ogogmad 6y ago
    > Meanwhile in 2020 Java programs still suck ass to use Isn't that due to memory usage, rather than speed?
    - otabdeveloper4 6y ago
      I agree with the poster below. GC's problem isn't speed, per se; the problem is that GC only works effectively if you assume infinite memory and cache. Once you start approaching memory limits you start feeling pain.
- bitwize 6y ago
  At a cost of requiring two to three times as much RAM as a non-GC solution to get that performance, as noted elsewhere in this thread. Indeed, it's been known for years[1] that to have comparable performance to an equivalent program without GC, a GC-enabled program needs several times as much memory available. Yeah, no thank you. The OS community should stick with Rust. [1] Matthew Hertz, Emery D. Berger. Quantifying the performance of garbage collection. ACM SIGPLAN Notices, volume 40, issue 10, October 2005. https://people.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf https://people.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf
  - dgellow 6y ago
    The OS community is almost only C and C++. If by community you mean the people working on operating systems. Or you're only considering a small subset to make your point?
  - pjmlp 6y ago
    The OS community seems to use other tools. > Swift is intended as a replacement for C-based languages (C, C++, and Objective-C). -- https://swift.org/about/ https://swift.org/about/ Reference Counting, chapter 5 from The Garbage Collection Handbook.
  - higerordermap 6y ago
    Yeah once you have a old paper that compares a not-particularly-efficient JVM of that time with manual memory management and concludes GC needs 3x memory, it is even easier to spread GC misinformation.
- rowanG077 6y ago
  How about you choose rust. You get 0% slowdown while also getting a more safe language in general.
  - higerordermap 6y ago
    Also never finish implementing, just blogging about it and mentioning rust everywhere. Plaudits to all involved.
02020202 6y ago
they say its go but the soruce code is full of C...
- tomku 6y ago
  From the README in the repo: "This repo is a fork of the Go repo (https://github.com/golang/go https://github.com/golang/go). Nearly all of Biscuit's code is in biscuit/." From one of the linked papers: "Biscuit has nearly 28 thousand lines of Go, 1546 lines of assembler, and no C." There is C code in biscuit/user/c, but it appears to be userland test programs, not kernel code.
  - stevefan1999 6y ago
    part of the bootloader seems to be written in C
als0 6y ago
[deleted]
- deaddodo 6y ago
  There never was a battle. Rust is the up and coming Systems Language (classic definition) and Go is the up and coming Systems Language (modern, colloquial definition). They’re in completely different spheres, with regards to practical production usage.
  - eloff 6y ago
    Yes, having used both I would say there is only a small overlap between them. Go generally is nicer when you can afford a GC and a runtime. By nicer, I mean you'll get things done faster. Rust generally is nicer when you can't, or you need better C interop, more low-level control, or you need a stronger type system for some reason. It's not as productive as Go, or at least I'm far from reaching that point after months of full time rust development, and I'm a pretty experienced polyglot developer.
    - anw 6y ago
      Just wondering, what parts of Rust do you feel take longer to code out vs doing so in Go or its contemporaries?
      - eloff 6y ago
        Most things take longer. Dealing with the more expressive type system, additional errors and warnings, figuring out generic functions, types and type constraints. Having the additional concept of ownership to think about, design around, and run smack into face-first. GC is a joy by comparison, it requires zero effort. Figuring out which abstraction to use, in Go there is typically only one obvious way to do it. Dealing with less mature libraries. In general there is just way more cognitive overhead working in Rust, much more to think about, more options to choose from, and more constraints. Some of this should pay itself back by avoiding certain classes of bugs - but I find on a solo project (I've used Go in teams, but not Rust) I get very little benefit here because all the code is written by me, and I'm experienced enough not to make a lot of the mistakes Rust can protect me from, like race conditions or nil pointers/interfaces most of the time. On a larger team you'd get more benefits here, especially if working with more junior developers. But I don't think you ever get to the same productivity you get with Go. So my advice is if you can afford a GC, and you don't need too much low-level C interop or performance optimiations, choose Go. I do want to add, that from a software craft perspective, I find Rust more elegant to read and write, but only really when I'm not pulling my hair out because of the borrow checker, or hairy details around closures, or whatever else. I do take a certain satisfaction in that elegance sometimes, but overall I'd rather get stuff done.
pcwalton 6y ago
Note that this isn't unmodified Go; it's Go with some features for memory management, in particular having to annotate loops with the number of trips (see section 6.3).
- deaddodo 6y ago
  It would have to be, unless you wanted to write a thin hypervisor in freestanding C, C++ or Rust to handle the syscalls/memory management required for the GC.
  - pjmlp 6y ago
    That is perfectly doable in Go. Just like C, C++ or Rust you would need a thin Assembly help and that is about it. https://people.inf.ethz.ch/wirth/ProjectOberon/Sources/Kernel.Mod.txt https://people.inf.ethz.ch/wirth/ProjectOberon/Sources/Kerne...
    - deaddodo 6y ago
      Freestanding C requires nothing more than the bootloader be written in assembler and that's an architectural dependency; most I/O mapped, embedded devices simply require the binary be loaded in a specific location (usually an offset on ROM). CRT0 itself can also be bootstrapped in C. That's quite different than a low-level kernel bootstrap, which is what you linked. Regardless of whether you do it in Assembler or the aforementioned languages, you must build a hoist outside of Golang.
      - pjmlp 6y ago
        Try to implement malloc/free in freestanding C, or soft floating point for that matter. Inline Assembly doesn't count as C, isn't part of ISO C and any language can have such extensions, including Go.
        deaddodo 6y ago
        Why would I need to, when it’s already been done? https://github.com/blanham/liballoc/ https://github.com/blanham/liballoc/ It’s not even uncommon, as liballoc is one of the top recommended allocators for hobby OSes. I never mentioned inline assembler. Whilst any decent OS will most definitely fall back on assembler, and I never claimed otherwise, it’s not a requirement. It, or another of the aforementioned, would be within the realm of standard Golang. You’re jumping through hoops to equate a language out of it’s intended use cases to ones completely within their’s. It’s not a knock on Go (or other GC’d languages), it’s just a reality of the compromise on specific features. You also couldn’t write an OS in a VM’d language without having a VM hypervisor; it doesn’t mean they’re bad languages.
robot 6y ago
This is interesting in that if you can use channels as described by the CSP book[1] you could build a kernel that is guaranteed to be free of concurrency bugs. This would be important because even if you have proven the functional correctness of a kernel, that typically excludes the concurrency aspect. [1] (https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf)
- arianvanp 6y ago
  But go doesn't provide channels in the CSP way nor does it do any model checking on it right? Like one thing it already gets wrong is that you can send mutable pointers around without clear ownership.
  - sythe2o0 6y ago
    If it's 'wrong' to be able to pass around mutable pointers, is the only language that is 'right' Rust? (and some Lisps maybe?)
    - remexre 6y ago
      I think you can do "real" ownership in ATS; or check ownership with a static analysis tool in many languages, including C; and you should be able make a hacky version as a library that dies at runtime in any language with parametric polymorphism and modules. "Modern C++" too, ish. Which Lisps are you thinking of? CL and Scheme both allow having multiple copies of mutable objects.
    - arianvanp 6y ago
      No. For example Erlang only allows you to send immutable values around. For very good reason.
- pcwalton 6y ago
  If "concurrency bugs" includes deadlocks, no, such a kernel would not be free of concurrency bugs. Any blocking receive operation on a channel can create deadlocks.
Ericson2314 6y ago
Reminds me of when I was excited about https://web.archive.org/web/20120104065532/http://web.cecs.pdx.edu/~kennyg/house/ https://web.archive.org/web/20120104065532/http://web.cecs.p... for a high level language kernel.
_alex_ 6y ago
[rust intensifies]
mlindner 6y ago
Go? Garbage collector pauses in an OS??
- zamadatix 6y ago
  From the abstract: The longest single GC-related pause suffered by NGINX was 115 microseconds; the longest observed sum of GC delays to a complete NGINX client request was 600 microsec- onds.
  - worik 6y ago
    Oh that's alright then
    - zamadatix 6y ago
      Depends on the use case, I just wanted to point out TFA (TFP?) considers this point a central note and goes into great detail about the question.
qlk1123 6y ago
But the development seems stopped. The latest commit was on Jun 8, 2019. At the time this project showed up, I gave it a try on QEMU. It was cool and I was like "I must join this!" As the result, except for one minor fix, I was not able to contribute more because the development environment was not comfortable. The patched go tree was not easy to follow. It also looks impossible for others to just rebase to later go. My take on this is that, although go provides many OS-like features, you still have to draw a clear line between the OS and the language it uses if you want it to be maintainable and evolvable. I am still obsessed to the idea of making an OS in go and somehow trying doing one in a very slow pace.
- remexre 6y ago
  > you still have to draw a clear line between the OS and the language it uses I'm hoping to avoid this in one of my own OS projects using Forth, actually; I think using a language with most features implemented outside of the core language implementation, but rather in libraries, helps a lot here. For example, exceptions are implemented in the OS code rather than by compiler magic (though there are some built-ins the exception system needs to call that are only really relevant if you're building an exception system). The actual Forth implementation is under 2000 lines of aarch64 assembly, after removing comments and whitespace.
  - qlk1123 6y ago
    English is not native to me so I'm not sure it is me that misunderstanding you or the reverse. By "no clear line", I mean something like building the OS on top of some of the language modules. Just like what biscuit did, the runtime and some other packages were integrated as a part of the OS. Such an approach brings inevitable impact to the project, because there will be no longer any easy way to keep up with the new features/progress in the language. On the other hand, you stated > I think using a language with most features implemented outside of the core language implementation, but rather in libraries, helps a lot here. I don't see how your experience contradicts with my previous statement. Would you clarify more? Also, I don't know Forth before. Thanks for enlarging my view.
siltpotato 6y ago
> Biscuit identifies bad citizens and kills them Yay.
tptacek 6y ago
See also, for something surprisingly and alarmingly close to this, gvisor: https://github.com/google/gvisor https://github.com/google/gvisor
- kitd 6y ago
  That looks more like a container runtime than an os kernel.
  - tptacek 6y ago
    Yes, that's what you'd think from the description. Take a closer look.
- deleted 6y ago
  [deleted]
- yencabulator 6y ago
  As far as I understand it, the biggest difference is that biscuit restricts itself to working with a statically allocated amount of memory, reserved for the kernel, and then manages GC etc to not run out of that. That's the greatest contribution of the paper, IMHO. gVisor, being a regular userspace program, has no such worries.
Ericson2314 6y ago
The best thing to use Rust for is writing garbage collectors :D
- Snelius 6y ago
  Oh lul :) thx
pjmlp 6y ago
Not to steal the thread, but for the Go sceptics in system programming, F-Secure decided also to prove them wrong and is shipping bare metal Go for their security solutions. https://labs.f-secure.com/blog/tamago/ https://labs.f-secure.com/blog/tamago/ https://www.f-secure.com/en/consulting/foundry https://www.f-secure.com/en/consulting/foundry
hendry 6y ago
Why don't they distribute an ISO to boot from Virtualbox?
haberman 6y ago
Note that in addition to the CPU overheads, the paper notes that Go's heap requires a factor of 2 to 3 of headroom to run efficiently: > A potential problem with garbage collection is that it > consumes a fraction of CPU time proportional to the > “headroom ratio” between the amount of live data and > the amount of RAM allocated to the heap. This section > explores the effect of headroom on collection cost. > > [...] > > In summary, while the benchmarks in §8.4 / Figure 7 > incur modest collection costs, a kernel heap with millions of live > objects but limited heap RAM might spend > a significant fraction of its time collecting. We expect > that decisions about how much RAM to buy for busy > machines would include a small multiple (2 or 3) of the > expected peak kernel heap live data size. > > [...] > > If CPU performance is paramount, > then C is the right answer, since it is faster (§8.4, §8.5). > If efficient memory use is vital, then C is also the right > answer: Go’s garbage collector needs a factor of 2 to 3 of > heap headroom to run efficiently (see §8.6).
- pjmlp 6y ago
  Matter of implementation. I also read the paper, but I also have read tons of other papers of systems implemented in other languages. While current implementation is quite good, there is lots of room for improvements, including improving the way Go deals with value types. Also with a small improvements to Go (untraced references), or even a //go: annotation, there would be more room for C like data structures, when required to go down that path. Without trying to diminish the work that went into this thesis, speaking from experience, when it was done it was done, performance improvements were most likely not pursued as the point to write an OS in Go as thesis was already proven.
  - throwaway34241 6y ago
    About untraced reference support is there something specific that you’re looking for? Although it doesn’t have a special type I’m pretty sure you can have regular Go object references / interfaces / arrays point to non-GC memory if you want (although you’ll have to cast the result from malloc or something similar). So most library functions should work with unmanaged memory OK by default (if they’re using standard slice/array types) which is not too bad as far as GC languages go.

252 points by appwiz 6y ago

Oh nice, the results are pretty good! > On a set of kernel-intensive benchmarks (including NGINX and Redis) the fraction of kernel CPU time Biscuit spends on HLL features (primarily garbage collection and thread stack expansion checks) ranges up to 13%. The longest single GC-related pause suffered by NGINX was 115 microseconds; the longest observed sum of GC delays to a complete NGINX client request was 600 microseconds. In experiments comparing nearly identical system call, page fault, and context switch code paths written in Go and C, the Go version was 5% to 15% slower. 10% slowdown in return for memory safety could be a worthwhile tradeoff in some cases. And GC pauses were almost not an issue (less than 1ms in the worst measured).

throwaway894345 6y ago
This is pretty remarkable and surprising that it's only 5-15% slower. The Go compiler isn't very aggressive about optimizing even ignoring GC overhead concerns. I'm also curious to understand better how they bootstrapped the dynamic memory and scheduling facilities since Go obviously relies on an underlying operating system for these things.
- jolux 6y ago
  I believe the Midori project at Microsoft found similar in trying to write an OS in C#.
  - deleted 6y ago
    [deleted]
  - jen20 6y ago
    Several posts from Joe Duffy, who led that project, are at [1]. [1]: http://joeduffyblog.com/2015/11/03/blogging-about-midori/ http://joeduffyblog.com/2015/11/03/blogging-about-midori/
- HonshinM 6y ago
  iirc they modified Go runtime
  - pjmlp 6y ago
    Obviously, usually a language runtime expects an underlying OS. When one isn't available, the runtime plays the role of an OS, and much be adapted as such.
- londons_explore 6y ago
  It's because typically for most applications the OS isn't very much overhead. If the kernel was only using 1% of the CPU time before, and you rewrite it in QBASIC for a 5x slowdown, your application only sees a ~4% slowdown...
  - sitkack 6y ago
    But didn't you just explain why our kernels should be in QBASIC? Or any other safe language? I feel like this is the next version of Christmas not being true. That unsafe languages were a dumb scam.
    - jolux 6y ago
      Most of it comes from a few bad decisions in C in particular that we ended up with stuck for a very long time (nulls, badly designed array type) OSes were written to run on very different hardware from what we use now and over time the hardware was adapted to running specifically C code well. High-level languages like we expect these days hardly existed at all before the mid 90s or so, and they have been gaining in dominance ever since.
      - pjmlp 6y ago
        You will notice that Go shares some authors with C. They did the same with C back in the day, versus what was being done in systems programming since the 60's. The language C builds on, B, is a spin off of BCPL, a language designed to Bootstrap CPL. Thanks to UNIX's success we ended up with a language whose original purpose was to bootstrap compilers, not to be a full stack programming language. Here, systems programming in 1961, 10 years before C was born, still being sold by Unisys. https://en.wikipedia.org/wiki/Burroughs_large_systems https://en.wikipedia.org/wiki/Burroughs_large_systems Or at US military, https://en.wikipedia.org/wiki/JOVIAL https://en.wikipedia.org/wiki/JOVIAL Or during the 70's, https://en.wikipedia.org/wiki/PL/8 https://en.wikipedia.org/wiki/PL/8 https://www.computerhistory.org/revolution/input-output/14/347 https://www.computerhistory.org/revolution/input-output/14/3... https://en.wikipedia.org/wiki/Interlisp https://en.wikipedia.org/wiki/Interlisp https://en.wikipedia.org/wiki/BLISS https://en.wikipedia.org/wiki/BLISS https://en.wikipedia.org/wiki/Modula-2 https://en.wikipedia.org/wiki/Modula-2 https://en.wikipedia.org/wiki/Modula-2%2B https://en.wikipedia.org/wiki/Modula-2%2B Remember that project Bell Labs stepped away from, which gave C's authors plenty of free time? https://en.wikipedia.org/wiki/PL/I https://en.wikipedia.org/wiki/PL/I https://multicians.org/myths.html https://multicians.org/myths.html "Thirty Years Later: Lessons from the Multics Security Evaluation" https://hack.org/mc/texts/classic-multics.pdf https://hack.org/mc/texts/classic-multics.pdf
kevin_thibedeau 6y ago
> 10% slowdown in return for memory safety could be a worthwhile tradeoff in some cases. And GC pauses were almost not an issue (less than 1ms in the worst measured). Implementing Nim's bounded time GC could cut that overhead down and defer work for idle periods. That should be first priority when implementing an OS in managed code.
darksaints 6y ago
Seeing these results touted as being pretty good makes me wonder why we haven't fully adopted microkernels yet. Their singular disadvantage, which is the need for two IPC calls vs one for a monolithic kernel, is completely irrelevant in comparison to these numbers. The average IPC cost of an L4 IPC call is about 500 cycles, which is about half a microsecond on a 1ghz processor. Cache locality is a big deal, and L4, which can fit entirely within a core's l2 cache, is king when it comes to cache locality. And you don't have to worry about memory safety either. Memory safety has been formally proven as part of the SeL4 spec.
- pa7ch 6y ago
  I've also wondered why we haven't seen more industrial effort around using sel4. Seems like it'd be perfect for mission critical firmware in cars and such too. For replacing linux, I guess there is still a lot of effort that would be required, but it seems worth it from my admittedly surface viewpoint.
- Matthias247 6y ago
  > Seeing these results touted as being pretty good makes me wonder why we haven't fully adopted microkernels yet. Their singular disadvantage, which is the need for two IPC calls vs one for a monolithic kernel, is completely irrelevant in comparison to these numbers. And that's even the worst case of synchronous/blocking calls. With different API design - e.g. asynchronous interactions as done with io_uring or IOCP - the latency becomes less important as long as the overall efficiency and thereby throughput is high.
- vlovich123 6y ago
  Depends on who "we" is. Apple's OS is largely a microkernel design & that ships on billions of devices. Build services for that OS at Apple from a safety/security/performance/battery life perspective is a dream compared to other OSes. Granted outside of that it's a bit rarer.
  - ksec 6y ago
    >Apple's OS is largely a microkernel design They are hybrid design, or some would even refer them to monolithic. Definitely not microkernel, although they are moving in that direction with DriverKit and other kit initiative.
soamv 6y ago
Yes, the throughput results are pretty great, as is this work in general. It's worth noting the memory usage, though -- "Go’s garbage collector needs a factor of 2 to 3 of heap headroom to run efficiently (see section 8.6)".
- sitkack 6y ago
  Amortize it over the life the machine. Go is no gold standard, but what does it mean in terms of total energy cost? Manufacturing cost? A twice larger cache and half the frequency ram bus? I dunno what the answer is, but I think being able to run a safe language for the OS gets us to an entirely different regime.
  - bitwize 6y ago
    Then use Rust.
    - pjmlp 6y ago
      Then improve compile times and borrow checker usability.
      - creata 6y ago
        If you don't mind, what are your grievances with Rust's borrow checker, and do you have any ideas on how it could be improved, or any alternative implementations of the idea of "ownership" and "borrowing" that do some things better?
        pjmlp 6y ago
        Here is one example, https://github.com/rust-lang/rust/issues/63818 https://github.com/rust-lang/rust/issues/63818 Alternative ideas, here are four, combining the productivity of automatic memory management with ownership, https://swift.org/blog/swift-5-exclusivity/ https://swift.org/blog/swift-5-exclusivity/ https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md https://github.com/apple/swift/blob/master/docs/OwnershipMan... https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/ https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in... "Linear Haskell", https://arxiv.org/abs/1710.09756 https://arxiv.org/abs/1710.09756 In the context of Rust, maybe showing lifetimes graphically in an IDE would help productivity versus languages that offer automatic memory management alongside ownership.
sitkack 6y ago
> The longest single GC-related pause suffered by NGINX was 115 microseconds; Wow! It is funny that we have these capabilities within arms (fingers) reach but it takes someone daring enough to put symbols in a certain order to show that a different world is possible.
- pjmlp 6y ago
  It is impossible to fight cargo cult with facts, only a generation change helps. Joe Duffy mentioned in one of his talks (Rust Keynote) that even with Midori running in front of them (it even powered part of Bing for a while), there were people on the Windows team quite sceptical that it was possible at all. Anti-GC cargo cult goes a long way.
otabdeveloper4 6y ago
We've been discussing the "10% slowdown" for two decades now, since at least the time of Java's first release. Microsoft even tried to rewrite Windows in C# on the premise. Meanwhile in 2020 Java programs still suck ass to use. (Sorry for the crude language, but it's true.) Microsoft memoryholed the whole C# thing and went back to promoting C++.
- bitwize 6y ago
  Yes, because GC is a woefully inefficient memory management strategy. Rust is not just the future of systems programming -- it's the future of all programming. You may not program directly in Rust the language, but borrow-checked static memory management is coming to an application language near you.
  - pjmlp 6y ago
    What is inefficient are programmers that don't learn to use the features for value types and deterministic memory management that many GC languages offer them.
  - pulse7 6y ago
    It is faster to write an app with GC than to fight the borrow checker. For OS kernel development Rust may be worth the fight, but for userland a language with GC is better in most cases (much easier to learn, use and maintain for relatively small speed and memory penalty).
- ogogmad 6y ago
  > Meanwhile in 2020 Java programs still suck ass to use Isn't that due to memory usage, rather than speed?
  - otabdeveloper4 6y ago
    I agree with the poster below. GC's problem isn't speed, per se; the problem is that GC only works effectively if you assume infinite memory and cache. Once you start approaching memory limits you start feeling pain.
bitwize 6y ago
At a cost of requiring two to three times as much RAM as a non-GC solution to get that performance, as noted elsewhere in this thread. Indeed, it's been known for years[1] that to have comparable performance to an equivalent program without GC, a GC-enabled program needs several times as much memory available. Yeah, no thank you. The OS community should stick with Rust. [1] Matthew Hertz, Emery D. Berger. Quantifying the performance of garbage collection. ACM SIGPLAN Notices, volume 40, issue 10, October 2005. https://people.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf https://people.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf
- dgellow 6y ago
  The OS community is almost only C and C++. If by community you mean the people working on operating systems. Or you're only considering a small subset to make your point?
- pjmlp 6y ago
  The OS community seems to use other tools. > Swift is intended as a replacement for C-based languages (C, C++, and Objective-C). -- https://swift.org/about/ https://swift.org/about/ Reference Counting, chapter 5 from The Garbage Collection Handbook.
- higerordermap 6y ago
  Yeah once you have a old paper that compares a not-particularly-efficient JVM of that time with manual memory management and concludes GC needs 3x memory, it is even easier to spread GC misinformation.
rowanG077 6y ago
How about you choose rust. You get 0% slowdown while also getting a more safe language in general.
- higerordermap 6y ago
  Also never finish implementing, just blogging about it and mentioning rust everywhere. Plaudits to all involved.

02020202 6y ago

they say its go but the soruce code is full of C...

tomku 6y ago
From the README in the repo: "This repo is a fork of the Go repo (https://github.com/golang/go https://github.com/golang/go). Nearly all of Biscuit's code is in biscuit/." From one of the linked papers: "Biscuit has nearly 28 thousand lines of Go, 1546 lines of assembler, and no C." There is C code in biscuit/user/c, but it appears to be userland test programs, not kernel code.
- stevefan1999 6y ago
  part of the bootloader seems to be written in C

als0 6y ago

[deleted]

deaddodo 6y ago
There never was a battle. Rust is the up and coming Systems Language (classic definition) and Go is the up and coming Systems Language (modern, colloquial definition). They’re in completely different spheres, with regards to practical production usage.
- eloff 6y ago
  Yes, having used both I would say there is only a small overlap between them. Go generally is nicer when you can afford a GC and a runtime. By nicer, I mean you'll get things done faster. Rust generally is nicer when you can't, or you need better C interop, more low-level control, or you need a stronger type system for some reason. It's not as productive as Go, or at least I'm far from reaching that point after months of full time rust development, and I'm a pretty experienced polyglot developer.
  - anw 6y ago
    Just wondering, what parts of Rust do you feel take longer to code out vs doing so in Go or its contemporaries?
    - eloff 6y ago
      Most things take longer. Dealing with the more expressive type system, additional errors and warnings, figuring out generic functions, types and type constraints. Having the additional concept of ownership to think about, design around, and run smack into face-first. GC is a joy by comparison, it requires zero effort. Figuring out which abstraction to use, in Go there is typically only one obvious way to do it. Dealing with less mature libraries. In general there is just way more cognitive overhead working in Rust, much more to think about, more options to choose from, and more constraints. Some of this should pay itself back by avoiding certain classes of bugs - but I find on a solo project (I've used Go in teams, but not Rust) I get very little benefit here because all the code is written by me, and I'm experienced enough not to make a lot of the mistakes Rust can protect me from, like race conditions or nil pointers/interfaces most of the time. On a larger team you'd get more benefits here, especially if working with more junior developers. But I don't think you ever get to the same productivity you get with Go. So my advice is if you can afford a GC, and you don't need too much low-level C interop or performance optimiations, choose Go. I do want to add, that from a software craft perspective, I find Rust more elegant to read and write, but only really when I'm not pulling my hair out because of the borrow checker, or hairy details around closures, or whatever else. I do take a certain satisfaction in that elegance sometimes, but overall I'd rather get stuff done.

pcwalton 6y ago

Note that this isn't unmodified Go; it's Go with some features for memory management, in particular having to annotate loops with the number of trips (see section 6.3).

deaddodo 6y ago
It would have to be, unless you wanted to write a thin hypervisor in freestanding C, C++ or Rust to handle the syscalls/memory management required for the GC.
- pjmlp 6y ago
  That is perfectly doable in Go. Just like C, C++ or Rust you would need a thin Assembly help and that is about it. https://people.inf.ethz.ch/wirth/ProjectOberon/Sources/Kernel.Mod.txt https://people.inf.ethz.ch/wirth/ProjectOberon/Sources/Kerne...
  - deaddodo 6y ago
    Freestanding C requires nothing more than the bootloader be written in assembler and that's an architectural dependency; most I/O mapped, embedded devices simply require the binary be loaded in a specific location (usually an offset on ROM). CRT0 itself can also be bootstrapped in C. That's quite different than a low-level kernel bootstrap, which is what you linked. Regardless of whether you do it in Assembler or the aforementioned languages, you must build a hoist outside of Golang.
    - pjmlp 6y ago
      Try to implement malloc/free in freestanding C, or soft floating point for that matter. Inline Assembly doesn't count as C, isn't part of ISO C and any language can have such extensions, including Go.
      - deaddodo 6y ago
        Why would I need to, when it’s already been done? https://github.com/blanham/liballoc/ https://github.com/blanham/liballoc/ It’s not even uncommon, as liballoc is one of the top recommended allocators for hobby OSes. I never mentioned inline assembler. Whilst any decent OS will most definitely fall back on assembler, and I never claimed otherwise, it’s not a requirement. It, or another of the aforementioned, would be within the realm of standard Golang. You’re jumping through hoops to equate a language out of it’s intended use cases to ones completely within their’s. It’s not a knock on Go (or other GC’d languages), it’s just a reality of the compromise on specific features. You also couldn’t write an OS in a VM’d language without having a VM hypervisor; it doesn’t mean they’re bad languages.

robot 6y ago

This is interesting in that if you can use channels as described by the CSP book[1] you could build a kernel that is guaranteed to be free of concurrency bugs. This would be important because even if you have proven the functional correctness of a kernel, that typically excludes the concurrency aspect. [1] (https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf)

arianvanp 6y ago
But go doesn't provide channels in the CSP way nor does it do any model checking on it right? Like one thing it already gets wrong is that you can send mutable pointers around without clear ownership.
- sythe2o0 6y ago
  If it's 'wrong' to be able to pass around mutable pointers, is the only language that is 'right' Rust? (and some Lisps maybe?)
  - remexre 6y ago
    I think you can do "real" ownership in ATS; or check ownership with a static analysis tool in many languages, including C; and you should be able make a hacky version as a library that dies at runtime in any language with parametric polymorphism and modules. "Modern C++" too, ish. Which Lisps are you thinking of? CL and Scheme both allow having multiple copies of mutable objects.
  - arianvanp 6y ago
    No. For example Erlang only allows you to send immutable values around. For very good reason.
pcwalton 6y ago
If "concurrency bugs" includes deadlocks, no, such a kernel would not be free of concurrency bugs. Any blocking receive operation on a channel can create deadlocks.

Ericson2314 6y ago

Reminds me of when I was excited about https://web.archive.org/web/20120104065532/http://web.cecs.pdx.edu/~kennyg/house/ https://web.archive.org/web/20120104065532/http://web.cecs.p... for a high level language kernel.

_alex_ 6y ago

[rust intensifies]

mlindner 6y ago

Go? Garbage collector pauses in an OS??

zamadatix 6y ago
From the abstract: The longest single GC-related pause suffered by NGINX was 115 microseconds; the longest observed sum of GC delays to a complete NGINX client request was 600 microsec- onds.
- worik 6y ago
  Oh that's alright then
  - zamadatix 6y ago
    Depends on the use case, I just wanted to point out TFA (TFP?) considers this point a central note and goes into great detail about the question.

qlk1123 6y ago

But the development seems stopped. The latest commit was on Jun 8, 2019. At the time this project showed up, I gave it a try on QEMU. It was cool and I was like "I must join this!" As the result, except for one minor fix, I was not able to contribute more because the development environment was not comfortable. The patched go tree was not easy to follow. It also looks impossible for others to just rebase to later go. My take on this is that, although go provides many OS-like features, you still have to draw a clear line between the OS and the language it uses if you want it to be maintainable and evolvable. I am still obsessed to the idea of making an OS in go and somehow trying doing one in a very slow pace.

remexre 6y ago
> you still have to draw a clear line between the OS and the language it uses I'm hoping to avoid this in one of my own OS projects using Forth, actually; I think using a language with most features implemented outside of the core language implementation, but rather in libraries, helps a lot here. For example, exceptions are implemented in the OS code rather than by compiler magic (though there are some built-ins the exception system needs to call that are only really relevant if you're building an exception system). The actual Forth implementation is under 2000 lines of aarch64 assembly, after removing comments and whitespace.
- qlk1123 6y ago
  English is not native to me so I'm not sure it is me that misunderstanding you or the reverse. By "no clear line", I mean something like building the OS on top of some of the language modules. Just like what biscuit did, the runtime and some other packages were integrated as a part of the OS. Such an approach brings inevitable impact to the project, because there will be no longer any easy way to keep up with the new features/progress in the language. On the other hand, you stated > I think using a language with most features implemented outside of the core language implementation, but rather in libraries, helps a lot here. I don't see how your experience contradicts with my previous statement. Would you clarify more? Also, I don't know Forth before. Thanks for enlarging my view.

siltpotato 6y ago

> Biscuit identifies bad citizens and kills them Yay.

tptacek 6y ago

See also, for something surprisingly and alarmingly close to this, gvisor: https://github.com/google/gvisor https://github.com/google/gvisor

kitd 6y ago
That looks more like a container runtime than an os kernel.
- tptacek 6y ago
  Yes, that's what you'd think from the description. Take a closer look.
deleted 6y ago
[deleted]
yencabulator 6y ago
As far as I understand it, the biggest difference is that biscuit restricts itself to working with a statically allocated amount of memory, reserved for the kernel, and then manages GC etc to not run out of that. That's the greatest contribution of the paper, IMHO. gVisor, being a regular userspace program, has no such worries.

Ericson2314 6y ago

The best thing to use Rust for is writing garbage collectors :D

Snelius 6y ago
Oh lul :) thx

pjmlp 6y ago

Not to steal the thread, but for the Go sceptics in system programming, F-Secure decided also to prove them wrong and is shipping bare metal Go for their security solutions. https://labs.f-secure.com/blog/tamago/ https://labs.f-secure.com/blog/tamago/ https://www.f-secure.com/en/consulting/foundry https://www.f-secure.com/en/consulting/foundry

hendry 6y ago

Why don't they distribute an ISO to boot from Virtualbox?

haberman 6y ago

Note that in addition to the CPU overheads, the paper notes that Go's heap requires a factor of 2 to 3 of headroom to run efficiently: > A potential problem with garbage collection is that it > consumes a fraction of CPU time proportional to the > “headroom ratio” between the amount of live data and > the amount of RAM allocated to the heap. This section > explores the effect of headroom on collection cost. > > [...] > > In summary, while the benchmarks in §8.4 / Figure 7 > incur modest collection costs, a kernel heap with millions of live > objects but limited heap RAM might spend > a significant fraction of its time collecting. We expect > that decisions about how much RAM to buy for busy > machines would include a small multiple (2 or 3) of the > expected peak kernel heap live data size. > > [...] > > If CPU performance is paramount, > then C is the right answer, since it is faster (§8.4, §8.5). > If efficient memory use is vital, then C is also the right > answer: Go’s garbage collector needs a factor of 2 to 3 of > heap headroom to run efficiently (see §8.6).

pjmlp 6y ago
Matter of implementation. I also read the paper, but I also have read tons of other papers of systems implemented in other languages. While current implementation is quite good, there is lots of room for improvements, including improving the way Go deals with value types. Also with a small improvements to Go (untraced references), or even a //go: annotation, there would be more room for C like data structures, when required to go down that path. Without trying to diminish the work that went into this thesis, speaking from experience, when it was done it was done, performance improvements were most likely not pursued as the point to write an OS in Go as thesis was already proven.
- throwaway34241 6y ago
  About untraced reference support is there something specific that you’re looking for? Although it doesn’t have a special type I’m pretty sure you can have regular Go object references / interfaces / arrays point to non-GC memory if you want (although you’ll have to cast the result from malloc or something similar). So most library functions should work with unmanaged memory OK by default (if they’re using standard slice/array types) which is not too bad as far as GC languages go.