11 ms·

In general you'll find that zig is easier to read than Rust (see the first version of this project in Rust [0]) because it's a simpler language. For kernel prog

In general you'll find that zig is easier to read than Rust (see the first version of this project in Rust [0]) because it's a simpler language. For kernel programming this is even more so the case: * zig has native support for arbitrary sized integers. In Rust I used to do bitshifts, Now I just have a packed struct of u3/u5/u7 whatever (see `src/pci/pci.zig`). Of course Rust has a bitflags crate but I didn't find it handy, this is a case where native support vs library support means a world of difference. * zig has native support for freestanding target. I used to have to build with Xargo for cross compiling to a custom target, I also was forced to `[nostd]`, and some features I was using forced me to use nightly Rust. In zig I have a simple `build.zig` and a simple `linker.ld` script, just works. * zig has nicer pointer handling. A lot of kernel programming is stuff that Rust considers unsafe anyway. It's not uncommon to have to write lines like `unsafe { (ptr as const u8) }` to deref a pointer in Rust, which is a pain because this kind of thing happens all of the time. Also you have to play around with mut a lot like this: `unsafe { &mut (address as mut _)`. It just felt wrong a lot of the time, where in zig you have either `const` or `var` and that's the end of it. * zig is really fun to write! this is something that comes up often in the community, after years of C it's just very refreshing. Some things zig is missing: * Package manager, this is coming soon. [1] * Missing documentation for inline assembly (I think this part is going to get overhauled, as Andrew Kelley is writing an assembler in zig atm [2]). I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming. [0] https://github.com/jzck/kernel-rs https://github.com/jzck/kernel-rs [1] https://github.com/ziglang/zig/issues/943 https://github.com/ziglang/zig/issues/943 [2] https://www.youtube.com/watch?v=iWRrkuFCYXQ https://www.youtube.com/watch?v=iWRrkuFCYXQ

guggle 7y ago
Thank you. > I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming. You're right. Still good for libraries though (or apps, but that may be outside of "system").
- joshbaptiste 7y ago
  Nim's GC is optional
  - Touche 7y ago
    How can it be optional if there is lots of code that assumes you are using GC? For example, as far as I can tell the stdlib doesn't do its own allocations. Does this mean you can't use the stdlib with GC disabled? Or am I missing something here?
    - qmmmur 7y ago
      A friend of mine wrote a DSL for audio using Nim with GC off. https://github.com/vitreo12/omni https://github.com/vitreo12/omni I don't know enough to comment but it may be useful to look at things in the wild. This project also heavily relies on calling in C to interface with environments and the SuperCollider scsynth.
    - alehander42 7y ago
      overally if you write an os you need to write your malloc/free .. and then you should be able (i think..) to use many of the gc-s anyway and more of stdlib. But you can also think of Nim as macro-able higher level C and write it like that (but indeed you probably still need this minimal allocation support)
- alehander42 7y ago
  you're right, it is confusing, but it is optional: some toy kernels already work in nim , and with latest work on memory, you should be able to use most of the language for kernel development ! not the perfect language for that yet though, but i hope we should see more nim os examples
  - Touche 7y ago
    Are you saying that the GC is optional? If you don't use it how do you allocate/free memory?
    - guggle 7y ago
      Didn't test this but some clues: https://nim-lang.org/docs/gc.html https://nim-lang.org/docs/gc.html Then I guess you would use new/dealloc
    - guggle 7y ago
      more here: https://nim-lang.org/araq/destructors.html https://nim-lang.org/araq/destructors.html
    - alehander42 7y ago
      overally i think for small os-es you can write easily a micro libc core in C or even in Nim where you define malloc/free etc and just use them directly as in C and zig. otherwise you should be able to use something like destructors eventually
    - mratsim 7y ago
      You call malloc/free, or if working with GPU cudaMalloc/free. You can write your own memory pool or object pools, you can use destructors, and even implement your own reference counting scheme. This is what I use for my own multithreading runtime in Nim and the memory subsystem makes it faster and more robust than any runtime (including OpenMP and Intel TBB) that I've been benchmarking against, see memory subsystem details here: https://github.com/mratsim/weave/tree/master/weave/memory https://github.com/mratsim/weave/tree/master/weave/memory Example of atomic refcounting in this PR here: https://github.com/mratsim/weave/blob/025387510/weave/datatypes/pledges.nim#L178-L212 https://github.com/mratsim/weave/blob/025387510/weave/dataty... Also one important thing, Nim current GC is based on TLSF (http://www.gii.upv.es/tlsf/ http://www.gii.upv.es/tlsf/) which is a memory reclamation scheme for real-time system, it provides provably bounded O(1) allocations. You can tune Nim GC with max-pauses for latency critical applications.
rapsey 7y ago
On the other hand safe Rust code will be memory safe..
- pron 7y ago
  It's Zig's goal to make it easy to write safe code, it's just that its approach to safety is very different from Rust's.
- woodrowbarlow 7y ago
  not if you have to keep using unsafe blocks.
  - cies 7y ago
    As an ancestor in this thread pointed out: it's hard/impossible to go without unsafe in a kernel. Proof (and they have a policy to only use unsafe when not otherwise possible): https://github.com/redox-os/kernel/search?q=unsafe&unscoped_q=unsafe https://github.com/redox-os/kernel/search?q=unsafe&unscoped_...
    - woodrowbarlow 7y ago
      yes, that's the point i'm making. saying "if you write your kernel in rust, it will be memory safe" is a little goofy, because writing kernels in rust requires you to drop into unsafe blocks way too often.
      - bluejekyll 7y ago
        Not as often as you would think, Phill Oppermann has shown this: https://os.phil-opp.com/ https://os.phil-opp.com/ Unsafe is definitely used, but it does not mean what I think is being implied, which is that it makes the software unsafe.
      - rapsey 7y ago
        The ammount of unsafe blocks is pretty small compared to the rest of the code. Which means those parts can be reviewed more extensively. If anything your argument is pretty goofy.
pjmlp 7y ago
> I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming. Have a look at Project Oberon, Singularity, Interlisp-D, Smalltalk, Modula-3 Topaz, Blue Bottle (AOS), CosmosOS, lilith (Crystal variant) given that their source is available.
dom96 7y ago
You can certainly use Nim for kernels. I've created a proof of concept long ago, and things are only improving with ARC fast becoming a great alternative to Nim's GC. https://github.com/dom96/nimkernel https://github.com/dom96/nimkernel
simias 7y ago
It's true that for bare metal programming Rust is quite far from 1.0. You can't even do inline assembly in stable rust... Arbitrary sized integers do sound very convenient, although I guess the counterargument would be that they simply can't map to machine types so you'll have to have magic behind the scenes which may not make complete sense in low level languages. Still, C has bitfields in structs which are sort of the same thing so why not. Pointer handling however I'm not sold on. I'd argue that the very cumbersome pointer handling in Rust is a feature, not a bug. It's explicitly discouraged, and for good reasons. Map your pointer into a clean and safe reference, slice or wrapper object ASAP and leave raw pointers to the messy details of C FFIs and ultra low-level code. If you end up having to mess with raw pointers everywhere in your codebase you're doing it wrong, as far as Rust is concerned. I think this is reasonable even for kernel code. After all typically you wouldn't dereference register pointers directly in C code because of volatility issues and making it obvious that you're accessing hardware and not RAM (readl/writel or similar), so having a safe wrapper adds no overhead in my experience.
- jackhalford 7y ago
  Ok maybe I was user pointers wrong in Rust. And I must confess that I never really understood lifetimes either, however * I don't think that low level code needs to be "messy", and * I don't think that adding an abstraction on top solves anything complexity wise. For memory paging I used to have a whole library in Rust that handled hierarchy of page tables with very abstract pagetable classes and interfaces [0]. Now I just have 80 lines of zig that handles everything [1]. I much prefer the latter. > I think this is reasonable even for kernel code. It's not, for some of the reasons that Linus doesn't want C++ code in the kernel [2] [0] https://github.com/gz/rust-x86/blob/28d5839933973e0b639ef354cb82023031cb9535/src/bits64/paging.rs https://github.com/gz/rust-x86/blob/28d5839933973e0b639ef354... [1] https://git.sr.ht/~jzck/kernel/tree/master/src/arch/x86/paging.zig https://git.sr.ht/~jzck/kernel/tree/master/src/arch/x86/pagi... [2] https://yarchive.net/comp/linux/c++.html https://yarchive.net/comp/linux/c++.html
  - CDSlice 7y ago
    The Rust code you wrote seems to be extremely heavy on boilerplate. A couple of macros could have drastically reduced the line count. I don't think what Linus wrote in 2004 on C++ has much relevance for 2020 Rust, especially for a new kernel that doesn't have a ton of contributors. Most of his complaints seem to be on how C++ abstractions can make it hard to review unknown code and how 2004 C++ code bases often had extremely messy code. Rust IMO doesn't suffer from those problems near as much as 2004 C++ did. Also, Google is using Rust for part of their new Fuchsia kernel so at least they think Rust has something good to offer kernel dev.
UncleOxidant 7y ago
> * zig has native support for arbitrary sized integers That's really cool.
eli_gottlieb 7y ago
Can Zig's enums and tagged unions be used to write algebraic data types?
- sanxiyn 7y ago
  Yes it can.
  - eli_gottlieb 7y ago
    Very nice!
celticmusic 7y ago
I looked over zig when it was first announced and really liked the idea, it was just too fresh for me to work with. But I've always had a mental note to go back once it got a bit more stable because I really do like the ideas behind it. It sounds to me like it's definitely getting there and that's awesome.
bogomipz 7y ago
>"zig has native support for arbitrary sized integers." I apologize if this is a naive question. Might you or someone else elaborate on where arbitrary sized integers are used or necessary in kernel programming? Don't they all ultimately need to get padded out for the CPU registers to work with them anyway?
- loeg 7y ago
  Packed binary structures. Common in device register space (the author alludes to PCI driver(s)).
sov 7y ago
Your point about unsafe pointer handling in Rust is specifically what dissuaded us from using it in an upcoming project. It really feels bad prepending all of the code that you actually care about being safe with `unsafe`.
- senderista 7y ago
  `unsafe` before a block simply means "the following code has been manually checked for memory safety, because the compiler is unable to automatically do so". Before a function it means "this can only be called from an unsafe block, because the compiler cannot enforce the preconditions it requires to ensure memory safety". What alternative term would make you feel less bad?

guggle 7y ago

Thank you. > I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming. You're right. Still good for libraries though (or apps, but that may be outside of "system").

joshbaptiste 7y ago
Nim's GC is optional
- Touche 7y ago
  How can it be optional if there is lots of code that assumes you are using GC? For example, as far as I can tell the stdlib doesn't do its own allocations. Does this mean you can't use the stdlib with GC disabled? Or am I missing something here?
  - qmmmur 7y ago
    A friend of mine wrote a DSL for audio using Nim with GC off. https://github.com/vitreo12/omni https://github.com/vitreo12/omni I don't know enough to comment but it may be useful to look at things in the wild. This project also heavily relies on calling in C to interface with environments and the SuperCollider scsynth.
  - alehander42 7y ago
    overally if you write an os you need to write your malloc/free .. and then you should be able (i think..) to use many of the gc-s anyway and more of stdlib. But you can also think of Nim as macro-able higher level C and write it like that (but indeed you probably still need this minimal allocation support)
alehander42 7y ago
you're right, it is confusing, but it is optional: some toy kernels already work in nim , and with latest work on memory, you should be able to use most of the language for kernel development ! not the perfect language for that yet though, but i hope we should see more nim os examples
- Touche 7y ago
  Are you saying that the GC is optional? If you don't use it how do you allocate/free memory?
  - guggle 7y ago
    Didn't test this but some clues: https://nim-lang.org/docs/gc.html https://nim-lang.org/docs/gc.html Then I guess you would use new/dealloc
  - guggle 7y ago
    more here: https://nim-lang.org/araq/destructors.html https://nim-lang.org/araq/destructors.html
  - alehander42 7y ago
    overally i think for small os-es you can write easily a micro libc core in C or even in Nim where you define malloc/free etc and just use them directly as in C and zig. otherwise you should be able to use something like destructors eventually
  - mratsim 7y ago
    You call malloc/free, or if working with GPU cudaMalloc/free. You can write your own memory pool or object pools, you can use destructors, and even implement your own reference counting scheme. This is what I use for my own multithreading runtime in Nim and the memory subsystem makes it faster and more robust than any runtime (including OpenMP and Intel TBB) that I've been benchmarking against, see memory subsystem details here: https://github.com/mratsim/weave/tree/master/weave/memory https://github.com/mratsim/weave/tree/master/weave/memory Example of atomic refcounting in this PR here: https://github.com/mratsim/weave/blob/025387510/weave/datatypes/pledges.nim#L178-L212 https://github.com/mratsim/weave/blob/025387510/weave/dataty... Also one important thing, Nim current GC is based on TLSF (http://www.gii.upv.es/tlsf/ http://www.gii.upv.es/tlsf/) which is a memory reclamation scheme for real-time system, it provides provably bounded O(1) allocations. You can tune Nim GC with max-pauses for latency critical applications.

rapsey 7y ago

On the other hand safe Rust code will be memory safe..

pron 7y ago
It's Zig's goal to make it easy to write safe code, it's just that its approach to safety is very different from Rust's.
woodrowbarlow 7y ago
not if you have to keep using unsafe blocks.
- cies 7y ago
  As an ancestor in this thread pointed out: it's hard/impossible to go without unsafe in a kernel. Proof (and they have a policy to only use unsafe when not otherwise possible): https://github.com/redox-os/kernel/search?q=unsafe&unscoped_q=unsafe https://github.com/redox-os/kernel/search?q=unsafe&unscoped_...
  - woodrowbarlow 7y ago
    yes, that's the point i'm making. saying "if you write your kernel in rust, it will be memory safe" is a little goofy, because writing kernels in rust requires you to drop into unsafe blocks way too often.
    - bluejekyll 7y ago
      Not as often as you would think, Phill Oppermann has shown this: https://os.phil-opp.com/ https://os.phil-opp.com/ Unsafe is definitely used, but it does not mean what I think is being implied, which is that it makes the software unsafe.
    - rapsey 7y ago
      The ammount of unsafe blocks is pretty small compared to the rest of the code. Which means those parts can be reviewed more extensively. If anything your argument is pretty goofy.

pjmlp 7y ago

> I don't know Nim, but I believe it has a garbage collector so it could be tricky to use for kernel programming. Have a look at Project Oberon, Singularity, Interlisp-D, Smalltalk, Modula-3 Topaz, Blue Bottle (AOS), CosmosOS, lilith (Crystal variant) given that their source is available.

dom96 7y ago

You can certainly use Nim for kernels. I've created a proof of concept long ago, and things are only improving with ARC fast becoming a great alternative to Nim's GC. https://github.com/dom96/nimkernel https://github.com/dom96/nimkernel

simias 7y ago

It's true that for bare metal programming Rust is quite far from 1.0. You can't even do inline assembly in stable rust... Arbitrary sized integers do sound very convenient, although I guess the counterargument would be that they simply can't map to machine types so you'll have to have magic behind the scenes which may not make complete sense in low level languages. Still, C has bitfields in structs which are sort of the same thing so why not. Pointer handling however I'm not sold on. I'd argue that the very cumbersome pointer handling in Rust is a feature, not a bug. It's explicitly discouraged, and for good reasons. Map your pointer into a clean and safe reference, slice or wrapper object ASAP and leave raw pointers to the messy details of C FFIs and ultra low-level code. If you end up having to mess with raw pointers everywhere in your codebase you're doing it wrong, as far as Rust is concerned. I think this is reasonable even for kernel code. After all typically you wouldn't dereference register pointers directly in C code because of volatility issues and making it obvious that you're accessing hardware and not RAM (readl/writel or similar), so having a safe wrapper adds no overhead in my experience.

jackhalford 7y ago
Ok maybe I was user pointers wrong in Rust. And I must confess that I never really understood lifetimes either, however * I don't think that low level code needs to be "messy", and * I don't think that adding an abstraction on top solves anything complexity wise. For memory paging I used to have a whole library in Rust that handled hierarchy of page tables with very abstract pagetable classes and interfaces [0]. Now I just have 80 lines of zig that handles everything [1]. I much prefer the latter. > I think this is reasonable even for kernel code. It's not, for some of the reasons that Linus doesn't want C++ code in the kernel [2] [0] https://github.com/gz/rust-x86/blob/28d5839933973e0b639ef354cb82023031cb9535/src/bits64/paging.rs https://github.com/gz/rust-x86/blob/28d5839933973e0b639ef354... [1] https://git.sr.ht/~jzck/kernel/tree/master/src/arch/x86/paging.zig https://git.sr.ht/~jzck/kernel/tree/master/src/arch/x86/pagi... [2] https://yarchive.net/comp/linux/c++.html https://yarchive.net/comp/linux/c++.html
- CDSlice 7y ago
  The Rust code you wrote seems to be extremely heavy on boilerplate. A couple of macros could have drastically reduced the line count. I don't think what Linus wrote in 2004 on C++ has much relevance for 2020 Rust, especially for a new kernel that doesn't have a ton of contributors. Most of his complaints seem to be on how C++ abstractions can make it hard to review unknown code and how 2004 C++ code bases often had extremely messy code. Rust IMO doesn't suffer from those problems near as much as 2004 C++ did. Also, Google is using Rust for part of their new Fuchsia kernel so at least they think Rust has something good to offer kernel dev.

UncleOxidant 7y ago

> * zig has native support for arbitrary sized integers That's really cool.

eli_gottlieb 7y ago

Can Zig's enums and tagged unions be used to write algebraic data types?

sanxiyn 7y ago
Yes it can.
- eli_gottlieb 7y ago
  Very nice!

celticmusic 7y ago

I looked over zig when it was first announced and really liked the idea, it was just too fresh for me to work with. But I've always had a mental note to go back once it got a bit more stable because I really do like the ideas behind it. It sounds to me like it's definitely getting there and that's awesome.

bogomipz 7y ago

>"zig has native support for arbitrary sized integers." I apologize if this is a naive question. Might you or someone else elaborate on where arbitrary sized integers are used or necessary in kernel programming? Don't they all ultimately need to get padded out for the CPU registers to work with them anyway?

loeg 7y ago
Packed binary structures. Common in device register space (the author alludes to PCI driver(s)).

sov 7y ago

Your point about unsafe pointer handling in Rust is specifically what dissuaded us from using it in an upcoming project. It really feels bad prepending all of the code that you actually care about being safe with `unsafe`.

senderista 7y ago
`unsafe` before a block simply means "the following code has been manually checked for memory safety, because the compiler is unable to automatically do so". Before a function it means "this can only be called from an unsafe block, because the compiler cannot enforce the preconditions it requires to ensure memory safety". What alternative term would make you feel less bad?