3 ms·

> The facts that these exist and the programmer has to always be consciously be aware of it This is what separates a systems programming language suitable for

> The facts that these exist and the programmer has to always be consciously be aware of it This is what separates a systems programming language suitable for OS and embedded development from managed languages, which are not. The complexity ultimately stems from unavoidable details of the hardware. Languages which do not offer similar representations will be incapable of making full use of the underlying hardware, including writing certain projects like bootloaders, firmwares, OSes, etc. Rust is pretty close to state-of-the-art in terms of providing reasonable abstractions over the hardware. It sounds to me like you're used to managed languages with a runtime which are great for certain applications, but unable to be specific enough about memory layout, how data is formatted in memory, etc. for some tasks. Those choices were made for you by the language runtime's authors and necessarily limit the language's applicability to some problems. Rust, C++, and other systems programming languages don't have such limitations, but require you to understand more of the complexity of what the system is actually doing. Any language which provides adequate representations for taking full advantage of the hardware is going to be on a similar order of complexity as Rust, C++, or other systems programming languages, because the hardware is complex. Managed languages can be nice for introducing folks to programming, precisely because much of the complexity is hidden in the runtime, but that can be a double edged sword when it comes time to approach a systems level task. Instead of wishing the language didn't offer those representations, it may be more productive to ask why C++ and Rust converged on such similar ones. Exploration of that question will be enlightening.

jacquesm 9mo ago
There should be no such difference. The bigger problem is that OS's enforce this duality when in fact there should only be application level software and an absolutely tiny core to handle IPC and scheduling. This then allows you to enforce the boundaries between various bits far more strictly.
- timschmidt 9mo ago
  It seems like you think that some shadowy cabal somewhere decided to differentiate systems languages from managed languages and keep them divided. That is not the case. The distinction describes how the language has been implemented, which is based on the choices of the language authors alone, and is usually down to practical considerations about how to implement the thing at all. Saying "There should be no such difference." is a bit like saying bicycles should be allowed on the highway and semi trucks should be accepted on walking paths. The difference is inherent in the thing. A result of how they were built. And what they can and can't accomplish as a result.
  - jacquesm 9mo ago
    > It seems like you think that some shadowy cabal somewhere decided to differentiate systems languages from managed languages and keep them divided. That is not the case. You could have made that point without the strawman, and what a ridiculous thing to say anyway. > The distinction describes how the language has been implemented, which is based on the choices of the language authors alone, and is usually down to practical considerations about how to implement the thing at all. That is so obvious I do not understand what point you are trying to make here. > Saying "There should be no such difference." is a bit like saying bicycles should be allowed on the highway and semi trucks should be accepted on walking paths. The difference is inherent in the thing. A result of how they were built. And what they can and can't accomplish as a result. No, the error is yours: you are interpreting my sentence in a way that is blatantly wrong and then argue with the outcome. I'm not saying that there shouldn't be 'trucks or bicycles' in terms of programming languages. What I'm saying is that the boundary between where you use a 'systems programming language' and where you use an 'application programming language' is artificial and that we are using too much of the former in a place where we probably should be using the latter.
    - timschmidt 9mo ago
      > What I'm saying is that the boundary between where you use a 'systems programming language' and where you use an 'application programming language' is artificial Well, everything humans have ever created is artificial, so I'm not sure even how to parse this sentence. There are objectively tasks which can be accomplished with systems languages which cannot be with others. That is an indisputable fact, and not determined by anything but the language specification and implementation. > we are using too much of the former in a place where we probably should be using the latter If I loosen this sentence to mean that we should write more code in languages which have the familiar properties of applications languages - memory and thread safety, low boilerplate, helpful tooling, reduced cognitive load - then I can agree with it. Otherwise it seems an opinion without reason. Where that code needs speed, uninterrupted control of execution, to be embedded, target wasm, or might want to do any of those things in the future, I think Rust is an excellent choice which provides many of those benefits. I'd rather not have runtimes sneaking into low level performance critical areas like kernels and drivers. But some folks are into that sort of thing - Microsoft famously worked on https://en.wikipedia.org/wiki/Singularity_%28operating_system%29 https://en.wikipedia.org/wiki/Singularity_%28operating_syste... in C# but then again C# has very similar memory semantics to Rust. Microkernels seem to work well enough, with several extant examples. Anyway, you do you! I look forward to ogling your OS project from the sidelines.
- vacuity 9mo ago
  The kernel, the memory manager and device drivers, the language runtimes for managed languages; would these not be written in "systems" languages? Essential complexity can only be rearranged, not removed. Nothing changes the fact that every interface between hardware and software needs to be bridged. On a different note: why should the kernel even handle IPC or scheduling? Those take the basic capabilities of context switching, timer management, and memory management. Even "core functionality" can be a context switch (or a few) away; is a syscall not just a message to a system server? Only the most basic form of communication is necessary to delegate arbitrary functionality, so a true microkernel should only introduce that as an abstraction. Everything else either follows from the hardware or is left to the whims of software.
  - jacquesm 9mo ago
    > The kernel, the memory manager and device drivers, the language runtimes for managed languages; would these not be written in "systems" languages? The kernel and memory manager: probably yes, the device drivers: not necessarily, the language runtimes for managed languages: not necessarily. > On a different note: why should the kernel even handle IPC or scheduling? Because any other solution will quickly run into chicken-and-the-egg style problems. > Those take the basic capabilities of context switching, timer management, and memory management. Yes. But only one timer, the rest should be free to use by other applications. > Even "core functionality" can be a context switch (or a few) away; is a syscall not just a message to a system server? No. A syscall is usually defined as a call to a ring one level in from the one where you currently are. But lots of things that are syscalls right now do not necessarily have to be. > Only the most basic form of communication is necessary to delegate arbitrary functionality, so a true microkernel should only introduce that as an abstraction. Indeed. And they do. > Everything else either follows from the hardware or is left to the whims of software. A lot of the stuff that 'follows from the hardware' can be dealt with at the application level.
    - vacuity 9mo ago
      > The kernel and memory manager: probably yes, the device drivers: not necessarily, the language runtimes for managed languages: not necessarily. The parts that touch hardware (or similarly bare kernel interfaces) must be so. Sure, you could split device driver implementations and so on, but somewhere there's a meaningful lower level of software within the system. > Because any other solution will quickly run into chicken-and-the-egg style problems. No. The kernel must provide for context switching. It would be like migrating threads IPC, but one-way. No threads, no scheduler, no dedicated data transfer. In other words, the bare minimum necessary to make a sensible abstraction around switching processes. seL4, according to its developers, is not absolutely a microkernel. I believe the rationale mainly points to the in-kernel scheduler, but seL4's IPC interacts with the scheduler and is noticeably more elaborate than a mere context switch. Even if seL4's IPC is, by most standards, minimal, I do not consider it to be so objectively. I described a meaningfully more minimal alternative. Delegating scheduling to userspace is trivial. If necessary, designate a scheduler to run if no scheduling decision is available. It has been done before, and the only usual objection is performance. > No. A syscall is usually defined as a call to a ring one level in from the one where you currently are. But lots of things that are syscalls right now do not necessarily have to be. Just as hardware interrupts can be abstracted into messages, syscalls can be abstracted into messages. I'm not saying that the hardware implementation directly conforms to the abstraction.
fooker 9mo ago
None of these have anything to do to do with memory layout or how data is formatted in the memory. You are arguing a different point than the one being discussed here. These are fundamentally hacks around the compiler’s inability to understand ownership and lifetimes, at least the way Rust (and C++) are designed. These exist in Rust because otherwise you would have to use unsafe blocks all the time to write any reasonable code.
- timschmidt 9mo ago
  Safe threading seems like fully exploiting hardware features to me. That's the biggest thing smart pointers and Rc, Arc, and Cell types enable. Perhaps I could have been clearer. The comment about memory layout was an additional point that dealing with hardware requires that you format, align, and position information in memory as the hardware expects, which requires either exposing those details, or encoding them in a runtime.
  - fooker 9mo ago
    Safe threading sounds great, as long as the language gets out of my way when I absolutely do not care about safe threading. I write compilers (and have contributed to the Rust compiler!), and there has been approximately zero times in twenty or so years that thread safety has been a concern.
    - timschmidt 9mo ago
      I write a bit of everything, including a little Rust CAD library which is a continuation of work I did in C++ 20 years ago. The C++ version was never multithreaded. The Rust library was fully multithreaded in an afternoon by importing Rayon behind a cargo feature, and using par_iter in place of iter in a few hot loops. That alone made the rewrite worth it.
      - fooker 9mo ago
        Great, how much faster was this multithreaded version for a real use case?

jacquesm 9mo ago

There should be no such difference. The bigger problem is that OS's enforce this duality when in fact there should only be application level software and an absolutely tiny core to handle IPC and scheduling. This then allows you to enforce the boundaries between various bits far more strictly.

timschmidt 9mo ago
It seems like you think that some shadowy cabal somewhere decided to differentiate systems languages from managed languages and keep them divided. That is not the case. The distinction describes how the language has been implemented, which is based on the choices of the language authors alone, and is usually down to practical considerations about how to implement the thing at all. Saying "There should be no such difference." is a bit like saying bicycles should be allowed on the highway and semi trucks should be accepted on walking paths. The difference is inherent in the thing. A result of how they were built. And what they can and can't accomplish as a result.
- jacquesm 9mo ago
  > It seems like you think that some shadowy cabal somewhere decided to differentiate systems languages from managed languages and keep them divided. That is not the case. You could have made that point without the strawman, and what a ridiculous thing to say anyway. > The distinction describes how the language has been implemented, which is based on the choices of the language authors alone, and is usually down to practical considerations about how to implement the thing at all. That is so obvious I do not understand what point you are trying to make here. > Saying "There should be no such difference." is a bit like saying bicycles should be allowed on the highway and semi trucks should be accepted on walking paths. The difference is inherent in the thing. A result of how they were built. And what they can and can't accomplish as a result. No, the error is yours: you are interpreting my sentence in a way that is blatantly wrong and then argue with the outcome. I'm not saying that there shouldn't be 'trucks or bicycles' in terms of programming languages. What I'm saying is that the boundary between where you use a 'systems programming language' and where you use an 'application programming language' is artificial and that we are using too much of the former in a place where we probably should be using the latter.
  - timschmidt 9mo ago
    > What I'm saying is that the boundary between where you use a 'systems programming language' and where you use an 'application programming language' is artificial Well, everything humans have ever created is artificial, so I'm not sure even how to parse this sentence. There are objectively tasks which can be accomplished with systems languages which cannot be with others. That is an indisputable fact, and not determined by anything but the language specification and implementation. > we are using too much of the former in a place where we probably should be using the latter If I loosen this sentence to mean that we should write more code in languages which have the familiar properties of applications languages - memory and thread safety, low boilerplate, helpful tooling, reduced cognitive load - then I can agree with it. Otherwise it seems an opinion without reason. Where that code needs speed, uninterrupted control of execution, to be embedded, target wasm, or might want to do any of those things in the future, I think Rust is an excellent choice which provides many of those benefits. I'd rather not have runtimes sneaking into low level performance critical areas like kernels and drivers. But some folks are into that sort of thing - Microsoft famously worked on https://en.wikipedia.org/wiki/Singularity_%28operating_system%29 https://en.wikipedia.org/wiki/Singularity_%28operating_syste... in C# but then again C# has very similar memory semantics to Rust. Microkernels seem to work well enough, with several extant examples. Anyway, you do you! I look forward to ogling your OS project from the sidelines.
vacuity 9mo ago
The kernel, the memory manager and device drivers, the language runtimes for managed languages; would these not be written in "systems" languages? Essential complexity can only be rearranged, not removed. Nothing changes the fact that every interface between hardware and software needs to be bridged. On a different note: why should the kernel even handle IPC or scheduling? Those take the basic capabilities of context switching, timer management, and memory management. Even "core functionality" can be a context switch (or a few) away; is a syscall not just a message to a system server? Only the most basic form of communication is necessary to delegate arbitrary functionality, so a true microkernel should only introduce that as an abstraction. Everything else either follows from the hardware or is left to the whims of software.
- jacquesm 9mo ago
  > The kernel, the memory manager and device drivers, the language runtimes for managed languages; would these not be written in "systems" languages? The kernel and memory manager: probably yes, the device drivers: not necessarily, the language runtimes for managed languages: not necessarily. > On a different note: why should the kernel even handle IPC or scheduling? Because any other solution will quickly run into chicken-and-the-egg style problems. > Those take the basic capabilities of context switching, timer management, and memory management. Yes. But only one timer, the rest should be free to use by other applications. > Even "core functionality" can be a context switch (or a few) away; is a syscall not just a message to a system server? No. A syscall is usually defined as a call to a ring one level in from the one where you currently are. But lots of things that are syscalls right now do not necessarily have to be. > Only the most basic form of communication is necessary to delegate arbitrary functionality, so a true microkernel should only introduce that as an abstraction. Indeed. And they do. > Everything else either follows from the hardware or is left to the whims of software. A lot of the stuff that 'follows from the hardware' can be dealt with at the application level.
  - vacuity 9mo ago
    > The kernel and memory manager: probably yes, the device drivers: not necessarily, the language runtimes for managed languages: not necessarily. The parts that touch hardware (or similarly bare kernel interfaces) must be so. Sure, you could split device driver implementations and so on, but somewhere there's a meaningful lower level of software within the system. > Because any other solution will quickly run into chicken-and-the-egg style problems. No. The kernel must provide for context switching. It would be like migrating threads IPC, but one-way. No threads, no scheduler, no dedicated data transfer. In other words, the bare minimum necessary to make a sensible abstraction around switching processes. seL4, according to its developers, is not absolutely a microkernel. I believe the rationale mainly points to the in-kernel scheduler, but seL4's IPC interacts with the scheduler and is noticeably more elaborate than a mere context switch. Even if seL4's IPC is, by most standards, minimal, I do not consider it to be so objectively. I described a meaningfully more minimal alternative. Delegating scheduling to userspace is trivial. If necessary, designate a scheduler to run if no scheduling decision is available. It has been done before, and the only usual objection is performance. > No. A syscall is usually defined as a call to a ring one level in from the one where you currently are. But lots of things that are syscalls right now do not necessarily have to be. Just as hardware interrupts can be abstracted into messages, syscalls can be abstracted into messages. I'm not saying that the hardware implementation directly conforms to the abstraction.

fooker 9mo ago

None of these have anything to do to do with memory layout or how data is formatted in the memory. You are arguing a different point than the one being discussed here. These are fundamentally hacks around the compiler’s inability to understand ownership and lifetimes, at least the way Rust (and C++) are designed. These exist in Rust because otherwise you would have to use unsafe blocks all the time to write any reasonable code.

timschmidt 9mo ago
Safe threading seems like fully exploiting hardware features to me. That's the biggest thing smart pointers and Rc, Arc, and Cell types enable. Perhaps I could have been clearer. The comment about memory layout was an additional point that dealing with hardware requires that you format, align, and position information in memory as the hardware expects, which requires either exposing those details, or encoding them in a runtime.
- fooker 9mo ago
  Safe threading sounds great, as long as the language gets out of my way when I absolutely do not care about safe threading. I write compilers (and have contributed to the Rust compiler!), and there has been approximately zero times in twenty or so years that thread safety has been a concern.
  - timschmidt 9mo ago
    I write a bit of everything, including a little Rust CAD library which is a continuation of work I did in C++ 20 years ago. The C++ version was never multithreaded. The Rust library was fully multithreaded in an afternoon by importing Rayon behind a cargo feature, and using par_iter in place of iter in a few hot loops. That alone made the rewrite worth it.
    - fooker 9mo ago
      Great, how much faster was this multithreaded version for a real use case?