4 ms·

Oh yes. We're aware, we've had and continue to have debates about how to tackle the monster... :-) The C++ stuff isn't that bad, in the end you can just preten

Oh yes. We're aware, we've had and continue to have debates about how to tackle the monster... :-) The C++ stuff isn't that bad, in the end you can just pretend it's messages with fixed layout structures. But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful.

rjzzleep 5y ago
Any idea why they did that as opposed to something like RPMsg?
- marcan_42 5y ago
  Apple have no reason to use existing standards when they can roll their own version tailored to their needs. It's for internal consumption, after all. This is very much a theme throughout their design.
  - rjzzleep 5y ago
    I'm aware of that, but that's not my question. Oftentimes there is a reason why they choose to ignore standards. They added a simple C++ (not full C++) layer on top of their driver code when they took FreeBSD drivers and integrated them into their OS. But there was a benefit to doing so, making the drivers arguably easier to compose. In this case the answer might be just that they were too time constrained to design something better. But I was just curious if the RPMsg framework has too many issues that would make it unsuitable.
    - gok 5y ago
      RPMsg isn't really a standard, is it? I think it was added in Linux 4.11, in April 2017, way after this stuff was introduced in Apple devices.
    - marcan_42 5y ago
      Taking a quick look at RPmsg, it seems it's from 2016. Apple have been doing coprocessors longer than that; Apple had their Secure Enclave in 2013 and that was already using their mailbox system, which is fundamentally different from the IRQs-only RPmsg that keeps all message passing entirely in shared memory. In general, this kind of message passing thing is easy to reinvent and everyone does it differently. You'll find tons of mailbox drivers in Linux already, all different.
- derefr 5y ago
  I’m guessing that what Apple did here was: 1. Probably years ago: refactor the relevant components (display driver, etc.) to run as Mach components with Mach IPC, in the form of: - One XCode project, with - an “IPC client” build target - an “IPC server” build target - a “shared datatypes library” with well-defined IPC serialization semantics, static-linked into both the IPC client and server - a single test suite that tests everything 2. Possibly when designing the M1, or possibly years ago for iOS: split the IPC server build target in two — one build target that builds an IPC-server firmware blob (RTOS unikernel), and another that builds a regular on-CPU kernel-daemon IPC server; 3. Basically do nothing in the IPC client build target — it should now be oblivious to whether its Mach messages are going to an on-CPU kernel daemon or to an RTOS-coprocessor-hosted daemon, as the Mach abstraction layer is taking care of the message routing. (Kinda like the location-obliviousness you get in Erlang when sending messages to PIDs.) This seems likely to me because there was (and still is!) both an Intel and an Apple Silicon macOS release; and they would want to share as much driver code between those releases as possible. So I think it’s very likely that they’ve written drivers structured to be “split across” Apple Silicon, while running “locally” on their Intel devices, in such a way that the differences between these approaches is effectively transparent to the kernel. To achieve #3 — and especially to achieve it while having only one shared test suite — the firmware would have to be speaking the same wire protocol to the IPC client that the in-kernel IPC daemon speaks. And, given that the in-kernel IPC daemons were designed to presume a reliable C++ interface, access to shared-memory copies of low-level Frameworks like IOKit, etc.; the firmware would need to provide this same environment to get the pre-existing code to run.
  - rjzzleep 5y ago
    That's the kind of theory I was looking for, thanks!
  - marcan_42 5y ago
    It's not Mach IPC, it's a bespoke thing only for the main DCP endpoint (they have a whole separate IPC framework for sub-drivers that have more sanely fully migrated to the DCP). It also has nothing to do with Intel, since these display controllers are only in Apple SoCs. They do have a non-DCP driver they're still shipping, that runs all the same code entirely within the macOS kext. I'm not sure exactly what it's for.
haberman 5y ago
> But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful. For what it's worth, this sounds somewhat similar to protobuf (which also supports dicts, arrays, etc). After spending many years trying to figure out the smallest, fastest, and simplest way to implement protobuf in https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb, the single best improvement I found was to make the entire memory management model arena-based. When you parse an incoming request, all the little objects (messages, arrays, maps, etc) are allocated on the arena. When you are done with it, you just free the arena. In my experience this results in code that is both simpler and faster than trying to memory-manage all of the sub-objects independently. It also integrates nicely with existing memory-management schemes: I've been able to adapt the arena model to both Ruby (tracing GC) and PHP (refcounting) runtimes. You just have to make sure that the arena itself outlives any reference to any of the objects within. (Protobuf C++ also supports arenas, that's actually where the idea of using arenas for protobuf was first introduced. But Protobuf C++ also has to stay compatible with its pre-existing API based on unique ownership, so the whole API and implementation are complicated by the fact that it needs to support both memory management styles).
- megous 5y ago
  For JSON I settled for no transformation to a different in-memory representation, just inline ondemand parsing of a JSON string buffer. Works nicely and you don't need to manage memory much at all. https://megous.com/git/megatools/tree/lib/sjson.h https://megous.com/git/megatools/tree/lib/sjson.h

rjzzleep 5y ago

Any idea why they did that as opposed to something like RPMsg?

marcan_42 5y ago
Apple have no reason to use existing standards when they can roll their own version tailored to their needs. It's for internal consumption, after all. This is very much a theme throughout their design.
- rjzzleep 5y ago
  I'm aware of that, but that's not my question. Oftentimes there is a reason why they choose to ignore standards. They added a simple C++ (not full C++) layer on top of their driver code when they took FreeBSD drivers and integrated them into their OS. But there was a benefit to doing so, making the drivers arguably easier to compose. In this case the answer might be just that they were too time constrained to design something better. But I was just curious if the RPMsg framework has too many issues that would make it unsuitable.
  - gok 5y ago
    RPMsg isn't really a standard, is it? I think it was added in Linux 4.11, in April 2017, way after this stuff was introduced in Apple devices.
  - marcan_42 5y ago
    Taking a quick look at RPmsg, it seems it's from 2016. Apple have been doing coprocessors longer than that; Apple had their Secure Enclave in 2013 and that was already using their mailbox system, which is fundamentally different from the IRQs-only RPmsg that keeps all message passing entirely in shared memory. In general, this kind of message passing thing is easy to reinvent and everyone does it differently. You'll find tons of mailbox drivers in Linux already, all different.
derefr 5y ago
I’m guessing that what Apple did here was: 1. Probably years ago: refactor the relevant components (display driver, etc.) to run as Mach components with Mach IPC, in the form of: - One XCode project, with - an “IPC client” build target - an “IPC server” build target - a “shared datatypes library” with well-defined IPC serialization semantics, static-linked into both the IPC client and server - a single test suite that tests everything 2. Possibly when designing the M1, or possibly years ago for iOS: split the IPC server build target in two — one build target that builds an IPC-server firmware blob (RTOS unikernel), and another that builds a regular on-CPU kernel-daemon IPC server; 3. Basically do nothing in the IPC client build target — it should now be oblivious to whether its Mach messages are going to an on-CPU kernel daemon or to an RTOS-coprocessor-hosted daemon, as the Mach abstraction layer is taking care of the message routing. (Kinda like the location-obliviousness you get in Erlang when sending messages to PIDs.) This seems likely to me because there was (and still is!) both an Intel and an Apple Silicon macOS release; and they would want to share as much driver code between those releases as possible. So I think it’s very likely that they’ve written drivers structured to be “split across” Apple Silicon, while running “locally” on their Intel devices, in such a way that the differences between these approaches is effectively transparent to the kernel. To achieve #3 — and especially to achieve it while having only one shared test suite — the firmware would have to be speaking the same wire protocol to the IPC client that the in-kernel IPC daemon speaks. And, given that the in-kernel IPC daemons were designed to presume a reliable C++ interface, access to shared-memory copies of low-level Frameworks like IOKit, etc.; the firmware would need to provide this same environment to get the pre-existing code to run.
- rjzzleep 5y ago
  That's the kind of theory I was looking for, thanks!
- marcan_42 5y ago
  It's not Mach IPC, it's a bespoke thing only for the main DCP endpoint (they have a whole separate IPC framework for sub-drivers that have more sanely fully migrated to the DCP). It also has nothing to do with Intel, since these display controllers are only in Apple SoCs. They do have a non-DCP driver they're still shipping, that runs all the same code entirely within the macOS kext. I'm not sure exactly what it's for.

haberman 5y ago

> But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful. For what it's worth, this sounds somewhat similar to protobuf (which also supports dicts, arrays, etc). After spending many years trying to figure out the smallest, fastest, and simplest way to implement protobuf in https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb, the single best improvement I found was to make the entire memory management model arena-based. When you parse an incoming request, all the little objects (messages, arrays, maps, etc) are allocated on the arena. When you are done with it, you just free the arena. In my experience this results in code that is both simpler and faster than trying to memory-manage all of the sub-objects independently. It also integrates nicely with existing memory-management schemes: I've been able to adapt the arena model to both Ruby (tracing GC) and PHP (refcounting) runtimes. You just have to make sure that the arena itself outlives any reference to any of the objects within. (Protobuf C++ also supports arenas, that's actually where the idea of using arenas for protobuf was first introduced. But Protobuf C++ also has to stay compatible with its pre-existing API based on unique ownership, so the whole API and implementation are complicated by the fact that it needs to support both memory management styles).

megous 5y ago
For JSON I settled for no transformation to a different in-memory representation, just inline ondemand parsing of a JSON string buffer. Works nicely and you don't need to manage memory much at all. https://megous.com/git/megatools/tree/lib/sjson.h https://megous.com/git/megatools/tree/lib/sjson.h