6 ms·

> However, as I dug into Apple’s libraries and kernel, I noticed that some spin locks were indeed available, and they did the spinning in kernel-space where the

> However, as I dug into Apple’s libraries and kernel, I noticed that some spin locks were indeed available, and they did the spinning in kernel-space where they could make a more informed choice with regards to load and scheduling. Those would have been an excellent choice for our use-case. > So how do you use them? Well, it turns out they’re not documented. They rely on a non-public function and flags which I had to duplicate in Firefox. Gotta love it. You need to reverse engineer the kernel to get good performance on MacOS.

lilyball 4y ago
The code comment links to Apple Darwin GitHub repos. It sounds like “undocumented” means “not in the SDK”, rather than requiring reverse-engineering.
shadowgovt 4y ago
At some level of performance demand, you need to reverse-engineer (or read the source code of, if available) the kernel to get good performance on any OS.
- fsflover 4y ago
  Except, you know, on a FLOSS OS.
  - shadowgovt 4y ago
    I slipped a note about the source code in... But on further reflection, I suspect for the best performance some reverse-engineering will still be needed even if all components are open (does the source code really tell you what assembly is generated? Does the assembly really tell you how the CPU will execute it?)
    - smoldesu 4y ago
      When was the last time you needed to read assembly code to debug a kernel-level issue in FOSS software? There's a lot of software you can use to profile the entire kernel and userspace[0], or explore the stack trace[1] of your software. If you're not using those and instead debugging your software by counting SIMD calls in the Ghidra editor, you're wasting an unconscionable amount of time. People read through assembly to fix compiler issues, not to rationalize what the kernel is doing. You're right that this skill is still required for "perfect" optimization, but it's not really relevant when comparing kernels. [0] http://www.sysprof.com/ http://www.sysprof.com/ [1] https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph
      - saagarjha 4y ago
        It’s often useful when confirming what code made it into the kernel you’re testing.
  - phendrenad2 4y ago
    You're definitely not writing performant BSD/Linux code without closely studying the API, and reading the manpages won't be sufficient.
    - dahfizz 4y ago
      There's a big difference between studying the API and not having an API. It's ridiculous that you have to call an unexported symbol for something as basic and fast locking.
      - phendrenad2 4y ago
        Some research borders on reverse-engineering. I know I've had to dig into the source of Wayland and X11 to do anything related to them. Sure, there's an "API". The API is the source code.
  - lapcat 4y ago
    There's quite a bit of macOS that's open source. The author links to source code in Firefox, and that source code has links to Apple's source code: > // For information about the following undocumented flags and functions see > // https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.h https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.... and > // https://github.com/apple/darwin-libplatform/blob/main/private/os/lock_private.h https://github.com/apple/darwin-libplatform/blob/main/privat...
    - cbsmith 4y ago
      Unfortunately, when it comes to performance, the devil is in the details, so even the possibility of stuff that isn't in Darwin means you're doing some reverse engineering.
      - saagarjha 4y ago
        This, as mentioned above, is true across OSes.
        cbsmith 4y ago
        No, it's not true across OSes. Some OSes are fully open source, and some OSes don't provide their source code. There are OSes like MacOS where the core bits have been open sourced to varying degrees over time. Notably, Apple appears to not be updating the Darwin project the same way they once did, and the XNU kernel source code was last updated in mid-2021 (and pretty sparingly). When I searched for os_unfair_lock_with_options, I didn't get any matches. Aside from some real-time Linuxy platforms where the kernel runs on top of a proprietary microkernel, you can find the source code for any Linux synchronization primitive.
        dagmx 4y ago
        Mid 2021 lines up with the last major release of macOS…updates to the OSS repo happen after each major release. macOS Ventura has yet to release. So your complaint there seems misplaced
nsgi 4y ago
I wonder what the Chromium team did
- LtWorf 4y ago
  clone webkit, done by apple, with access to their internal documentation
  - wongarsu 4y ago
    > done by apple, with access to their internal documentation I'm pretty sure there was some antitrust investigation over Microsoft doing the same thing: giving their own software an unfair advantage on their operating system due to the insights the Office team could gain from the kernel team. Microsoft made a convincing argument that the relevant teams never talked to each other, and the Office developers just reverse-engineered undocumented Windows APIs, the same any other developer would have had to.
    - tick_tock_tick 4y ago
      Yes but Apple also sells the hardware so that makes it legal (for some reason I still don't understand).
      - LtWorf 4y ago
        I remember there was a comic strip that apple and google get away with stuff because they don't have an S in the name, like micro$oft.
      - Melatonic 4y ago
        Google also pays Apple and ungodly amount to make Google the default search engine - who knows what they have going on behind the scenes.
  - saagarjha 4y ago
    Google has thrown out most of the code that this would be relevant for.
phendrenad2 4y ago
It's funny because in the 1,990s this was cause for the whole government to go after Microsoft and talk about forcibly breaking their browser division away from their OS division.
- cestith 4y ago
  IE was far from using a single undocumented API call.
  - jaywalk 4y ago
    I think it's safe to assume that Safari is also far from using a single undocumented API call.
    - cestith 4y ago
      That may be true. It's also using an open source browser engine, isn't being used to embrace/enhance/extinguish web features, and isn't built deeply into the market dominant OS by a company already found to be using other anticompetitive practices. Don't get me wrong, I'm not nominating Apple for sainthood. They're far from the territory Microsoft was treading at the time.
drewg123 4y ago
That's always been true on MacOS, especially if you care about performance, are doing something weird, or both. I worked on OS-Bypass HPC drivers for a pre-infiniband/ROCE NIC in the early/mid 2000s. About 90% of my time doing the driver was spent reading the kernel and IOKit sources, trying to figure out how I could use the highest performance interfaces possible in a stable way. For example, Apple really wanted you to use IOKit primitives (which used Mach IPC) to call functionality in your driver, but microbenchmarks showed it was about 50% higher latency than simple BSD ioctls from the BSD side of the kernel. That fit nicely with our driver model (which was split into OS-agnostic and OS specific code, and which used ioctls on every other *nix, and even on Windows), and got us slightly better performance.
deleted 4y ago
[deleted]

lilyball 4y ago

The code comment links to Apple Darwin GitHub repos. It sounds like “undocumented” means “not in the SDK”, rather than requiring reverse-engineering.

shadowgovt 4y ago

At some level of performance demand, you need to reverse-engineer (or read the source code of, if available) the kernel to get good performance on any OS.

fsflover 4y ago
Except, you know, on a FLOSS OS.
- shadowgovt 4y ago
  I slipped a note about the source code in... But on further reflection, I suspect for the best performance some reverse-engineering will still be needed even if all components are open (does the source code really tell you what assembly is generated? Does the assembly really tell you how the CPU will execute it?)
  - smoldesu 4y ago
    When was the last time you needed to read assembly code to debug a kernel-level issue in FOSS software? There's a lot of software you can use to profile the entire kernel and userspace[0], or explore the stack trace[1] of your software. If you're not using those and instead debugging your software by counting SIMD calls in the Ghidra editor, you're wasting an unconscionable amount of time. People read through assembly to fix compiler issues, not to rationalize what the kernel is doing. You're right that this skill is still required for "perfect" optimization, but it's not really relevant when comparing kernels. [0] http://www.sysprof.com/ http://www.sysprof.com/ [1] https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph
    - saagarjha 4y ago
      It’s often useful when confirming what code made it into the kernel you’re testing.
- phendrenad2 4y ago
  You're definitely not writing performant BSD/Linux code without closely studying the API, and reading the manpages won't be sufficient.
  - dahfizz 4y ago
    There's a big difference between studying the API and not having an API. It's ridiculous that you have to call an unexported symbol for something as basic and fast locking.
    - phendrenad2 4y ago
      Some research borders on reverse-engineering. I know I've had to dig into the source of Wayland and X11 to do anything related to them. Sure, there's an "API". The API is the source code.
- lapcat 4y ago
  There's quite a bit of macOS that's open source. The author links to source code in Firefox, and that source code has links to Apple's source code: > // For information about the following undocumented flags and functions see > // https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.h https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.... and > // https://github.com/apple/darwin-libplatform/blob/main/private/os/lock_private.h https://github.com/apple/darwin-libplatform/blob/main/privat...
  - cbsmith 4y ago
    Unfortunately, when it comes to performance, the devil is in the details, so even the possibility of stuff that isn't in Darwin means you're doing some reverse engineering.
    - saagarjha 4y ago
      This, as mentioned above, is true across OSes.
      - cbsmith 4y ago
        No, it's not true across OSes. Some OSes are fully open source, and some OSes don't provide their source code. There are OSes like MacOS where the core bits have been open sourced to varying degrees over time. Notably, Apple appears to not be updating the Darwin project the same way they once did, and the XNU kernel source code was last updated in mid-2021 (and pretty sparingly). When I searched for os_unfair_lock_with_options, I didn't get any matches. Aside from some real-time Linuxy platforms where the kernel runs on top of a proprietary microkernel, you can find the source code for any Linux synchronization primitive.
        dagmx 4y ago
        Mid 2021 lines up with the last major release of macOS…updates to the OSS repo happen after each major release. macOS Ventura has yet to release. So your complaint there seems misplaced

nsgi 4y ago

I wonder what the Chromium team did

LtWorf 4y ago
clone webkit, done by apple, with access to their internal documentation
- wongarsu 4y ago
  > done by apple, with access to their internal documentation I'm pretty sure there was some antitrust investigation over Microsoft doing the same thing: giving their own software an unfair advantage on their operating system due to the insights the Office team could gain from the kernel team. Microsoft made a convincing argument that the relevant teams never talked to each other, and the Office developers just reverse-engineered undocumented Windows APIs, the same any other developer would have had to.
  - tick_tock_tick 4y ago
    Yes but Apple also sells the hardware so that makes it legal (for some reason I still don't understand).
    - LtWorf 4y ago
      I remember there was a comic strip that apple and google get away with stuff because they don't have an S in the name, like micro$oft.
    - Melatonic 4y ago
      Google also pays Apple and ungodly amount to make Google the default search engine - who knows what they have going on behind the scenes.
- saagarjha 4y ago
  Google has thrown out most of the code that this would be relevant for.

phendrenad2 4y ago

It's funny because in the 1,990s this was cause for the whole government to go after Microsoft and talk about forcibly breaking their browser division away from their OS division.

cestith 4y ago
IE was far from using a single undocumented API call.
- jaywalk 4y ago
  I think it's safe to assume that Safari is also far from using a single undocumented API call.
  - cestith 4y ago
    That may be true. It's also using an open source browser engine, isn't being used to embrace/enhance/extinguish web features, and isn't built deeply into the market dominant OS by a company already found to be using other anticompetitive practices. Don't get me wrong, I'm not nominating Apple for sainthood. They're far from the territory Microsoft was treading at the time.

drewg123 4y ago

That's always been true on MacOS, especially if you care about performance, are doing something weird, or both. I worked on OS-Bypass HPC drivers for a pre-infiniband/ROCE NIC in the early/mid 2000s. About 90% of my time doing the driver was spent reading the kernel and IOKit sources, trying to figure out how I could use the highest performance interfaces possible in a stable way. For example, Apple really wanted you to use IOKit primitives (which used Mach IPC) to call functionality in your driver, but microbenchmarks showed it was about 50% higher latency than simple BSD ioctls from the BSD side of the kernel. That fit nicely with our driver model (which was split into OS-agnostic and OS specific code, and which used ioctls on every other *nix, and even on Windows), and got us slightly better performance.

deleted 4y ago

[deleted]