4 ms·

At some level of performance demand, you need to reverse-engineer (or read the source code of, if available) the kernel to get good performance on any OS.

fsflover 4y ago
Except, you know, on a FLOSS OS.
- shadowgovt 4y ago
  I slipped a note about the source code in... But on further reflection, I suspect for the best performance some reverse-engineering will still be needed even if all components are open (does the source code really tell you what assembly is generated? Does the assembly really tell you how the CPU will execute it?)
  - smoldesu 4y ago
    When was the last time you needed to read assembly code to debug a kernel-level issue in FOSS software? There's a lot of software you can use to profile the entire kernel and userspace[0], or explore the stack trace[1] of your software. If you're not using those and instead debugging your software by counting SIMD calls in the Ghidra editor, you're wasting an unconscionable amount of time. People read through assembly to fix compiler issues, not to rationalize what the kernel is doing. You're right that this skill is still required for "perfect" optimization, but it's not really relevant when comparing kernels. [0] http://www.sysprof.com/ http://www.sysprof.com/ [1] https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph
    - saagarjha 4y ago
      It’s often useful when confirming what code made it into the kernel you’re testing.
- phendrenad2 4y ago
  You're definitely not writing performant BSD/Linux code without closely studying the API, and reading the manpages won't be sufficient.
  - dahfizz 4y ago
    There's a big difference between studying the API and not having an API. It's ridiculous that you have to call an unexported symbol for something as basic and fast locking.
    - phendrenad2 4y ago
      Some research borders on reverse-engineering. I know I've had to dig into the source of Wayland and X11 to do anything related to them. Sure, there's an "API". The API is the source code.
- lapcat 4y ago
  There's quite a bit of macOS that's open source. The author links to source code in Firefox, and that source code has links to Apple's source code: > // For information about the following undocumented flags and functions see > // https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.h https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.... and > // https://github.com/apple/darwin-libplatform/blob/main/private/os/lock_private.h https://github.com/apple/darwin-libplatform/blob/main/privat...
  - cbsmith 4y ago
    Unfortunately, when it comes to performance, the devil is in the details, so even the possibility of stuff that isn't in Darwin means you're doing some reverse engineering.
    - saagarjha 4y ago
      This, as mentioned above, is true across OSes.
      - cbsmith 4y ago
        No, it's not true across OSes. Some OSes are fully open source, and some OSes don't provide their source code. There are OSes like MacOS where the core bits have been open sourced to varying degrees over time. Notably, Apple appears to not be updating the Darwin project the same way they once did, and the XNU kernel source code was last updated in mid-2021 (and pretty sparingly). When I searched for os_unfair_lock_with_options, I didn't get any matches. Aside from some real-time Linuxy platforms where the kernel runs on top of a proprietary microkernel, you can find the source code for any Linux synchronization primitive.
        dagmx 4y ago
        Mid 2021 lines up with the last major release of macOS…updates to the OSS repo happen after each major release. macOS Ventura has yet to release. So your complaint there seems misplaced

fsflover 4y ago

Except, you know, on a FLOSS OS.

shadowgovt 4y ago
I slipped a note about the source code in... But on further reflection, I suspect for the best performance some reverse-engineering will still be needed even if all components are open (does the source code really tell you what assembly is generated? Does the assembly really tell you how the CPU will execute it?)
- smoldesu 4y ago
  When was the last time you needed to read assembly code to debug a kernel-level issue in FOSS software? There's a lot of software you can use to profile the entire kernel and userspace[0], or explore the stack trace[1] of your software. If you're not using those and instead debugging your software by counting SIMD calls in the Ghidra editor, you're wasting an unconscionable amount of time. People read through assembly to fix compiler issues, not to rationalize what the kernel is doing. You're right that this skill is still required for "perfect" optimization, but it's not really relevant when comparing kernels. [0] http://www.sysprof.com/ http://www.sysprof.com/ [1] https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph
  - saagarjha 4y ago
    It’s often useful when confirming what code made it into the kernel you’re testing.
phendrenad2 4y ago
You're definitely not writing performant BSD/Linux code without closely studying the API, and reading the manpages won't be sufficient.
- dahfizz 4y ago
  There's a big difference between studying the API and not having an API. It's ridiculous that you have to call an unexported symbol for something as basic and fast locking.
  - phendrenad2 4y ago
    Some research borders on reverse-engineering. I know I've had to dig into the source of Wayland and X11 to do anything related to them. Sure, there's an "API". The API is the source code.
lapcat 4y ago
There's quite a bit of macOS that's open source. The author links to source code in Firefox, and that source code has links to Apple's source code: > // For information about the following undocumented flags and functions see > // https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.h https://github.com/apple/darwin-xnu/blob/main/bsd/sys/ulock.... and > // https://github.com/apple/darwin-libplatform/blob/main/private/os/lock_private.h https://github.com/apple/darwin-libplatform/blob/main/privat...
- cbsmith 4y ago
  Unfortunately, when it comes to performance, the devil is in the details, so even the possibility of stuff that isn't in Darwin means you're doing some reverse engineering.
  - saagarjha 4y ago
    This, as mentioned above, is true across OSes.
    - cbsmith 4y ago
      No, it's not true across OSes. Some OSes are fully open source, and some OSes don't provide their source code. There are OSes like MacOS where the core bits have been open sourced to varying degrees over time. Notably, Apple appears to not be updating the Darwin project the same way they once did, and the XNU kernel source code was last updated in mid-2021 (and pretty sparingly). When I searched for os_unfair_lock_with_options, I didn't get any matches. Aside from some real-time Linuxy platforms where the kernel runs on top of a proprietary microkernel, you can find the source code for any Linux synchronization primitive.
      - dagmx 4y ago
        Mid 2021 lines up with the last major release of macOS…updates to the OSS repo happen after each major release. macOS Ventura has yet to release. So your complaint there seems misplaced