5 ms·

Given a syscall that does nothing, a full round-trip under BSD would require about 40μs, whereas on a user-space Mach system it would take just under 500μs Thi

Given a syscall that does nothing, a full round-trip under BSD would require about 40μs, whereas on a user-space Mach system it would take just under 500μs This is still true, but to a lesser extent, on MacOSX (or at least was, a decade ago). I was writing HPC drivers for a cluster interconnect. So performance was critical. We had been using the BSD ioctl system to communicate with our drivers because we used ioctls in all our other drivers (Linux, Solaris, FreeBSD, Windows). I did some microbenchmarks and noticed that it was far slower than FreeBSD or Linux ioctls & complained. Apple suggested that I re-write the app/driver communication using IOKit, which is Mach based. The result was something that was twice as slow.

big_chungus 7y ago
So the obvious question is how it has changed over the years. What has apple done to pull latency down to a comparable time?
- drewg123 7y ago
  I doubt that they care. In the period where I worked on Mac drivers and kept close track of the Darwin sources (roughly 2003->2013) I saw very little in the way of performance improvements. AFAIK, they still don't support 15 year old technologies like MSI-X that permit efficient multi-queue network drivers. Have you ever built a project by hand on MacOS and then on Linux (or FreeBSD)? Have you noticed how absurdly, painfully slow it is running autoconf on MacOSX? That's because MacOS system calls are horrifically slow compared to Linux / BSD.
  - SomeOldThrow 7y ago
    What does “horrifically slow” mean compared to a monolithic syscall? I haven’t run autoconf on linux in years.
    - drewg123 7y ago
      Small number of minutes vs less than a minute. The last time I built something by hand in both places, it seemed to take 2x to 3x as long to run all the autoconf checks on MacOS. That's basically a fork / exec / open /close sort of benchmark.
      - panpanna 7y ago
        (and general filesystem performance)
    - panpanna 7y ago
      You should try it yourself. We moved almost all development machines to Linux after someone demonstrated how fast Linux built some C project compared to osx and windows.
      - SomeOldThrow 7y ago
        Spent a while thinking why I don’t see this much. I work mostly with higher level languages, so the beefiest things I run are probably webpack and the typescript language server. I wonder if maybe the problem is fork or some derivative effect. That, or I just don’t have any syscall heavy loads in my life, full stop.
    - asveikau 7y ago
      It doesn't need autoconf to see this. I have seen it in a project with a simple make -j, compiling with clang in both places. Builds finish much faster on Linux. To be fair this may be many things and not just mach syscalls.
  - ridiculous_fish 7y ago
    It's `fork` in particular which is much slower. It has to fiddle with Mach port rights, doesn't do overcommit, etc. And autotools is very fork heavy. Apple provides posix_spawn which is much much faster. Running /usr/bin/false 1000 times in the fish shell is nearly twice as fast (2.25s to 1.25s) when using posix_spawn instead of fork, on the Mac. Autotools doesn't use posix_spawn because there's no benefit on Linux, but it is what Apple's frameworks use internally for process launching.
    - geofft 7y ago
      > Autotools doesn't use posix_spawn because there's no benefit on Linux Wait, autotools don't make syscalls at all, do they? I thought it's just shell and make. Is the problem that /bin/sh and /usr/bin/make use fork and it would help to have a Bourne shell and GNU-compatible make that used posix_spawn?
AHTERIX5000 7y ago
The way XNU converged from its microkernel nature to a more monolithic one has led to all kinds of funkiness. You still have more syscall overhead (and VM but message passing isn't probably the culprit idk) but can't really trust Mach to separate kernel's internal systems like one would in a typical microkernel. For presumably optimization reasons arbitrary pointers (rather than checked messages) are passed around different parts of the kernel. And that exposes some quite bad security issues from time to time.
logicchains 7y ago
>Given a syscall that does nothing, a full round-trip under BSD would require about 40μs, whereas on a user-space Mach system it would take just under 500μs Point of reference: I work in HFT, and it's possible to process a market data update from an exchange, reprice a complex financial instrument, decide whether to place an order, and send that order back towards the exchange all in under 5us. Another point of reference: raising one float to the power of another (a^b) could be done over a 100,000 times in the same time it takes for Mach to do a single syscall.
- la_barba 7y ago
  I really don't mean this as a knock on the work you do.. but what is the actual complexity of the operations you're describing? From what little I understand of it, all HFT operations are designed to be as simple as possible. Like.. I've read that the decision to the order engine could be as simple as a comparison between two numbers (some kind of arbitrage model), and the price adjustment could be just a simple addition/alpha adjustment. Basically, the offline/non-realtime stuff in HFT is way more complicated than the live/real-time stuff.
  - logicchains 7y ago
    Still more complex than a no-op syscall, especially including message encoding/decoding.
- gok 7y ago
  That 500 microsecond number is from a 1993 paper on 1991 hardware (a 50 MHz 486DX-50). Your HFT code wouldn't load on it, and in 500usec it would be lucky to do a single float^float power operation.
  - AnimalMuppet 7y ago
    > in 500usec it would be lucky to do a single float^float power operation. The rest of your post seems OK, but this claim is almost certainly off. 2000 float^float operations takes a full second? Even on a 486DX, no way.
xenadu02 7y ago
Have you run a test recently? Trivial mach IPC round-trip (same process) seems to be about ~8.5µs in my test, so around 4µs per call. I'm sure if you transferred port rights, memory, etc the cost would go up. edit: I would advise anyone to run your own tests. Anecdotes aren't data and some things like syscalls are not nearly as expensive on modern hardware as they once were. Sometimes things can get ridiculously faster than you remember. For example the original iPhone took nearly 200ns to do objc_msgSend() [1]. In 2016 modern hardware did the same thing in 2.6ns [2]. That's two orders of magnitude improvement. So saying "message passing is slow" is not a correct statement... it's almost as cheap as a C++ virtual method call. [1] https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performance-comparisons-of-common-operations-2016-edition.html https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performa... [2] https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performance-comparisons-of-common-operations-2016-edition.html https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performa...
- mcguire 7y ago
  What is a BSD syscall on the same hardware?

big_chungus 7y ago

So the obvious question is how it has changed over the years. What has apple done to pull latency down to a comparable time?

drewg123 7y ago
I doubt that they care. In the period where I worked on Mac drivers and kept close track of the Darwin sources (roughly 2003->2013) I saw very little in the way of performance improvements. AFAIK, they still don't support 15 year old technologies like MSI-X that permit efficient multi-queue network drivers. Have you ever built a project by hand on MacOS and then on Linux (or FreeBSD)? Have you noticed how absurdly, painfully slow it is running autoconf on MacOSX? That's because MacOS system calls are horrifically slow compared to Linux / BSD.
- SomeOldThrow 7y ago
  What does “horrifically slow” mean compared to a monolithic syscall? I haven’t run autoconf on linux in years.
  - drewg123 7y ago
    Small number of minutes vs less than a minute. The last time I built something by hand in both places, it seemed to take 2x to 3x as long to run all the autoconf checks on MacOS. That's basically a fork / exec / open /close sort of benchmark.
    - panpanna 7y ago
      (and general filesystem performance)
  - panpanna 7y ago
    You should try it yourself. We moved almost all development machines to Linux after someone demonstrated how fast Linux built some C project compared to osx and windows.
    - SomeOldThrow 7y ago
      Spent a while thinking why I don’t see this much. I work mostly with higher level languages, so the beefiest things I run are probably webpack and the typescript language server. I wonder if maybe the problem is fork or some derivative effect. That, or I just don’t have any syscall heavy loads in my life, full stop.
  - asveikau 7y ago
    It doesn't need autoconf to see this. I have seen it in a project with a simple make -j, compiling with clang in both places. Builds finish much faster on Linux. To be fair this may be many things and not just mach syscalls.
- ridiculous_fish 7y ago
  It's `fork` in particular which is much slower. It has to fiddle with Mach port rights, doesn't do overcommit, etc. And autotools is very fork heavy. Apple provides posix_spawn which is much much faster. Running /usr/bin/false 1000 times in the fish shell is nearly twice as fast (2.25s to 1.25s) when using posix_spawn instead of fork, on the Mac. Autotools doesn't use posix_spawn because there's no benefit on Linux, but it is what Apple's frameworks use internally for process launching.
  - geofft 7y ago
    > Autotools doesn't use posix_spawn because there's no benefit on Linux Wait, autotools don't make syscalls at all, do they? I thought it's just shell and make. Is the problem that /bin/sh and /usr/bin/make use fork and it would help to have a Bourne shell and GNU-compatible make that used posix_spawn?

AHTERIX5000 7y ago

The way XNU converged from its microkernel nature to a more monolithic one has led to all kinds of funkiness. You still have more syscall overhead (and VM but message passing isn't probably the culprit idk) but can't really trust Mach to separate kernel's internal systems like one would in a typical microkernel. For presumably optimization reasons arbitrary pointers (rather than checked messages) are passed around different parts of the kernel. And that exposes some quite bad security issues from time to time.

logicchains 7y ago

>Given a syscall that does nothing, a full round-trip under BSD would require about 40μs, whereas on a user-space Mach system it would take just under 500μs Point of reference: I work in HFT, and it's possible to process a market data update from an exchange, reprice a complex financial instrument, decide whether to place an order, and send that order back towards the exchange all in under 5us. Another point of reference: raising one float to the power of another (a^b) could be done over a 100,000 times in the same time it takes for Mach to do a single syscall.

la_barba 7y ago
I really don't mean this as a knock on the work you do.. but what is the actual complexity of the operations you're describing? From what little I understand of it, all HFT operations are designed to be as simple as possible. Like.. I've read that the decision to the order engine could be as simple as a comparison between two numbers (some kind of arbitrage model), and the price adjustment could be just a simple addition/alpha adjustment. Basically, the offline/non-realtime stuff in HFT is way more complicated than the live/real-time stuff.
- logicchains 7y ago
  Still more complex than a no-op syscall, especially including message encoding/decoding.
gok 7y ago
That 500 microsecond number is from a 1993 paper on 1991 hardware (a 50 MHz 486DX-50). Your HFT code wouldn't load on it, and in 500usec it would be lucky to do a single float^float power operation.
- AnimalMuppet 7y ago
  > in 500usec it would be lucky to do a single float^float power operation. The rest of your post seems OK, but this claim is almost certainly off. 2000 float^float operations takes a full second? Even on a 486DX, no way.

xenadu02 7y ago

Have you run a test recently? Trivial mach IPC round-trip (same process) seems to be about ~8.5µs in my test, so around 4µs per call. I'm sure if you transferred port rights, memory, etc the cost would go up. edit: I would advise anyone to run your own tests. Anecdotes aren't data and some things like syscalls are not nearly as expensive on modern hardware as they once were. Sometimes things can get ridiculously faster than you remember. For example the original iPhone took nearly 200ns to do objc_msgSend() [1]. In 2016 modern hardware did the same thing in 2.6ns [2]. That's two orders of magnitude improvement. So saying "message passing is slow" is not a correct statement... it's almost as cheap as a C++ virtual method call. [1] https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performance-comparisons-of-common-operations-2016-edition.html https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performa... [2] https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performance-comparisons-of-common-operations-2016-edition.html https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performa...

mcguire 7y ago
What is a BSD syscall on the same hardware?