4 ms·

The performance tradeoffs are very different for different languages. In Java, virtual threads add no FFI costs and virtually no performance impact not because

by pron 3y ago

The performance tradeoffs are very different for different languages. In Java, virtual threads add no FFI costs and virtually no performance impact not because the baseline is lower but because memory management in Java is just so incredibly efficient these days (although that does come at the cost of a higher footprint). So the impact and the tradeoffs are not the same. Allocating heap memory in a very general way is simply faster -- even in absolute terms -- in Java than in a low-level language. Java, unlike .NET or Rust, also doesn't allow pointers into the stack, so we can do very efficient copying "in the shadow" of cache misses, and that takes care of FFI.

vacuity 3y ago
> also doesn't allow pointers into the stack, so we can do very efficient copying "in the shadow" of cache misses How does this work?
- pron 3y ago
  First, the use-case where virtual threads offer the most benefit is servers with high concurrency (due to Little's law https://youtu.be/07V08SB1l8c?si=rwTQrnHBnp4NGrj7 https://youtu.be/07V08SB1l8c?si=rwTQrnHBnp4NGrj7), which means we're talking about a very large number of threads. That, in turn, means that the state of all those threads cannot fit in the CPU cache, so even the most efficient implementation possible, i.e. one that simply changes the sp register to point to a new stack will incur an expensive cache miss. That means that copying small sections of the stack is almost free (because a cache miss+copy isn't all that more expensive than just a cache miss). Copying portions of the stack means that you can continue executing code directly in OS threads (rather than change the stack pointer), so the overhead for FFI is zero. However, you cannot copy portions of stack if there are pointers into the stack. In .NET and in low-level languages there are pointers into the stack which makes copying inefficient. Furthermore, managing these stack portions efficiently in the heap requires extremely efficient dynamic memory management, which is something that languages with good GCs do better than low-level languages with more direct memory management.

by pron 3y ago

vacuity 3y ago

> also doesn't allow pointers into the stack, so we can do very efficient copying "in the shadow" of cache misses How does this work?

pron 3y ago
First, the use-case where virtual threads offer the most benefit is servers with high concurrency (due to Little's law https://youtu.be/07V08SB1l8c?si=rwTQrnHBnp4NGrj7 https://youtu.be/07V08SB1l8c?si=rwTQrnHBnp4NGrj7), which means we're talking about a very large number of threads. That, in turn, means that the state of all those threads cannot fit in the CPU cache, so even the most efficient implementation possible, i.e. one that simply changes the sp register to point to a new stack will incur an expensive cache miss. That means that copying small sections of the stack is almost free (because a cache miss+copy isn't all that more expensive than just a cache miss). Copying portions of the stack means that you can continue executing code directly in OS threads (rather than change the stack pointer), so the overhead for FFI is zero. However, you cannot copy portions of stack if there are pointers into the stack. In .NET and in low-level languages there are pointers into the stack which makes copying inefficient. Furthermore, managing these stack portions efficiently in the heap requires extremely efficient dynamic memory management, which is something that languages with good GCs do better than low-level languages with more direct memory management.