4 ms·

> I think the actual issue with TLS is that compilers will happily hoist TLS address computations across function calls. TLS has its own issues, but in every s

> I think the actual issue with TLS is that compilers will happily hoist TLS address computations across function calls. TLS has its own issues, but in every situation where this optimisation is fine with TLS, it's fine with FLS too.¹ If you call a function in a fibers / stackful coroutines environment, and the function has a context switch inside (say it does some blocking I/O), by the time the function returns the original context is valid again. > I do not think that FLS (at least by default) is the right solution as it would consume a lot of memory, Depends on what you store in FLS. As with TLS, there are situations when it's beneficial for memory, compared with storing replicated copies of data all over the place in arguments and data structures, and situations when it isn't beneficial. In some it will consume less memory than stackless coroutines, which store copies of the same data in async structures anyway. > increase the switch overhead If we're still discussing the article's case that stackless coroutines are better, FLS switch overhead is usually lower than stackless/async overhead. In the stackless/async version you have compiler-generated loads and stores to context-specific structs that need to be executed every time something context switches for an await. FLS overhead is updating a single pointer. Even that may be free, because you're going to update some kind of context structure pointer anyway. Even in the base case, stackless will update an equivalent pointer at absolute minimum, but often more. This is really the same argument for TLS being sometimes useful in threaded programs specifically for performance, as opposed to explicitly passing around copies of pointers and storing them in data structures all over the place. (Of course there are non-performance reasons why people choose TLS or avoid it too.) > add complexity for little benefit The benefit is faster programs, and a simpler programming model. You may reasonably disagree :-) But consider, the Linux kernel is basically all fibers with FLS, and that's a top performer :-) With regard to programming model: In Linux, it was found to be very much simpler to use fibers (called "tasks" in Linux) than to build an async I/O state machine for filesystems. This is why Linux didn't get useful async I/O in filesystems for a long time, because the state machine version of filesystem operations, though people attempted, was just far too complex in practice to implement for all corner cases, so its async properties were unreliable. -- ¹ At least no problems that are specific to FLS. If you're pointing out there are issues with TLS (without fibers) due to shared library loading/unloading or lazy memory instantiation, that's an issue but it does not support the article's implication that TLS is ok while FLS is not.

gpderetta 6y ago
So the compilation issue is only with TLS, an hypothetical FLS would not have it. The alternative to FLS is not storing duplicated data but use TLS and making sure that the compiler does not miscompile your code, either because you are extremely careful or because the compiler provides an option for it. Unfortunately compiler writers do not really see a future in stackful coroutines and are not really willing to provide a 'couroutine-safe' TLS mode. Regarding the switch overhead, the fastest switch routine I have implemented does not store or load anything from any context-specific struct, only swaps a couple of registers, so an extra load might have a cost (trivial for async io, but can be noticeable for generators). Also, the way TLS is implemented on x86-64 for example is via a segment register which would need to be changed at every coroutine switch; updating it is significantly more expensive compared to a normal register [1]. Finally, as far as I know, the kernel does not use thread local/fiber local storage, instead heavily relies on per-cpu data. I advocate the same solution for coroutines, where, if you use a 1:1 mapping from OS threads to cores, the per-cpu data would correspond to the TLS of the underlying OS thread. [1] Agner reports a reverse throughput of ~13 cycles on Neahalem. I think he stopped testing on newer architectures as it is a rarely used instruction; it needs to be expensive because it is not just a register load but it also has to update shadow limit registers. It probably has second order effects on the TLB as well.
- jlokier 6y ago
  > Finally, as far as I know, the kernel does not use thread local/fiber local storage, instead heavily relies on per-cpu data. It does use FLS. Everywhere you see "current->" or "current_thread_info()" in the Linux kernel, it's FLS, not per-cpu. There's a lot of those, some of them wrapped in other macros and inlines. "current" refers to the current task, which is the kernel's name for what the article calls fibers. Per-cpu data is used in the kernel as well, and I agree with you about using the analogous thing for coroutines which is TLS for certain things. Indeed, a FLS implementation almost certainly uses TLS for its root-of-data pointer and keeping tracking of the current thread's coroutine queue. > Agner reports a reverse throughput of ~13 cycles on Neahalem. [...] it is not just a register load but it also has to update shadow limit registers You're thinking of a segment register load, but these days we would use the WRFSBASE or WRGSBASE instructions, which don't update the segment descriptor (shadow limit etc) and don't use the GDT/LDT. Windows has supported this for userspace scheduling for a long time (but Windows has good fiber support with native FLS anyway), Java coroutines want to use it for performance on Linux, and Linux kernel looks like it will enable this soon; the CPUs have supported it for years. When those instructions aren't available, since the "current coroutine" is almost certainly a TLS variable itself that must be updated on context switch, use that as the base for FLS too. Then FLS access is one extra level of pointer indirection on access, but still hoisted and cached. For portable coroutines-and-threads code, use a portable TLS variable or pthread_getspecific() for this.

gpderetta 6y ago

So the compilation issue is only with TLS, an hypothetical FLS would not have it. The alternative to FLS is not storing duplicated data but use TLS and making sure that the compiler does not miscompile your code, either because you are extremely careful or because the compiler provides an option for it. Unfortunately compiler writers do not really see a future in stackful coroutines and are not really willing to provide a 'couroutine-safe' TLS mode. Regarding the switch overhead, the fastest switch routine I have implemented does not store or load anything from any context-specific struct, only swaps a couple of registers, so an extra load might have a cost (trivial for async io, but can be noticeable for generators). Also, the way TLS is implemented on x86-64 for example is via a segment register which would need to be changed at every coroutine switch; updating it is significantly more expensive compared to a normal register [1]. Finally, as far as I know, the kernel does not use thread local/fiber local storage, instead heavily relies on per-cpu data. I advocate the same solution for coroutines, where, if you use a 1:1 mapping from OS threads to cores, the per-cpu data would correspond to the TLS of the underlying OS thread. [1] Agner reports a reverse throughput of ~13 cycles on Neahalem. I think he stopped testing on newer architectures as it is a rarely used instruction; it needs to be expensive because it is not just a register load but it also has to update shadow limit registers. It probably has second order effects on the TLB as well.

jlokier 6y ago
> Finally, as far as I know, the kernel does not use thread local/fiber local storage, instead heavily relies on per-cpu data. It does use FLS. Everywhere you see "current->" or "current_thread_info()" in the Linux kernel, it's FLS, not per-cpu. There's a lot of those, some of them wrapped in other macros and inlines. "current" refers to the current task, which is the kernel's name for what the article calls fibers. Per-cpu data is used in the kernel as well, and I agree with you about using the analogous thing for coroutines which is TLS for certain things. Indeed, a FLS implementation almost certainly uses TLS for its root-of-data pointer and keeping tracking of the current thread's coroutine queue. > Agner reports a reverse throughput of ~13 cycles on Neahalem. [...] it is not just a register load but it also has to update shadow limit registers You're thinking of a segment register load, but these days we would use the WRFSBASE or WRGSBASE instructions, which don't update the segment descriptor (shadow limit etc) and don't use the GDT/LDT. Windows has supported this for userspace scheduling for a long time (but Windows has good fiber support with native FLS anyway), Java coroutines want to use it for performance on Linux, and Linux kernel looks like it will enable this soon; the CPUs have supported it for years. When those instructions aren't available, since the "current coroutine" is almost certainly a TLS variable itself that must be updated on context switch, use that as the base for FLS too. Then FLS access is one extra level of pointer indirection on access, but still hoisted and cached. For portable coroutines-and-threads code, use a portable TLS variable or pthread_getspecific() for this.