5 ms·

> By switching to virtual threads, you traded off the safeguards once provided by your (bounded!) thread pool and OS scheduler. The OS scheduler provides no sa

by pron 1y ago

> By switching to virtual threads, you traded off the safeguards once provided by your (bounded!) thread pool and OS scheduler. The OS scheduler provides no safeguards here (you can use the thread-per-task executor with OS threads - you'll just get an OOME much sooner), and as the virtual thread adoption guide says, you should replace bounded thread pools with a semaphore when needed because it's just a better construct. It allows you to limit concurrency for different resources in an easy and efficient way even when different threads make use of different resources. In this case, by switching to virtual threads you get to use more appropriate concurrency constructs without needing to manage threads directly, and without sharing them among tasks.

ndriscoll 1y ago
Wouldn't there be significant overhead in waking tasks from IO just to put them back to sleep on a semaphore vs. there being fewer threads servicing the IO ring for a given type of task?
- mdavidn 1y ago
  Presumably the semaphore would be used to limit the number of concurrent virtual threads. Incoming tasks can be rejected proactively if a system would overflow its limit on concurrent tasks. This approach uses a semaphore’s non-waiting acquire and release operations. See the “bulkhead pattern.” Alternatively, if a system has a fixed number of threads ingesting tasks, these would respond to backpressure by waiting for available permits before creating more virtual threads.
- pron 1y ago
  It should be the other way around. The concurrency-limited resource is usually accessed by IO (some service), so a thread would wait and then acquire the semaphore before doing the IO.
  - ndriscoll 1y ago
    I might just be taking the specific use-cases you're talking about for granted? I tend to think the reason you'd use task-specific thread pools in the first place is because the contended resource is CPU and you want different priorities for different types of tasks. E.g. if you have different types of long-lived connections (e.g. websockets) and want to make sure that even if you have 100 type A connections and 50,000 type B connections, you want ~even CPU to be split between A and B. You could use semaphores, but then you're usually waking a B (in response to some socket IO you were waiting on) just to put it to sleep. It seems like it'd make more sense to use two platform thread pools (with the different thread groups waiting on different sockets)? Is your advice more about things like object pools or connection pools?
    - pron 1y ago
      The CPU is only contended once it's at 100%. I am not aware of any interactive server (as opposed to batch processing) that behaves well at 100% CPU (beyond short bursts), so I think that the hope you can actually balance the CPU using the OS scheduler (which isn't all that good) to produce good outcomes at 100% CPU is more myth than reality. So yes, I'm talking about resources accessed via IO, because there isn't all that much you can do about a CPU that's at 100% for long durations other than getting more CPU (horizontally or vertically). However, when you have some background batch computational tasks of some limited concurrency (otherwise you'd be in trouble) then you should have some low-priority platform threads servicing that. Virtual threads work due to Little's law by allowing a high number of threads. For such low-priority background tasks you probably want a very low number of threads (often one), so the ability to have tens of thousands, or millions, of virtual threads won't help you for those background tasks anyway.
never_inline 1y ago
Stupid question: Why not provide a threadpool-like construct which may not necessarily keep threads around but limits their number?
- jeroenhd 1y ago
  The amount of actual threads is limited, by default to the amount of CPU cores the system has. The problem isn't the amount of threads itself, but the fact that the threadpool will keep taking on more work, and that it's efficient enough to blow past the CPU+I/O bottleneck. You can achieve the same problem with a threadpool if your threadpool is efficient enough. Standard concurrency tools like semaphores are capable of reducing the amount of work being done at the same time. You could also simulate classic thread pools by using concurrent lists/sets/etc to replicate traditional work queues, but you'd probably lose any advantage you'd gained from switching to green threads in the first place.
- 708145_ 1y ago
  Of course it possible to limit the number of virtual threads. A web server can have a limit on number of virtual threads too, and queue incoming request before dispatching to to workers (virtual threads). As other have said, this can be achieved with a semaphore and the bulkhead pattern. You can also limit the number of number of connections. I would expect any "production ready" web server using virtual threads having some sort of limiting. That must be the case right?
- pron 1y ago
  What you usually want to limit isn't the number of threads but the number of threads doing a particular operation, such as accessing some service, so it's more flexible (and efficient) to just guard that particular operation with a semaphore. The only place where you may want to limit the number of threads is at the entry to the system, but since the semaphore is the construct for limiting the number of anything, you might as well just use that there, too. For example, if the number of requests currently being processed is too high (above the semaphore's number of leases), you don't accept new connections.
- baguettebasket 1y ago
1718627440 1y ago
The OS won't schedule you when your write is blocking, because the pipe is full, same with reading when the pipe is empty.

by pron 1y ago

ndriscoll 1y ago

Wouldn't there be significant overhead in waking tasks from IO just to put them back to sleep on a semaphore vs. there being fewer threads servicing the IO ring for a given type of task?

mdavidn 1y ago
Presumably the semaphore would be used to limit the number of concurrent virtual threads. Incoming tasks can be rejected proactively if a system would overflow its limit on concurrent tasks. This approach uses a semaphore’s non-waiting acquire and release operations. See the “bulkhead pattern.” Alternatively, if a system has a fixed number of threads ingesting tasks, these would respond to backpressure by waiting for available permits before creating more virtual threads.
pron 1y ago
It should be the other way around. The concurrency-limited resource is usually accessed by IO (some service), so a thread would wait and then acquire the semaphore before doing the IO.
- ndriscoll 1y ago
  I might just be taking the specific use-cases you're talking about for granted? I tend to think the reason you'd use task-specific thread pools in the first place is because the contended resource is CPU and you want different priorities for different types of tasks. E.g. if you have different types of long-lived connections (e.g. websockets) and want to make sure that even if you have 100 type A connections and 50,000 type B connections, you want ~even CPU to be split between A and B. You could use semaphores, but then you're usually waking a B (in response to some socket IO you were waiting on) just to put it to sleep. It seems like it'd make more sense to use two platform thread pools (with the different thread groups waiting on different sockets)? Is your advice more about things like object pools or connection pools?
  - pron 1y ago
    The CPU is only contended once it's at 100%. I am not aware of any interactive server (as opposed to batch processing) that behaves well at 100% CPU (beyond short bursts), so I think that the hope you can actually balance the CPU using the OS scheduler (which isn't all that good) to produce good outcomes at 100% CPU is more myth than reality. So yes, I'm talking about resources accessed via IO, because there isn't all that much you can do about a CPU that's at 100% for long durations other than getting more CPU (horizontally or vertically). However, when you have some background batch computational tasks of some limited concurrency (otherwise you'd be in trouble) then you should have some low-priority platform threads servicing that. Virtual threads work due to Little's law by allowing a high number of threads. For such low-priority background tasks you probably want a very low number of threads (often one), so the ability to have tens of thousands, or millions, of virtual threads won't help you for those background tasks anyway.

never_inline 1y ago

Stupid question: Why not provide a threadpool-like construct which may not necessarily keep threads around but limits their number?

jeroenhd 1y ago
The amount of actual threads is limited, by default to the amount of CPU cores the system has. The problem isn't the amount of threads itself, but the fact that the threadpool will keep taking on more work, and that it's efficient enough to blow past the CPU+I/O bottleneck. You can achieve the same problem with a threadpool if your threadpool is efficient enough. Standard concurrency tools like semaphores are capable of reducing the amount of work being done at the same time. You could also simulate classic thread pools by using concurrent lists/sets/etc to replicate traditional work queues, but you'd probably lose any advantage you'd gained from switching to green threads in the first place.
708145_ 1y ago
Of course it possible to limit the number of virtual threads. A web server can have a limit on number of virtual threads too, and queue incoming request before dispatching to to workers (virtual threads). As other have said, this can be achieved with a semaphore and the bulkhead pattern. You can also limit the number of number of connections. I would expect any "production ready" web server using virtual threads having some sort of limiting. That must be the case right?
pron 1y ago
What you usually want to limit isn't the number of threads but the number of threads doing a particular operation, such as accessing some service, so it's more flexible (and efficient) to just guard that particular operation with a semaphore. The only place where you may want to limit the number of threads is at the entry to the system, but since the semaphore is the construct for limiting the number of anything, you might as well just use that there, too. For example, if the number of requests currently being processed is too high (above the semaphore's number of leases), you don't accept new connections.
baguettebasket 1y ago

1718627440 1y ago

The OS won't schedule you when your write is blocking, because the pipe is full, same with reading when the pipe is empty.