5 ms·

I think virtual threads are huge. The problem with regular threads is (a) multi-kb memory stack per thread and (b) consuming a file handle. Either of those s

• riku_iki 3y ago
  > Either of those severely limits the scalability you can avoid both issues by using 20yo executorservice.
  • pron 3y ago
    If the code is simple, blocking code, then the number of threads required in the pool is the average total duration of a request times the fanout times the request rate. That number can easily reach many thousands and more.
    • riku_iki 3y ago
      yes, you shouldn't add blocking code into executorservice..
      • kaba0 3y ago
        Wtf, where on Earth do you put blocking code then? Firing off some long-running task in a background thread through executors is bog-standard usecase.
        • riku_iki 3y ago
          discussion was about specific context: avoiding overhead from spawning millions of threads, in this case you shouldn't have any blocking code at all, all API should utilize epoll underneath or something similar.
          • kaba0 3y ago
            And where do you handle callbacks?
            • riku_iki 3y ago
              there are tons of variations, depending on your logic and API. The closest to virtual threads is ForkJoinPool and RecursiveTask, where you can have code like regular blocking code: var f = async_api_returns_future(); ... var res = f.join(); but join() won't block OS/JVM thread, but make it to perform other tasks in the queue. Or you can design API which will receive executorService as params, and run callback there, e.g.: async_call(Callable callback, ExecutorService threadPool);
              • pron 3y ago
                One way to see how different virtual threads are from our old mechanisms is to ask yourself how many IO operations you can have in flight. There are two options: either the operations are blocking, in which case the number will be equal to the (very limited) number of threads in all of your thread pools combined, or the operations are non-blocking, in which case thread context that is so necessary for troubleshooting and JFR profiles is lost (e.g. JFR can't know on behalf of whom is some IO operation performed because the "owner" of some operation -- in the design of the Java platform -- can only be a thread). Virtual threads allow you to have hundreds of thousands (or even millions) of I/O operations in flight (which you need for high throughput as a result of Little's law) while still preserving observable context. BTW, as for fork-join's `join`, not only is it designed for pure computation only and cannot help with IO, but every `join` increases the depth of the stack, so it is fundamentally limited in how much it can help. FJ is designed for pure computation workloads, so in practice that's not a huge limitation, but virtual threads are designed for IO workloads. I apologise for not going into more depth here, but as you can imagine, with a user base numbering in the many millions, we can only afford to put in depth explanations in texts and videos that gain a large audience, but once you've familiarised yourself with the material I'll gladly answer specific questions (and perhaps your questions will help us improve our material, too).
      • pron 3y ago
        Then you either don't get the same scalability that virtual threads give you or you get it but with asynchronous code that requires not just more work but can't enjoy the same observability/debuggability on the Java platform.
        • riku_iki 3y ago
          could you give example what requires more work exactly and where virtual threads give more "observability"?..
          • pron 3y ago
            Sure. Because handling server requests typically requires IO, if you wish not to block you need some way to sequence operations that is different from the ordinary sequential composition of the language (beforeIO(); blockingIO(); afterIO()). Similarly, other language constructs that build on top of basic sequential composition -- loops, exceptions, try/finally -- no longer work across the IO boundary. Instead you must reach for an asynchronous composition DSL (such as the one offered by CompletableFuture) which is not as composable as the basic language primitives. Moreover, the platform now has no insight about your composition. Exceptions, which are designed to give context in the form of a thread stack trace, simply don't know about the context as it's not composed through the normal composition (in plain terms, stack traces in asynchronous code don't give you the operation's context). Debuggers cannot step through the asynchronous flow because the platform's built in debugging support works only by stepping through threads, and profilers are no longer able to assign IO to operations: a server that's under heavy load may show up as idle thread pools in a profiler because the platform cannot assign an asynchronous operation to some asynchronous pipeline such as CompletableFutures because these are not observable constructs of the Java platform. Virtual threads give you the same scalability as asynchronous code but in a way that fits with the design of the Java platform. All language constructs work and compose well, debuggers step through code, and profilers can understand what's going on. That's not to say that some other platform could not be designed around a different construct, but the Java platform -- from language, through libraries and bytecode, and all the way to the VM and its tooling interfaces -- was designed around the idea that sequential composition occurs by sequencing operations on a single thread. And virtual threads are just Java threads.
• fulafel 3y ago
  You could easily have a million threads if you use multi-kb stacks. Million times multi-kb means multi-gb, that's still 3-4 orders of magnitude less than big memory servers/VMs. (and 1 order of magnitude less than a normal laptop) What do you mean by using a file handle, is this a Windows platform thing? On *ix, threads don't use up file descriptors (but you can still have a million fd's at least on linux for other stuff if you want).
  • thfuran 3y ago
    But "multi-kb" in this context probably actually means about 1MB.
    • fulafel 3y ago
      What do you base this on? The stacks and kernel bookkeeping shouldn't use nearly this much at least on linux. Keep in mind that thread stacks have are lazily allocated virtual memory so won't use as much physical memory as the thread stack size setting shows. If these threads are handling TCP connections and L7 protocol processing on top, you're going to have nontrivial both kernel and userspace memory usage per connection too that may dwarf the thread overhead. Here's a linux kernel dev (Ingo Molnar) benchmarking Linux in 2002 and starting just shy of 400k threads in 4 GB: https://lkml.iu.edu/hypermail/linux/kernel/0209.2/1153.html https://lkml.iu.edu/hypermail/linux/kernel/0209.2/1153.html - though on a 32 bit systems lots of objects things are 50% the size compared to current 64 bit. But still gives you a ballpark.
  • zmmmmm 3y ago
    > On *ix, threads don't use up file descriptors Thanks - this caused me to dig into the specific scenario where creating threads was exhausting file handles in my experience and you are correct - consuming a file handle is indeed not intrinsic to creating a new thread in Linux. It's insanely easy for literally anything you do with the thread to consume a file handle, but of course, that applies to virtual threads as well. Thanks!