4 ms·

> For example, the simplest way to write a network server that can handle more than one client at a time is to fork a new thread for every client. This seems m

by vemv 5y ago

> For example, the simplest way to write a network server that can handle more than one client at a time is to fork a new thread for every client. This seems much debatable. For me the simplest way is creating a thread-per-core pool, and have clients integrate with said pool via a queue. It's a well-understood paradigm that has worked for a long time. I get the sense of abstraction of having a "thread" per client, but is it really the simplest choice if it needed creating a multi-year project like Loom first? A precedent that comes to mind is Clojure core.async. It pursues the same sense of abstraction. But does one gain much if the implementation has its own quirks, flaws, etc? Any given abstraction is one bug away from leaking. So, it's well plausible that the same can happen with Loom - more stuff to learn, more underlying complexity, bigger surface area for bugs. It all seems too gratuitous too me, so my high-bar question/questioning remains :)

gravypod 5y ago
All implementations of in-process thread multiplexing is sort of like a garbage collector. You have a resource, you manage the allocation of a resource to something that consume it, etc. A lot of house keeping and some overhead code that needs to execute before any work is done (each time). In my mind, it makes sense to have a single, well tested, and abstracted implementation that all other systems can use. You can see a need for this at least in the Android space: RxJava, Kotlin coroutines, etc. Also, in a user-managed thread multiplexing system you can't guarantee "fair" scheduling. If you have some DOS bugs in your server you can lock up an entire serving thread. A language-level multiplexing system will not have these same kinds of edge cases. There are many places this sort of thing becomes helpful. We could also see concurrent processing utilities that could be very helpful. See this for more info on how this could lead to cleaner programs: https://www.youtube.com/watch?v=f6kdp27TYZs https://www.youtube.com/watch?v=f6kdp27TYZs
hilbertseries 5y ago
The value of loom, or fibers is essentially in creating a thread abstraction that performs comparably to an event loop. A common pattern in Java is to spin up X threads to perform a blocking network operation. Or to have X threads each with a connection to a client. If you want a web server to process 50 requests concurrently, then having one request serving thread per core won’t cut it. These approaches of spinning up lots of Java threads are fine, until they aren’t, when you have too many threads. Loom is not a panacea and doesn’t make any new thing possible, it makes certain things easier. And it still will have its limitations.
- vemv 5y ago
  An informed thread-based approach is quite more careful than just spinning "lots of threads" and can certainly achieve a concurrency of 50, or 10000. Let's say we want a concurrency of 1000 and have 8 CPUs available. I'd have small thread pools, one per logical kind of task. Maybe this would result in 20 threads total for the whole JVM. Concurrency would be represented by the items being held in BlockingQueues of max size 1000. Items would be transferred from thread to thread by communicating via said queues (pipeline pattern). CPU utilization would be practically ideal as there's always a thread having some kind of work at hand. And the system would be actually concurrent since the queues can have interleaved workloads (e.g. work for users #1, #2, #1 again, #3, etc). The max queue size allows to communicate backpressure to e.g. a load balancer.
  - hilbertseries 5y ago
    I don’t really follow the way you define terms here. Concurrently means processing at the same time, if work is held in a queue it’s not being processed. If you reduce the number of threads to 1 and leave the queue at 10k. You’re claiming that this is concurrently processing 10k messages, but you only have one thread. Is the idea here that you’ve reimplemented scheduling on top of the jvm? This is exactly what project loom is trying to protect the developer from having to do.
    - vemv 5y ago
      As long as there's more than 1 thread processing the queue, the queue is being concurrently processed. Particularly, the queue items will be picked up in a nondeterministic order, driven by CPU utilization. > Is the idea here that you’ve reimplemented scheduling on top of the jvm? No, I'm implementing a fairly vanilla pipeline pattern, just with simpler, more explicit blocks than those used by async frameworks/abstractions. IIRC it's described well in the JCIP book.
      - closeparen 5y ago
        It sounds like you're talking about a CPU-heavy workload. In this context often it's an IO-bound workload; the service makes upstream requests to a database and other services. Most of the request lifetime is waiting for responses. To get reasonably many requests started and waiting on those responses, either you need to write in async/callback oriented style, or you need a mechanism to make many thousands of naive, blocking routines coexist efficiently.
kaba0 5y ago
If a given request does use blocking calls, it will “steal” a valuable place in the thread pool until that IO call returns. So you either have to write in a reactive style or live with decreases max throughput. Reactive style has plenty of warts with non-intuitive debugging, error handling, etc. These will be solved by loom.
pron 5y ago
> It's a well-understood paradigm that has worked for a long time. The situation is quite simple, and governed by Little's law. If your number of threads is greater than the average request rate times the average processing latency (of processing a request entirely on a single thread), then you're all good and it's a well-understood paradigm that's worked for many years. If, on the other hand, that number of threads is smaller than that multiple, then that well-understood paradigm is well-understood to have crashed servers for many years, requiring many more of them even when their utilisation is low. If you're on one side of this, then you don't have a problem to begin with. Loom exists because increasingly more people are on the other side of this problem, where they need multiple servers.

by vemv 5y ago

gravypod 5y ago

All implementations of in-process thread multiplexing is sort of like a garbage collector. You have a resource, you manage the allocation of a resource to something that consume it, etc. A lot of house keeping and some overhead code that needs to execute before any work is done (each time). In my mind, it makes sense to have a single, well tested, and abstracted implementation that all other systems can use. You can see a need for this at least in the Android space: RxJava, Kotlin coroutines, etc. Also, in a user-managed thread multiplexing system you can't guarantee "fair" scheduling. If you have some DOS bugs in your server you can lock up an entire serving thread. A language-level multiplexing system will not have these same kinds of edge cases. There are many places this sort of thing becomes helpful. We could also see concurrent processing utilities that could be very helpful. See this for more info on how this could lead to cleaner programs: https://www.youtube.com/watch?v=f6kdp27TYZs https://www.youtube.com/watch?v=f6kdp27TYZs

hilbertseries 5y ago

The value of loom, or fibers is essentially in creating a thread abstraction that performs comparably to an event loop. A common pattern in Java is to spin up X threads to perform a blocking network operation. Or to have X threads each with a connection to a client. If you want a web server to process 50 requests concurrently, then having one request serving thread per core won’t cut it. These approaches of spinning up lots of Java threads are fine, until they aren’t, when you have too many threads. Loom is not a panacea and doesn’t make any new thing possible, it makes certain things easier. And it still will have its limitations.

vemv 5y ago
An informed thread-based approach is quite more careful than just spinning "lots of threads" and can certainly achieve a concurrency of 50, or 10000. Let's say we want a concurrency of 1000 and have 8 CPUs available. I'd have small thread pools, one per logical kind of task. Maybe this would result in 20 threads total for the whole JVM. Concurrency would be represented by the items being held in BlockingQueues of max size 1000. Items would be transferred from thread to thread by communicating via said queues (pipeline pattern). CPU utilization would be practically ideal as there's always a thread having some kind of work at hand. And the system would be actually concurrent since the queues can have interleaved workloads (e.g. work for users #1, #2, #1 again, #3, etc). The max queue size allows to communicate backpressure to e.g. a load balancer.
- hilbertseries 5y ago
  I don’t really follow the way you define terms here. Concurrently means processing at the same time, if work is held in a queue it’s not being processed. If you reduce the number of threads to 1 and leave the queue at 10k. You’re claiming that this is concurrently processing 10k messages, but you only have one thread. Is the idea here that you’ve reimplemented scheduling on top of the jvm? This is exactly what project loom is trying to protect the developer from having to do.
  - vemv 5y ago
    As long as there's more than 1 thread processing the queue, the queue is being concurrently processed. Particularly, the queue items will be picked up in a nondeterministic order, driven by CPU utilization. > Is the idea here that you’ve reimplemented scheduling on top of the jvm? No, I'm implementing a fairly vanilla pipeline pattern, just with simpler, more explicit blocks than those used by async frameworks/abstractions. IIRC it's described well in the JCIP book.
    - closeparen 5y ago
      It sounds like you're talking about a CPU-heavy workload. In this context often it's an IO-bound workload; the service makes upstream requests to a database and other services. Most of the request lifetime is waiting for responses. To get reasonably many requests started and waiting on those responses, either you need to write in async/callback oriented style, or you need a mechanism to make many thousands of naive, blocking routines coexist efficiently.

kaba0 5y ago

If a given request does use blocking calls, it will “steal” a valuable place in the thread pool until that IO call returns. So you either have to write in a reactive style or live with decreases max throughput. Reactive style has plenty of warts with non-intuitive debugging, error handling, etc. These will be solved by loom.

pron 5y ago

> It's a well-understood paradigm that has worked for a long time. The situation is quite simple, and governed by Little's law. If your number of threads is greater than the average request rate times the average processing latency (of processing a request entirely on a single thread), then you're all good and it's a well-understood paradigm that's worked for many years. If, on the other hand, that number of threads is smaller than that multiple, then that well-understood paradigm is well-understood to have crashed servers for many years, requiring many more of them even when their utilisation is low. If you're on one side of this, then you don't have a problem to begin with. Loom exists because increasingly more people are on the other side of this problem, where they need multiple servers.