7 ms·

The problem that's described here - "green" threads being CPU bound for too long and causing other requests to time out is one that is common to anything that u

• notyourday 7y ago
  > The problem that's described here - "green" threads being CPU bound for too long and causing other requests to time out is one that is common to anything that uses an event loop and is not unique to gevent. node.js also suffers from this. Isn't node.js IO native threaded?
  • orisho 7y ago
    I don't know, but I was actually discussing time when the thread is performing pure CPU operations and not doing any IO. In this case the JS code does not yield to the event loop, so any other requests waiting will stall. This is actually an important property of JavaScript that allows it to work without locks (as long as you do not await, your code runs completely synchronously and will not be interrupted). It's not a flaw, but definitely requires awareness. Gevent code that doesn't spawn more than one native thread (the one that runs the event loop) also has this property - you don't need locks as long as you do not perform IO. In Python's case it can be more tricky, as you might end up yielding to the event loop by indirectly performing IO when logging or something similar. In JavaScript, the only case, AFAIK, where this can happen is when you await. Nothing else will cause you to yield to the event loop.
    • notyourday 7y ago
      Ok, so it is opposite in nodejs? There's a single JS thread but as soon as it performs any async IO ( which is the only IO that one should be performing in node ), that IO creates a thread servicing that IO request. In addition to that, if one is crazy enough to run long computational operations there's setTimeout(), setImmediate() and process.nextTick() and friends. Though I would say it doing any real compute work inline rather than farming it out to workers is a boneheaded idea regardless of the language used.
      • orisho 7y ago
        AFAIK there is only ever just one thread (that runs user code). setTimeout, etc are all also invoked by the event loop. I can vaguely recall (but it's 1AM so I didn't verify - take with a grain of salt) that nodejs uses an a thread pool for IO, but I guess that's only for IO for which there is no async API, otherwise that would be wasteful. I imagine a quick search about the event loop and said thread pool would yield better researched answers :)
      • bernawil 7y ago
        Nodejs is single process, single thread and single core. Chrome used to run different tabs as threads of a process but that ended since Spectre and Meltdown. Back to single process since then. async IO creates a thread servicing that IO request Nope. The event is a single FIFO queue implemented with libuv. It differentiates between sync and async function and basically goes around the queue running synchronous functions to completion every time it find one and async ones for a set time before switching to next one.
        • notyourday 7y ago
          > Nodejs is single process, single thread and single core. ps -eLf disagrees: $ ps -eLf|grep node deploynode 16438 16173 16438 0 11 00:21 pts/6 00:00:01 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16439 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16440 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16441 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16442 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16443 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16444 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16445 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16446 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16447 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16438 16173 16448 0 11 00:21 pts/6 00:00:00 /opt/bin/node node-le-manager-api 3550 deploynode 16579 8353 16579 0 1 00:27 pts/4 00:00:00 grep node
  • crimsonalucard 7y ago
    The implementation of libuv under nodejs (and the state of the art python async/await) actually uses a total of three threads. It's a complicated model and it's not even strictly correct to call it an "event loop." I don't completely understand it and very few people do. From a high level you can think of it as a single thread that can only context switch during an IO call. Otherwise the function has to run and block all the way to termination. <---- This is what rachel is complaining about.
    • dvdgsng 7y ago
      Any good reads on that at hand?
      • dvdgsng 7y ago
        Ok, libuv docs seem just fine. Nevermind :)
  • Rapzid 7y ago
    There are Linux system calls in wide use that are blocking. Both node.js and Golang use native threads to make the system calls and "park and wait" for the response. The runtimes then take the information from the call and incorporate it back into the user code threads(the one user thread in node.js).
• rossmohax 7y ago
  Beam VM (Erlang/Elixir) is another example which copes with the problem very well due to preemption.
  • Floegipoky 7y ago
    To expand on this, by allowing preemption on function calls and simply not providing a loop mechanism, you can guarantee an upper bound on how long a process may need to wait until it can be scheduled.
    • cbsmith 7y ago
      Guarantee is a strong term. If you have an infinite number of processes to running, there aren't guarantees on how long until it is scheduled. Similarly, with cooperative multi-tasking as in Gevent, you can manipulate scheduling to try to provide better guarantees about wait times. It's just... you can't ignore the problem.
• crimsonalucard 7y ago
  In NodeJS and python "async/awaits" you don't have to deal with deadlocks and mutexes and other ugly concurrency primitives/problems. The reduction in complexity of the program is huge which is something rachel completely discounts.
  • crimsonalucard 7y ago
    So in another reply some guy pulled off a deadlock in node by creating his own external lock primitive outside of the node stdlib which in itself doesn't contain a single lock primitive. Technically, I'm wrong, it can happen, but only if you go out of your way to do it: See the conversation here: https://news.ycombinator.com/item?id=22515100 https://news.ycombinator.com/item?id=22515100
  • Gibbon1 7y ago
    I'm not sure discounts. I mostly do firmware in that environment you can use either preemptive or cooperative multitasking. That latter is much less likely to result in corrupted state and results in smaller code size. Which is why I like it. But you pay with latency and task starvation problems as Rachel complains of.
• CameronNemo 7y ago
  Can't a CPU bound task just make itself preemptible in Python by calling some dummy async function, like asyncio.sleep(0)?
  • coleifer 7y ago
    Yes, `gevent.sleep()` or `time.sleep()` (if you monkey-patch) will yield control back to the scheduler.
  • anaphor 7y ago
    Doesn't that mean you have to somehow know when it's been running too long and then yield back control somehow? What if the slow operation is something that's atomic, like multiplying some huge numbers?
    • CameronNemo 7y ago
      Well the example given is decoding JSON. If that is happening in a long loop, you can yield once per loop and be safe. Not all problems are neatly broken apart like that, but in those cases how much of a chance does the server have to not timeout regardless, you know? Note that once per loop might be too often, but you can just measure how long a loop run typically takes and compare to how soon you want to preempt the task, then yield at the right interval.
      • emj 7y ago
        Seems like abstractions will bite you there, most will just do cool_library.unmarshall(request) in some variant, those libraries will not have the same method for yielding as you have.
        • TeMPOraL 7y ago
          Abstractions are meant to be broken! One could probably work around this problem by adding new functions to cool_library, or modifying existing ones, whose code would be copy-pasted from the library but with some asyncio.sleep(0) calls spliced in strategic places :). For legacy projects, it may make more sense to cheat like this than to rewrite the whole project in a saner tech stack.
• jashmatthews 7y ago
  > You cannot spin up 100K native threads to handle 100K requests that are IO bound (which is very common in a microservice architecture, since requests will very quickly block on requests to other services). Or well, you can, but the native thread context switch overhead is very quickly going to grind the machine to a halt as you grow. Why? Bigger MySQL servers regularly use 10k native pthreads which spend most of their time waiting on IO. Context switch overhead isn’t linear with the number of threads! The Linux scheduler runs O(logN). Moving from 10k MySQL threads to 1 thread per core will give you less than 10% extra throughput. Doing IO means context switching. Goroutines don’t get a free pass here.
  • orisho 7y ago
    There are other costs to regular context switching as opposed to goroutines/greenlets (the green threads that gevent uses). I don't remember the details but specific attention was paid to the point of making context switching and other resource consumption by these green threads cheaper than native threads, so I suggest reading about it in greenlet/Golang docs :) You can also try searching for C10K which was the term people used to discuss how to achieve 10K connections, and is often associated with cooperative threading. For example, the cost of the context switch itself (storing all registers) is more significant with native threads. Just try spinning up 100K threads that each print a line and then sleep for 10ms, and see how high your CPU usage gets. Also, doing IO does not necessarily mean context switching - it means calling into the kernel (system calls). If you use an async IO operation (read/write from a socket) and then continue to the next thread, by the time you're done with all ready threads, you're likely to have some sockets ready to read from again, so you might not context switch at all. Kernel developers are working on even reducing the need for syscalls with io_uring, which is designed to allow you to perform IO without system calls.
    • fulafel 7y ago
      There shouldn't be any 100k hard limit for threads at least in Linux, though you need enough memory for 100k stacks of course. You need to increase some default limits for it though (https://stackoverflow.com/a/26190804 https://stackoverflow.com/a/26190804) Assuming a generous(?) 20 kB per thread in stack and other corresponding OS bookkeeping inforation you could have 1k threads in 20 MB, or 1M threads in 20 GB. Doing 100 Hz timer wakeups and IOs concurrently in 100k threads makes 10 M wakeups/second, that takes a chunk of CPU independent of green / native threads choice. Performance vs kernel threads will depend on the green threads implementation.
• adev_ 7y ago
  > You cannot spin up 100K native > threads to handle 100K requests that are IO bound (which is very common in a microservice architecture, since requests will very quickly block on requests to other services) The Varnish server does that all the time and it is not a problem. Modern OS handle a large number of threads much better than 10 years ago. In case of python however, you can not do that due to the GIL and the fact python handle native threading terribly.
  • mborch 7y ago
    From the docs: > We rarely recommend running with more than 5000 threads. If you seem to need more than 5000 threads, it’s very likely that there is something not quite right about your setup, and you should investigate elsewhere before you increase the maximum value.
    • adev_ 7y ago
      > We rarely recommend running with more than 5000 threads. If you seem to need more than 5000 threads, it’s very likely that there is something not quite right about your setup, and you should investigate elsewhere before you increase the maximum value. Yes, and still 5000 will be more than enough to serve most of production load. Even on high traffic websites.
• delusional 7y ago
  As long as my 100k threads are all waiting on io the kernel doesn't need to switch to them. I get that once you reach full utelization, the best the kernel can do is to make it fair, and that's going to cost. That's not a general statement like "100k threads grinds the machine to a halt".
• akvadrako 7y ago
  Actually Linux can handle a few million threads just fine - you only need to increase the max pid. You should just update your reasoning and say you can’t have 100M threads waiting for blocking IO.
• williamDafoe 7y ago
  I have worked on a system that spins up 50k threads on a CPU with 72 threads and it works just fine, it's the google web page fetching system. If that didn't work then Google wouldn't fetch web pages. At all.