3 ms·
> The weakness of callbacks is the lack of multicore support. You suggest that gevent won't work in preemptive environment if it wants to support multiple core
by j2labs 15y ago
> The weakness of callbacks is the lack of multicore support.
You suggest that gevent won't work in preemptive environment if it wants to support multiple cores, alas I'm still not convinced as to why.
The greenlets are cooperatively scheduled coroutines, as we know, but with proper multicore support I don't see why they couldn't adapted to be preemptively scheduled. If we're imagining that Python now has multicore support, we can also imagine greenlets being updated to work with it. Assuming gevent would receive no modifications to work with such an important update to Python is why I accused you of speculating.
> It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen.
That's correct. They are cooperatively scheduled coroutines. If someone wants to context switch, they can write code that does it or wait for I/O events. The lack of an explicit yield statement does not prevent coroutines from switching. In fact, greenlets are quite powerful here, more powerful than what's offered by Twisted and Tornado, because they don't have to yield directly to the caller. You could imitate CPS which is impossible with the explicit context switch model.
I am not disagreeing with you on this point, btw, just trying to be more explicit for other readers about the cooperatively scheduled nature.
Interested readers might want to check this out: http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing http://en.wikipedia.org/wiki/Computer_multitasking#Cooperati...
> I'm curious how large gevent code bases are in the wild
Well, eventlet itself was written to support Second Life and sits behind multiple big video games.
Gevent is in production use for many large companies, most notably Spotify. They are moving to gevent from twisted for performance and cleaner code.
> I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.
My statement is facetious, but the reality is that Erlang got concurrency right and Python is trying to glue concurrency ideas into the language as an afterthought. If Python didn't have it's lousy GIL we wouldn't have to pull such tricks to work concurrency into a single threaded system. Alas, it has the GIL.
Now, to conclude some of my thoughts, I believe gevent is an excellent way to build web systems. The coroutines are short-lived, exist to fetch some data with an easy nonblocking interface and then concat some strings together to produce the web page / API output.
If folks were building CPU bound systems it would straight-up be a mistake to use Python for that. Ocaml could be a better choice, alas, as you say, it has the same issue with a GIL though the computations would execute a LOT faster.
- sausagefeet 15y agoAll gevent code written today assumes that between I/O events all actions will happen atomically with respect to the rest of the program. That means you can safely do any multi-step operation on a shared resource you want without issue. The second you want to run 2 gevent threads in parallel this guarantee goes out the window. Gevent could be modified for multicore suppor if Python got it, but that still breaks every piece of gevent code written today. This is the same problem Twisted and Tornado have.
- j2labs 15y agoGevent wasn't designed for multicore because it doesn't exist today. It's fair to assume some work would have to be done to continue using it safely. Edit: Solve the problems you actually have.
- sausagefeet 15y agoAnd that's exactly my point, the entire framework doesn't scale to multicore without a lot of work, just like callbacks. Which was my initial claim.
- j2labs 15y agoThat's entirely dependent on what the work being done is. It's typical for gevent users to use the built-in queue systems to pass messages between coroutines, similar to Erlang's actor model. I do t believe a lot of work would be required if gevent users stuck to this style, as gevents docs suggest.
- sausagefeet 15y agoThe guarantees of the framework no longer exist when you want to do things in parallel. Sure, not every line of code using gevent would break from this, but that's not the point. Even people using queues are often still playing with shared resources, which would be unsafe in multicore still.