8 ms·

> The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. It is inten

by j2labs 15y ago

> The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. It is intended to create an environment similar to preemptive threads, yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. > - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. This is pure speculation. If Python's multicore handling was better (existed?), it's likely gevent would be implemented similar to other systems where you have threads allocated to each core and the concurrency primitives can then be executed on cores according to some algorithm. > - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. While this is true, I don't think it is likely to occur. Gevent's creator agrees: http://www.gevent.org/intro.html#cooperative-multitasking http://www.gevent.org/intro.html#cooperative-multitasking > On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is. You're an Erlang programmer. You have to have that opinion. (sausagefeet and I are friends IRL so I know his biases)

sausagefeet 15y ago
> Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. The weakness of callbacks is the lack of multicore support. It is not pure speculation that gevent simply won't work in a preemptive environment if it wants to utilize multiple cores. The sentence right about this makes my point: > yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Why is that? It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. You can depend on 'my_dict[foo] += 1' to happen atomically. If you want to utilize multiple cores with gevent, as code using it is written now, you can't because all of the code out there depend on operations between I/O events happening atomically. This isn't a secret in the callback/gevent world. On the contrary, it's presented as a selling point. No longer worry about those locking primitives those real threading libraries give you because we are single threaded. (Ocaml/Lwt has the same problem). > While this is true, I don't think it is likely to occur I'll concede it's far less likely. I'm curious how large gevent code bases are in the wild, a large application that one cannot fit in their entire head could very well have a lot of the same race conditions that a preemptively threaded application has. > You're an Erlang programmer. You have to have that opinion Really? I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.
- j2labs 15y ago
  > The weakness of callbacks is the lack of multicore support. You suggest that gevent won't work in preemptive environment if it wants to support multiple cores, alas I'm still not convinced as to why. The greenlets are cooperatively scheduled coroutines, as we know, but with proper multicore support I don't see why they couldn't adapted to be preemptively scheduled. If we're imagining that Python now has multicore support, we can also imagine greenlets being updated to work with it. Assuming gevent would receive no modifications to work with such an important update to Python is why I accused you of speculating. > It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. That's correct. They are cooperatively scheduled coroutines. If someone wants to context switch, they can write code that does it or wait for I/O events. The lack of an explicit yield statement does not prevent coroutines from switching. In fact, greenlets are quite powerful here, more powerful than what's offered by Twisted and Tornado, because they don't have to yield directly to the caller. You could imitate CPS which is impossible with the explicit context switch model. I am not disagreeing with you on this point, btw, just trying to be more explicit for other readers about the cooperatively scheduled nature. Interested readers might want to check this out: http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing http://en.wikipedia.org/wiki/Computer_multitasking#Cooperati... > I'm curious how large gevent code bases are in the wild Well, eventlet itself was written to support Second Life and sits behind multiple big video games. Gevent is in production use for many large companies, most notably Spotify. They are moving to gevent from twisted for performance and cleaner code. > I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought. My statement is facetious, but the reality is that Erlang got concurrency right and Python is trying to glue concurrency ideas into the language as an afterthought. If Python didn't have it's lousy GIL we wouldn't have to pull such tricks to work concurrency into a single threaded system. Alas, it has the GIL. Now, to conclude some of my thoughts, I believe gevent is an excellent way to build web systems. The coroutines are short-lived, exist to fetch some data with an easy nonblocking interface and then concat some strings together to produce the web page / API output. If folks were building CPU bound systems it would straight-up be a mistake to use Python for that. Ocaml could be a better choice, alas, as you say, it has the same issue with a GIL though the computations would execute a LOT faster.

by j2labs 15y ago

sausagefeet 15y ago

> Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. The weakness of callbacks is the lack of multicore support. It is not pure speculation that gevent simply won't work in a preemptive environment if it wants to utilize multiple cores. The sentence right about this makes my point: > yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Why is that? It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. You can depend on 'my_dict[foo] += 1' to happen atomically. If you want to utilize multiple cores with gevent, as code using it is written now, you can't because all of the code out there depend on operations between I/O events happening atomically. This isn't a secret in the callback/gevent world. On the contrary, it's presented as a selling point. No longer worry about those locking primitives those real threading libraries give you because we are single threaded. (Ocaml/Lwt has the same problem). > While this is true, I don't think it is likely to occur I'll concede it's far less likely. I'm curious how large gevent code bases are in the wild, a large application that one cannot fit in their entire head could very well have a lot of the same race conditions that a preemptively threaded application has. > You're an Erlang programmer. You have to have that opinion Really? I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.

j2labs 15y ago
> The weakness of callbacks is the lack of multicore support. You suggest that gevent won't work in preemptive environment if it wants to support multiple cores, alas I'm still not convinced as to why. The greenlets are cooperatively scheduled coroutines, as we know, but with proper multicore support I don't see why they couldn't adapted to be preemptively scheduled. If we're imagining that Python now has multicore support, we can also imagine greenlets being updated to work with it. Assuming gevent would receive no modifications to work with such an important update to Python is why I accused you of speculating. > It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. That's correct. They are cooperatively scheduled coroutines. If someone wants to context switch, they can write code that does it or wait for I/O events. The lack of an explicit yield statement does not prevent coroutines from switching. In fact, greenlets are quite powerful here, more powerful than what's offered by Twisted and Tornado, because they don't have to yield directly to the caller. You could imitate CPS which is impossible with the explicit context switch model. I am not disagreeing with you on this point, btw, just trying to be more explicit for other readers about the cooperatively scheduled nature. Interested readers might want to check this out: http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing http://en.wikipedia.org/wiki/Computer_multitasking#Cooperati... > I'm curious how large gevent code bases are in the wild Well, eventlet itself was written to support Second Life and sits behind multiple big video games. Gevent is in production use for many large companies, most notably Spotify. They are moving to gevent from twisted for performance and cleaner code. > I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought. My statement is facetious, but the reality is that Erlang got concurrency right and Python is trying to glue concurrency ideas into the language as an afterthought. If Python didn't have it's lousy GIL we wouldn't have to pull such tricks to work concurrency into a single threaded system. Alas, it has the GIL. Now, to conclude some of my thoughts, I believe gevent is an excellent way to build web systems. The coroutines are short-lived, exist to fetch some data with an easy nonblocking interface and then concat some strings together to produce the web page / API output. If folks were building CPU bound systems it would straight-up be a mistake to use Python for that. Ocaml could be a better choice, alas, as you say, it has the same issue with a GIL though the computations would execute a LOT faster.