3 ms·
The viral nature of twisted is not limited to twisted and shows up in lots of places. It's actually a consequence of the problem that "concurrency is Hard". T
by adamtj 13y ago
The viral nature of twisted is not limited to twisted and shows up in lots of places. It's actually a consequence of the problem that "concurrency is Hard".
There is a tradeoff to be made between pre-emptive and cooperative multitasking. Every platform must choose one or the other or a mix of both, and then deal with the consequences. Threads, of course, have a well-known class of very dangerous bugs that make using them difficult. Processes make pre-emptive multitasking much safer by erecting certain barriers, but at the cost of being much more tedious.
These days, callbacks, events and other forms of cooperative multitasking are the new shiny. They avoid the big problems of threads and multiple processes, but have some problems of their own. First and simplest is that you can only use one CPU per process without mixing in threads or multiple processes. Second, if something doesn't cooperate, it blocks everything else. You have to be very careful about when and where you yield the CPU. Because of that, everything must be event-driven or use callbacks, or you risk trashing your throughput and/or latency.
Threads have a similar problem: if you use them, you're stuck using thread-safe libraries. It's not as bad as the cooperative case though. The thread-safety of a library can be independant of the code that uses it. In the cooperative case, you don't have that independence. You're not just stuck with cooperative-safe libraries. You are restricted to events or callbacks that are compatible with the cooperative framework used in your application. If your application uses events, then the library must use events instead of callbacks, and it must use the same types of events to plug into your event loop. If not, then you need a compatibility layer. Some cooperative languages reduce the problem by mandating how events and/or callbacks work for all libraries. Python doesn't have that, which where the viral nature of twisted comes from. That's why you're talking about twisted-compatible libraries and not just python libraries, and why this a non-issue with node.js database libraries.
edit: wording
- rdtsc 13y ago> Threads have a similar problem: if you use them, you're stuck using thread-safe libraries. One pattern that I like personally is the actor pattern. Building systems out of independent isolated concurrent components that share data _mostly_ by sending messages. Erlang at the top of this paradigm but one can replicate that in other languages and libraries. There is Akka for Scala and others as well. It is basically an object, a thread running one of its methods and a queue. This object also has an address or handle. Objects communicate by sending messages to each other's handles. Now of course as you say one needs to use libraries that allow this concurrency. So far I find this pattern best matches real world scenarios. Real world is built of many concurrent pieces but each individual piece could be a small sequential island. This doesn't apply just to intelligent agents but to server requests as well. Each client connection and context can be sequential but it can run concurrently with other potential client context. Ideally if one such client request crashes it shouldn't crash the main OS process. If one such client goes into a tight loop it shouldn't block the rest of the requests. This is hard to do without sacrificing heavily by copying data and or using isolated heaps (can use real OS processes up to some point, but also it is very resource heavy). > That's why you're talking about twisted-compatible libraries and not just python libraries, and why this a non-issue with node.js database libraries. Very good point. I didn't think of it that way. Say if Python was born with Twisted bolted on to start we wouldn't be having this problem. But it is interesting how a mono-culture of concurrency is a good thing in certain respects. And to give Guido props I think he is trying to bring that in -- as with Python, it is desirable to have one best way to do the same thing. Concurrency wasn't like that historically. And he is trying to fix it. I personally disagree with the approach he took (either in promoting or looking at deferred or explicit co-routines based on generators)