5 ms·
> it's like Twisted is contagious. I can't quite put my finger on it. I can. Twisted is viral (not in a good way always). Once you start using Twisted you ar
by rdtsc 13y ago
> it's like Twisted is contagious. I can't quite put my finger on it.
I can. Twisted is viral (not in a good way always). Once you start using Twisted you are doomed and stuck in a parallel reality of Twisted-only libraries forever. I have used Twisted professionally for 4+ years, this is not something I just read on a blog.
You first say "Oh, I know look at the cute example of parsing a line protocol with it, let my try that. That was easy, I should adopt it". And then it got you. You want to fetch data from a particular database. That's easy, github has a project in Python that does it. Oh but you need it to return a deferred now. You can't simply make a connection and block because it blocks the reactor. This library is no good, you either have to pray someone who knows Twisted wrote a Twisted version of it, or you have to write your own.
As you pointed out, you can use code to combine threaded and deferred code but it is not fun or easy. Make sure you think twice before adopting it.
BTW the best way to learn it is to use it. Don't read about it. Reading about it won't help much. It is one of those subjects, like riding a bike if you don't know how. Once you get it you get it and it becomes easy. What is not easy is library fragmentation.
You have been down-voted and perhaps I'll be too for my negativity. But I am worried about infatuation with Twisted and Promises in recent (last year and on) Python design emails and moving them more to be part of the standard library.
I think this is a not healthy move for Python and a step forward. I'll speak my mind and if you feel this is too negative, just downvote, I'll understand, but I think it has to be said. They are copying what they are seeing in other language hint: node.js (Javascript). I feel there is a bit of jealousy, Python is maturing, its last exciting thing was 3.0. PyPy has been relegated to the magic academic research corner (unfairly I think). But, node.js and Javascript on the server is not successful because it has a callback style of concurrency. There are other reasons. So copying that aspect won't make Python more appealing. Heck, Python had an awesome callback library for ages -- Twisted, when Javascript was still use to make buttons flash and Java applets were the hot thing. Did it take off? Somewhat, but not really. It was famously hard to grok. The freshest breath of air so far in practical Python use on the server is using greenlet. But that also has been pushed to sort of "ah it is dirty, we don't like, how will you ever know if your code switches co-routines". Well it is not perfect but it is most practical and performant to date. I wish that became part of core Python. I wish they copied more from Go, Rust or Erlang rather than a callback style concurrency with some extra sauce on top. A
Anyway, ending my rant here, and hoping I didn't upset too many people already.
- ekimekim 13y agoAgreed. I make heavy use of greenlet/gevent, and one of the primary advantages is that you can import and use naive sequential libraries, and they won't block your process (assuming they're pure python so the monkey patching works properly). As for "not knowing when you might switch coroutine" - I contend that if it matters, you're doing it wrong. Note that, unlike proper threading, things can't happen at the same time / interleaved with simple statements, so it's still very easy to do atomic actions without special locking - eg. there's no way self.x += 1 could switch in the middle (unless someone is badly abusing the __add__ method).
- lvh 13y agoThe problem is still that other coroutines can mess with your shared state, and you have no queues as to where they get to do that. Neither Twisted nor tulip has that problem. The fact that it can't happen in __iadd__ is just one place where it can't happen. That's a far cry from not having to care about this problem.
- rdtsc 13y ago> The problem is still that other coroutines can mess with your shared state, I think you mis-understood his point. The point is if you treat your green threads like regular threads and you properly release and acquire locks ("with" contexts help here too) then it won't happen. The other way is to use queues for example. In most small examples, say updating a shared dictionary or list, many won't bother serializing green thread access to it. That should work fine. > The problem is still that other co-routines can mess with your shared state, and you have no queues as to where they get to do that. Neither Twisted nor tulip has that problem. greenlet based green threads dispatch underneath just like Twisted's reactor does based on a select/poll/epoll/kqueue system call. Like it is possible to update half the items in a dictionary, then call a function that does IO from a green thread, and another green thread gets dispatched and now updates the dictionary, so, now one thread sees a logically inconsistent shared piece of data. Now replace the above threads with callbacks. You have a TCP listening socket (is it called a factory or protocol, I forgot my terminology already) that dispatches callbacks when data arrives. In those callbacks you update some part of the dictionary then realize you need data from a data base so you make a database call, get a deferred and add a callback to continue updating when data arrives back from database. In the meantime your original socket fires and another callback goes to your shared dictionary and now sees it in an inconsistent state. This thing exists, I have used it a few number of times: https://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.DeferredSemaphore.html https://twistedmatrix.com/documents/8.2.0/api/twisted.intern... It is a semaphore in Twisted. It is also used to reduce concurrency but it can be used to protecte shared data if needed. And I did need it in a couple of places. This myth that callback based programming frees one from the need to use locks and somehow it magically infuses code with unicorns and easy shortcuts is very common. The only way to dispell it to use the source. Just read the source of twisted and see how it works. Look at greenlet and eventlet's hub's source (or gevent) and see how it works. Yes, greenlet + monkey patching libraries can hide IO dispatching points your code. It is a valid downside. So to be safe you an just treat them as regular threads. But it allows you to use libraries -- one of the biggest reason one would pick Python for. Now that we are on the subject, what is one sane way to drastically reduce the need for locks and semaphores? Use actors like Erlang does. A class with a thread and a queue attached to it. Then avoid sharing data instead send messages. You'll be hit with a serious performance issue because of memory copying, because well, there is no free lunch. Twisted and other callback mechanisms are certainly not a free lunch either. > ... and you have no queues as to where they get to do that. Heh heh, an accidental funny typo. As actually if you did have queues, you might not have to worry about shared data access ;-)