4 ms·

It's not hype really - it's me getting fed up with authors arguing that their software solves every problem. Redis is very good at being data structures store.

by mdomans 10y ago

It's not hype really - it's me getting fed up with authors arguing that their software solves every problem. Redis is very good at being data structures store. In regard to Python - generators make sense, coroutines, much less. That's my opinion, but over the span of 10 years most programmers I met professionally either never used coroutines outside of pet projects or use them sparingly. I'd actually like to see a big project with coroutines implemented using the async/await.

btilly 10y ago
I disbelieve. Take working in Tornado. I've been writing in Tornado, and all I've needed to do is decorate functions with @gen.coroutine then use generator syntax. All the detailed plumbing to implement the event loop is done for me, and I never interact with it directly. Nor would there be any value in my doing so. It needs to be implemented once, and implemented well. Once that is done there is little to no value in messing with it. You'd just be creating the opportunity for disaster with little to no corresponding value. I actually experienced this exact disaster. In this project, Tornado has to interact with Redis. So I used tornado-redis which tried to be implemented directly. Unfortunately it was not well implemented, and one coroutine could easily get a message meant for another. Untangling the mess promised to be a lot of work. However it was literally the work of minutes to switch my class to the synchronous Redis library, write a ThreadPoolExecutor wrapper for my class, and have Tornado interact with that. It would be slightly better if the native version had worked, but this got 95% of the benefit for under 1% of the work. You could describe this as "using coroutines sparingly". I'd describe it as using them sensibly. Coroutines are a useful bit of plumbing to enable a programming style, but their flexibility is also a burden. You should use them to build a sensible abstraction, then use that. For another example of this, look at Scheme. Scheme internally implements continuations, and function calls are just a special case of that. Continuations are, of course, a building block for coroutines, generators, and many other programming constructs. However a sensible Scheme programmer just implements functions. They may be using a project that somewhere does coroutines, continuations, and all sorts of other fun stuff. But the day to day code you write doesn't do that. There is no need to add the mental overhead from using the construct all of the time "just because it is there".
- mdomans 10y ago
  To a certain (high) degree I agree. First of all, when I'm wrote this article by coroutines I meant specifically Python async/await pattern which I don't find easy to use or read. And I agree on Tornado. Over the years I've seen many APIs that abstract away stuff like implementation details of cooperation yet are in fact coroutines - e.g. goroutines in Go, tasklets in Stackless. Even Erlangs processes are, on the VM level, cooperatively scheduled green threads. For me there are 3 problems when you design concurrency APIs. One is performance and language internals. Second is API that you expose to the end programmer. Third problem is what you can achieve as a programmer using those API based on how integrated concurrency is into the language. All the examples of higher level platforms I know are good for highly concurrent apps: Erlang, Go, Node.js, GCD - they were designed and throughly integrated into the language. In that context threading in Python really feels bolted on. And for something different: seeing how some people reacted I'm diving deep to make 2-3 articles of tour-de-concurrency. What do you think?

by mdomans 10y ago

btilly 10y ago

I disbelieve. Take working in Tornado. I've been writing in Tornado, and all I've needed to do is decorate functions with @gen.coroutine then use generator syntax. All the detailed plumbing to implement the event loop is done for me, and I never interact with it directly. Nor would there be any value in my doing so. It needs to be implemented once, and implemented well. Once that is done there is little to no value in messing with it. You'd just be creating the opportunity for disaster with little to no corresponding value. I actually experienced this exact disaster. In this project, Tornado has to interact with Redis. So I used tornado-redis which tried to be implemented directly. Unfortunately it was not well implemented, and one coroutine could easily get a message meant for another. Untangling the mess promised to be a lot of work. However it was literally the work of minutes to switch my class to the synchronous Redis library, write a ThreadPoolExecutor wrapper for my class, and have Tornado interact with that. It would be slightly better if the native version had worked, but this got 95% of the benefit for under 1% of the work. You could describe this as "using coroutines sparingly". I'd describe it as using them sensibly. Coroutines are a useful bit of plumbing to enable a programming style, but their flexibility is also a burden. You should use them to build a sensible abstraction, then use that. For another example of this, look at Scheme. Scheme internally implements continuations, and function calls are just a special case of that. Continuations are, of course, a building block for coroutines, generators, and many other programming constructs. However a sensible Scheme programmer just implements functions. They may be using a project that somewhere does coroutines, continuations, and all sorts of other fun stuff. But the day to day code you write doesn't do that. There is no need to add the mental overhead from using the construct all of the time "just because it is there".

mdomans 10y ago
To a certain (high) degree I agree. First of all, when I'm wrote this article by coroutines I meant specifically Python async/await pattern which I don't find easy to use or read. And I agree on Tornado. Over the years I've seen many APIs that abstract away stuff like implementation details of cooperation yet are in fact coroutines - e.g. goroutines in Go, tasklets in Stackless. Even Erlangs processes are, on the VM level, cooperatively scheduled green threads. For me there are 3 problems when you design concurrency APIs. One is performance and language internals. Second is API that you expose to the end programmer. Third problem is what you can achieve as a programmer using those API based on how integrated concurrency is into the language. All the examples of higher level platforms I know are good for highly concurrent apps: Erlang, Go, Node.js, GCD - they were designed and throughly integrated into the language. In that context threading in Python really feels bolted on. And for something different: seeing how some people reacted I'm diving deep to make 2-3 articles of tour-de-concurrency. What do you think?