7 ms·
There is opinionated, and then there is stupidly opinionated. This article crosses that line then stays there. It offers one useful point, but that one is har
by btilly 10y ago
There is opinionated, and then there is stupidly opinionated. This article crosses that line then stays there. It offers one useful point, but that one is hardly news.
For example we have claims like this, Do people use coroutines? Yes, but not in production code. This claim is trivially false. There is lots of production code that has the word "yield" in it. Heck, plenty of production code uses Tornado, which at its heart is nothing BUT coroutines. And most programmers have no trouble reading it.
Moving on we see, Also, cooperative concurrency aka coroutines was abandoned a fair amount of time ago. I wonder why? I'm sure that people who use Redis and Node.js would be surprised to discover that cooperative concurrency has no place in our modern software world.
He even concludes that people don't need concurrency. Which is obviously ridiculous - just look at any distributed system.
But let's go back to why people move from cooperative concurrency to preemptive. Answer it will make it clear why we'll always have cooperative concurrency in the programming mix.
Cooperative concurrency means that you don't need locks and don't have race conditions. However one poorly programmed routine can block your whole system. Preemptive concurrency means that a bad routine can no longer block your whole system, but now you have opened up the guarantee of complex race conditions. Programmers are really, really bad at understanding race conditions. So bad that almost all of our software has them. So bad that people who write automated code verification tools have learned the hard way to not report on them because programmers won't understand what you are talking about. (Sadly true. In http://web.stanford.edu/~engler/BLOC-coverity.pdf http://web.stanford.edu/~engler/BLOC-coverity.pdf look for the phrase, "...for many years we gave up on checkers that flagged concurrency errors.")
Therefore if you want concurrency with no locking overhead, and no race conditions, then coroutines are great. But once your application becomes big and messy enough, it becomes better to accept the problems that come with preemption.
Now if you're going to preempt, there is a ton of advice to be offered on how to do it well. For example you'll generally have fewer problems with coarse-grained processes than with threads. Queues and worker pools are a great solution for some types of problem, and I first did that about 15 years ago. It was hardly an original idea when I first did it - it was suggested to me by someone who had been using that technique since the 1980s. And it wasn't original with him either.
However there are a ton of use cases that this doesn't cover. For example it doesn't do much to help you handle high volumes of traffic to your webserver, and it won't let you create a distributed data store for low-latency access to terabytes of data. So put celery in your tool bag, know what it is good for, and don't make the mistake of thinking that it solves all problems.
- mdomans 10y agoI personally quite dislike Redis, which is highly unconcurrent. And don't get me started on Node.js. Of course people do use coroutines - I simple don't think they're a good idea. I think they're a bad idea. They're neither readable nor introduce better performance. And yes, I really don't think people think they need concurrency - they just want this_A and this_B to happen ASAP and with no blocking. On your explanation of coroutines and preemptive - I agree, though I consider both equally bad. The point I was making that Celery runs lockless - and that'd be an interesting idea to back port to Python. And I know queues are old idea. What I like about them is the ability to get rid of "locks everywhere". An example of what I consider good concurrency model is Grand Central Dispatch with atomic ops running via queues.
- btilly 10y agoThis is your response to having some of your glaring mistakes pointed out? You ignore having been proven wrong, then proceed to state your personal opinions as if people should care? I have news for you. Redis and Node.js are widely used because they are able to solve real problems that real people have. They work in practice despite your personal opinions to the contrary. Moving on, you're dead wrong about concurrency. How many programs are running at once on your computer? While you are loading a web page, do you want your browser to continue to be able to respond to you? Those are two simple cases where people want concurrency. Those are also two cases that Celery doesn't handle. At one point they would have been written with cooperative multi-tasking. Today these are generally written with preemptive instead. (Though there are some exceptions. PuTTY is a fun one. That implemented coroutines in C with an impressive preprocessor hack. See http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html for details.) And back to Celery, things that don't require locking, can and should get by without locking. This is a useful observation, but hardly a universally applicable one. And waving the magic Celery want doesn't change that. If I have 500 files to process, and that data needs to wind up nicely summarized in a database, Celery will only let me get rid of locking in my code because I rely on locking being implemented in the database. If you care about data and race conditions, at some point you have to solve hard problems. And you'll need to lock. But only do it when you need to.