13 ms·

Celery, non-blocking code and quest against coroutines

129 points by mdomans 10y ago

denom 10y ago
Celery's pretty nice, I've used it in several projects with rabbitMQ and it's useful for IO bound workloads. The main drawback with it is the overhead of dispatching the job and waiting for the result. With threads, there's no overhead of starting up a separate process. That said, I'd recommend celery to anyone where latency is not a problem.
- kerkeslager 10y ago
  Could you explain what you mean by "The main drawback with it is the overhead of dispatching the job and waiting for the result."?
  - denom 10y ago
    Well there are a couple drawbacks. The author is describing celery as a solution to the GIL thread lock problem. However the time it takes to dispatch the job[1] and fire up the worker is much greater than the time it takes to create a thread. So while celery is _a_ solution to the problem, it's not an equivalent solution. Just something to be aware of. The type of computation you're modeling is limited by the async/dispatched nature of celery. Threads can communicate data structures to each other. With celery, that's awkward with the latency and async semantics. [1] I've used rabbitmq to dispatch jobs over to celery in the past, here's a good overview of latency in that process. TLDR requests that hit the wire are on the order of milliseconds: https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/ https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory...
    - kerkeslager 10y ago
      I guess I'm just not sure what people are expecting here. The point of Celery isn't to parallelize and synchronize, it's to fire off a task, probably on an entirely different machine, and forget about it. If you're trying to sync up afterward, you're going to run into problems because it's not designed for that.
  - crucialfelix 10y ago
    The overhead is messaging to rabbit, having that create ids for it and queue it; then celery pulls the next job from the exchange, unpacks the job description with its arguments, and then calls the function (wrapped in exception handlers etc. etc.) Celery is suitable for "send an email" or "rebuild a value and put in memcache" but not suitable for "when the OS finally returns the file handle I requested, could you read the file into memory please" Totally different time scales and use-cases.
    - kerkeslager 10y ago
      I guess what I'm confused about is the "waiting for the result" in the parent comment. Celery is pretty much entirely fire-and-forget, and you're usually firing to an entirely different server. If this is being used to achieve parallelization instead of asynchronous processing, then it's no wonder that the parent is having trouble, because they're using Celery for a task for which it's wholly inappropriate.
      - crucialfelix 10y ago
        Agreed. There are a lot of bells and whistles with celery and I've tried them and now avoid them all. I don't nest or chain tasks. I don't use periodic tasks because they can pile up badly and block everything. I have my own simple loop that calls the most urgent periodic-task. I built an angular button that would call a task and would show spinning in-progress and then show when the task completes. Works on development, didn't work on production. why why why ? well, no time to fiddle with celery so I moved on. over the years I've come to mistrust its complexity.
        kerkeslager 10y ago
        > I don't nest or chain tasks. Yeah, I think nested/chained tasks are pointless. The point of celery is to offload the task to an asychronous worker (probably on another machine) so if you're already on the worker there's no point sending another task to the same worker--it doesn't achieve anything above doing the task inline. > I don't use periodic tasks because they can pile up badly and block everything. This sounds like you're either running the periodic task too frequently, or you need more workers. I've seen a few cases where people tried running a 20-minute task every second on a single worker--of course this is going to pile up. Periodic tasks are among the most predictable, so they're easiest to prevent from piling up: simply make sure that the following relation holds true: w > t / p w = number of workers t = how long it takes for the task to run p = the period (a task that runs ever 20 minutes has period 20 minutes) The > should be greater than by an order of magnitude to be safe in case t increases.
- brianwawok 10y ago
  I think the idea is for many things you don't sit around and wait for Celery to complete. User clicks "send email"? You throw a send email task in Celery. If it takes .1 seconds or 12 seconds, the user doesn't have to wait for it, because they don't care. This is perfect Celery use. Now when you are loading a page to display to the user, and that page makes a bunch of celery tasks that you NEED To complete.. this is more iffy.. in most cases, you may just want to run those inline in the webrequest. Needing a bunch of tasks to complete to show a user a result is not an ideal use case of Celery.
MollyR 10y ago
Its an interesting use case the author solved with celery. I've used celery with a few django and flask projects. It's nice, but it added a lot of overhead for me. So I've used http://flower.readthedocs.io/en/latest/ http://flower.readthedocs.io/en/latest/ to help handle the monitoring. For some really small sites and very specific use cases, I actually wrote some small custom helper scripts to just have cron jobs. If you are using django, I found this library helpful. https://bitbucket.org/wnielson/django-chronograph/ https://bitbucket.org/wnielson/django-chronograph/
- brianwawok 10y ago
  The biggest overhead is really setting up RabbitMQ right? Because Django and Celery play nice, you can just deploy one codebase and run 1 Django process and 1 Celery + Celery Beat process pretty easy... Bummer you lose so many Celery features when you use Amazon or Google Cloud Queues instead of RabbitMQ....
  - mdomans 10y ago
    AFAIK you can have RabitMQ SaaS these days. And yes, RabbitMQ is very good.
    - brianwawok 10y ago
      cloudamqp? I guess, not terribly cheap though.
      - mdomans 10y ago
        I think there are a few other options. But deploying RabbitMQ isn't that hard. I can actually write something about that. Is Linode cheap enough for you?
        brianwawok 10y ago
        I mean last time I needed RabbitMQ I just used https://github.com/Mayeu/ansible-playbook-rabbitmq https://github.com/Mayeu/ansible-playbook-rabbitmq and it took 3 minutes. But I hear a lot of people saying not to use Celery because RabbitMQ is hard.. maybe people just have different standards for hard?
forgottenpass 10y ago
Design patterns are just that, design patterns. With upsides, downsides, suitable use cases and unsuitable use cases. I was originally going through pointing out all the nonsense that was outright wrong or just ruffled my feathers, but this whole article is punching above it's weight. And boils down to the following: The author doesn't like using concurrent programing techniques in a single-threaded process. The author does like a job queue that manages task execution across multiple processes. This has fuck all to do with the GIL beyond the fact the presence of the GIL might have contributed to the popularity of the former in python and that celery uses multiprocessing over multithreading. Designing a piece of software is more than cargo-culting the latest fads. Who cares if asyncio exists? Yes, we "really need" it to the extent that people want to design their programs like that. No, you don't "really need" to use it.
- markbnj 10y ago
  The title is off I agree, it's not like Celery fixes cpython's problematic data structures. I don't think it was a bad article, though.
  - forgottenpass 10y ago
    I wouldn't have minded reading someones process of learning a new design pattern if it wasn't asserting so many things that are silly or flat out wrong during the process. That's why I originally wrote, "this is a bad article." I edified that bit out because it detracts from my point unless I take the time to go through and fisk the article.
    - mdomans 10y ago
      Well, I'm more about advertising the lockless programming pattern :) Point out the silly, please, I like it when someone actually makes honest effort to teach me something.
      - scott_s 10y ago
        The title, mainly. Most readers will take that to mean that you fixed the GIL itself - that is, removed it - rather than are presenting a pattern that avoids it. Your title sets up an adversarial relationship with such readers, because their attitude when they start is going to be "no, you didn't", which will cause them to miss your actual message.
aduitsis 10y ago
I have used Gearman for a long time and I like it a lot, very simple to use, lightweight, does its job and doesn't get in the way. It's one of the few cases where something literally just works. If one would just want a job queue to fanout jobs in a network transparent fashion, would moving to Celery provide any advantages? With regard to Celery's broker, which is most preferable, Redis or RabbitMQ?
- mdomans 10y ago
  RabbitMQ - it's simply better, I can elaborate but it'd take an article on it's own. Unfortunately I never really used Gearman in high load environment, so can't comment on advantages of Celery over Gearman
omni 10y ago
> In the last 10 years I've never met a case where someone pointed out that "this code needs to be concurrent, but single threaded". I wish I could have slept through the circa 2012 Node.js hype cycle like this person clearly did.
- omni 10y ago
  To follow up on my own snark: > Do people use coroutines? Yes, but not in production code. Now I'm wondering if I'm the one who's confused. Either the author has also never heard of Go or goroutines don't meet his/her definition of coroutines.
  - mdomans 10y ago
    I know of goroutines and comparing Python's coroutines to goroutines is like saying that Porsche and Tata car's are same, since both have 4 wheels.
    - omni 10y ago
      Fair enough, I thought you were speaking more generally since the next sentence was: > I may be opinionated, but I've done concurrency in many languages and never ever have I seen anything less readable than coroutines.
  - weberc2 10y ago
    I'm guessing it's the latter, and perhaps not without cause. Traditionally, coroutines require the programmer to explicitly yield. With Go, you don't have to think about that much; you can pretty much treat them as lightweight threads and trust the scheduler to do the right thing. Goroutines aren't pre-empted, but the yield points aren't explicit (though they are well defined and thus easy to reason about).
    - jerf 10y ago
      "Traditionally, coroutines require the programmer to explicitly yield." When terms move from academia into common usage, they inevitably blur. Academia had a specific definition of coroutine that includes that. Programmers now use "coroutines" in a way that I have a hard time distinguishing from "threads", Pythonic "generators", and in the worst cases "something vaguely concurrent". Any given author may have a clear idea of what they mean (and in particular, I'm not making any specific accusations about the author of this piece) but the term is rapidly approaching uselessness in the general programming community.
sametmax 10y ago
The TL;DR (tl;dr: this explains how Celery obsoletes GIL and coroutines) is only telling half of the story. Celery solves some cases for Python: - background tasks; - tasks queues; - cron tasks. And it's nice, I used it extensively. But in NO WAY it totally solves the GIL problem nor does it render coroutines obsolete. Here are some celery limits: - you need to calibrate the number of worker you have to match your workload. - You are limited to x number of blocking operations, where x is the number of workers. It means you cannot do massively parallel I/O such as network operations with it (e.g: a web server). - Tasks don't have access to your main process memory, and vice versa. - Tasks cannot communicate with each others; - You must juggle with the workflow of your tasks (is it ready ? it it dead ?). You can use await stuff() with a try/except; - Celery is an additional process to setup and start, with backends to choose from and tuning to do. It's a lot more work than just importing an async lib. Granted, celery is a very useful piece of software, but not the silver bullet this article depicts.
- mdomans 10y ago
  On Celery limits: - celery has autoscale which works _awesome_ - autoscale? also, having bounds isn't such a bad idea in reality :) In regard to what you've written about tasks - yes, and point I was trying to make is that Celery model is a good example of what can be done to enable concurrent programming in Python
- deleted 10y ago
  [deleted]
- asksol 10y ago
  > - You are limited to x number of blocking operations, where x >is the number of workers. It means you cannot do massively >parallel I/O such as network operations with it (e.g: a web >server). Celery can use eventlet/gevent instead of multiprocessing for executing tasks, so this should be possible (granted, not sure if using it as a web server is a great idea) >- Tasks cannot communicate with each others; This is not true, they can send messages to each other >- You must juggle with the workflow of your tasks (is it ready >? it it dead ?). You can use await stuff() with a try/except; If you have to juggle it means your workflow design is not good enough. I.e. you shouldn't wait for other tasks, you should have callbacks and errbacks. Also, note that a try/except does not guarantee the operation will be completed (e.g your asyncio app can be killed). I'm not sure there's much point in comparing these, they are wildly different concepts: Celery is a distributed system, asyncio is for async I/O, the GIL is only a problem if you can't start n instances of your app.
garethrees 10y ago
The article makes claims that are way too broad: "fixed Python's GIL problem", "obsoletes GIL and coroutines". Maybe the claims are reasonable when addressing the author's particular use case, but they are not true in general and phrasing them in this way seems designed to provoke. It's important to be clear about which use cases are addressed by a given system. For example, if you have compute-bound tasks that operate on non-shared data then it makes a lot of sense to distribute them via a multiprocessing system like Celery. But if you have I/O-bound tasks that operate on shared data then it doesn't — implementing the necessary locking and communication is very hard. In this case, coroutines are simple to program and reliable, and because you're I/O-bound you're not losing anything by being single-threaded.
- mdomans 10y ago
  Here's the deal, most tasks I've seen are both CPU and I/O bound, only in different part of the same task. Of course you can very aggressively optimise the design, but sometimes it becomes so unreadable, the cost you incur through complication makes it not worth the effort. What I'm trying to advocate is that the distinction of CPU bound and I/O bound makes sense in CS classroom, but most of the tasks in production code are a mix of both. Therefore, the distinction is really between blocking and non-blocking, and Celery enables "almost readable" non-blocking coding
  - garethrees 10y ago
    Thinking about where the computation is bound is only one half of the space of use cases: you also have to consider the shared/non-shared axis. There's a 2x2 matrix of task types: non-shared shared compute-bound multiprocessing ??? I/O-bound either coroutines I put ??? in the upper right because as far as I know Python doesn't have any general solutions here. Also, if "most tasks you've seen" are compute-bound at some points, then of course it makes sense to use Celery! But you should bear in mind that other people may have seen other kinds of task.
    - mdomans 10y ago
armitron 10y ago
Terrible post, worst I've read today by far. Not only does the author completely NOT understand why the GIL is a problem, but he also misunderstands asynchronicity which is hilarious given his recommendations.
- mdomans 10y ago
  And why GIL is the problem, in your opinion? Honest question, explain your point of view.
  - jerf 10y ago
    In a nutshell, if your thesis was true, it would be true. If Celery was the solution to the GIL, nobody would still be talking about the GIL. Celery may be slick and well-developed, but sharing stuff out to processes has been well-established tech for a long time. If that were all people wanted, the problem would be solved. Since it's not, people must want different things. It's great that you solved your problems with it. Many other people have too. Many others have problems that are less amenable to this approach.
    - mdomans 10y ago
      Assuming that people will stop talking about something because it's not a problem or it isn't real is a bit ... uninformed :) The main point I was trying to make is that if you can force the programmer to guarantee safety, you don't need to rewrite GIL. And approach Celery enforces you to use, can be, with some extra syntax sugar, back ported to Python to have nice threading API
      - jerf 10y ago
        Very well, then I'll also point out that among the people still talking about the GIL are those who know enough about Python to successfully take steps to remove it, which is pretty much the highest possible level of knowledge of Python you can ask for. This is probably a topic "in the air" precisely because of the story of someone's most recent attempt/implementation of this a couple of days ago. It is not the case that only "people who don't really know Python" are talking about the GIL and considering the problem unsolved. If you were right, these people would not be talking about it. I would strongly, but politely, suggest that you should consider the possibility that there either things you don't know about the issue or problems that others have that you don't. It's not like Celery is the only obscure library nobody's ever heard of that has this basic approach; "multiprocessing" is built into the standard library now, for instance.
zzzeek 10y ago
> this explains how Celery obsoletes [emphasis mine] GIL and coroutines > Celery is a whole project and there's tons of coordination before you fire your first task - it's true. well that's why it's not really something that "obsoletes" threading (or your fancy-pants you-kids-today concurrency system of choice). Geez can people no longer write without hyperbole as a default position?
- coldtea 10y ago
  >Geez can people no longer write without hyperbole as a default position? Oh, the irony. (One or a few articles are hyperbolic != "people can no longer write without hyperbole as the default")
  - JustSomeNobody 10y ago
    One or a few? Come on. Most articles to day are this way. And programming today is hyperbole induced cargo-cultism if it's anything at all.
JustSomeNobody 10y ago
Python doesn't have a GIL "problem". Just stahp.
- mdomans 10y ago
  Have you seen a program of any Python conference :D
  - JustSomeNobody 10y ago
    So? Calling it a "problem" is crap. The GIL was a decision that has repercussions on certain types of programming, sure, but it's not a "problem".
    - mdomans 10y ago
      It is for the community. Personally I consider GIL to be an awesome idea and it'd work if we went with a concurrency model that avoids implicit locking
FunctionalGuy 10y ago
Or you can use a language like Erlang which has first-class support for this sort of architecture.
TheGuyWhoCodes 10y ago
Celery is great and works well but it definitely doesn't fix the GIL problem. Having shared memory is a must for some workloads, or can save you a ton of money, especially when taking about caching. Sometimes caching per process isn't feasible but caching for all threads is, even using multi processing isn't good enough because you can only use the main process memory if it's simple arrays.
- mdomans 10y ago
  Talking about shared memory benefits and overheads in Python is a pretty abstract problem, if you account for how much objects Python interpreter creates and what OS does under certain load levels. I'm merely advocating a different model for concurrency in Python :)
sciurus 10y ago
Celery boils down to a nice abstraction over forking another process to do your work. You still have the scalability problems that might make you want to write concurrent single-threaded code, just at a different layer of your stack. Take the author's example of scraping 1000 webpages. Let's say their computer running celery has enough memory to run 50 celery processes. This means they can request 50 web pages concurrently. Most of the time those celery processes are going to be sitting idle, waiting on the remote web host to respond. This is terribly inefficient compared to coroutines with asyncio/aiohttp, or even using a thread pool since urllib will release the GIL and let another thread run when it's blocked on network i/o. You could have a single process performing the work just as fast as 50 celery processes.
- asksol 10y ago
  Celery can use async I/O too! Using the gevent pool you can have e.g. 4 celery worker instances with 1000 threads each.
  - sciurus 10y ago
    Right, but as you mention elsewhere in the thread there are caveats, like discussed at http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html#concurrency-eventlet http://docs.celeryproject.org/en/latest/userguide/concurrenc... . At this point the blog post author's contention that celery is an better alternative to doing async i/o breaks down.
dikaiosune 10y ago
I recently opted to use Celery for a similar-ish task to the author's update_metrics job. Having generally been OK with Python's single-threaded performance and simplicity, I was not particularly pleased with a solution that involved an additional backend service (RabbitMQ), separate logfiles, fighting with the Python module system during setup, and ultimately a lot less available documentation than for items which live in the standard library. All of this just to kick off a background process. I haven't had my coffee yet and am feeling a bit grumpy this morning, but I'm really not sure I would be willing to use it again for a project where I had to manage the code and the deployment. Regardless of my mood, celery definitely isn't a panacea for these problems -- it just happens to ameliorate ones where there's already a larger deployment to manage.
wacowz 10y ago
Every time I read an article like this, all I can think is, this is what Elixir and Erlang are built for. I love Python and use it on a daily basis, but developing concurrent systems in Python is beginning to look like a questionable choice, while there exists tools better suited for this. Celery works great, and the integration with django is superb. However, it's consistently been a memory hog for us. Hell, we had to write a shell script to regularly check for memory use by celery and take necessary precautionary steps.
- brianwawok 10y ago
  What level of concurrency is required though? 99.9% of web request only needs concurrency at the user level, which standard python + fork will do just nicely. Making a realtime system to crunch some massive numbers in 50 micros? Then you might want some C or Java concurrency. But much of the world is not coded at that level.
  - wacowz 10y ago
    It's often the case that some work needs to be done outside of the request/response lifecycle, eg external api calls, data aggregation, sending notifications etc. Most of it is essentially IO bound. We happen to have lots of such tasks and therefore use celery, but there's only so much celery can do, given it's concurrency level is bound by the number of cores available since it uses multiprocessing. An actor based approach or sth like Go's goroutines would be a better fit.
- bitwize 10y ago
  Developing concurrent systems in Python on Unix is a questionable choice. On Windows it's a doddle, because Windows supports true async I/O. See: http://pyparallel.org http://pyparallel.org
  - wacowz 10y ago
    The numbers for pyparallel are quite impressive.
cyberpanther 10y ago
Celery does solve the GIL problem and in some use cases is better suited even if the GIL wasn't a problem. However, it is an additional "system" that you have to add to a code base. If you're writing a small program that is easily distributed and needs "non-blocking" features, Celery brings a whole host of extra requirements. This becomes additionally problematic for Python when other languages implement these "non-blocking" features into the language. Thus you can implement these things in a few lines of code, where with Python you must learn a new framework and have a complicated install to use a simple feature. If Python wants to compete with other languages in the future it will have to keep up with these "non-blocking" features without a huge system overhead expense. I see this same problem with Django Channels which lets you implement web sockets and such with Django. Django Channels relies on Redis and solves more problems than just web sockets. However, if I just want a web socket, doing it in Django Channels produces too much overhead. Writing a web socket in Tornado can be done in few lines of code without the extra "system" overhead.
seomis 10y ago
Why is it that Python's single-threadedness is always singled out over every other language in the general family? Ruby, PHP, Perl, and JS are all single-threaded (Perl ithreads and JRuby/Jython caveats notwithstanding). Celery itself is just a framework for fork management, something that has been available to these languages since their inception.
- nawitus 10y ago
  JavaScript's "logic" may be single-threaded, but it's asynchronous event loop solves a lot of problems that a single-threaded synchronous model has (like what Python has by default).
  - seomis 10y ago
    Asynchronous event loops aren't a core part of the JavaScript: They're from the DOM API and Node standard library. Though it is significant that nothing comparable exists for Python that has reached the same robustness and support.
- Rafert 10y ago
  And as a Rails dev I was thinking: "hey, somebody is taking a shot at Python's single-threadedness for a change". I'm not familiar with Celery, I gather it is similar to Sidekiq/Que/Resque/DelayedJob/etc?
fideloper 10y ago
So if I'm reading this correctly, the author is using a queue system (Celery, presumably with a backend such as RabbitMQ?) to run background jobs, but then waiting for them all to complete before finishing a task. In effect, you sorta get concurrency (assuming multiple workers are processing jobs). Or does Celery also provide a way to do threading without using a backend service (e.g RabbitMQ, beanstalkd, sqs)? Basically, I'm confused.
- brianwawok 10y ago
  You could totally do concurrency through Reddis or something super lightweight right on your webserver... could make sense. Really seems better to throw your task back to the backend pool though, and let a web request finish.
asksol 10y ago
Author of Celery here (not the article). Celery solves a different problem: it's a distributed system that will help you run your tasks on multiple machines, not an alternative to asyncio/twisted/tornado etc. The GIL is often easily worked around by starting N instances of your app, but that doesn't work for all applications (games, video, audio). Celery won't help you in those cases, but you can write C extensions. 99.99% of the time you shouldn't even consider using posix threads in Python, as most libraries (including popular ones) are not thread-safe, resulting in spending precious time fixing tricky bugs. Celery is also not in contrast to coroutines, actually it's quite common to use Celery as a distributed layer on top of async I/O frameworks (top tip: routing CPU-bound tasks to a prefork worker means they will not block your event loop). Another thing, the article says `scrape_url.subtask(args=(url,)),` is not very readable, but the idiomatic way to write this is: `scrape_url.s(url)` (yup we have a one letter method name, Django has Q we have .s). See more examples here: http://docs.celeryproject.org/en/latest/userguide/canvas.html http://docs.celeryproject.org/en/latest/userguide/canvas.htm...
- brianwawok 10y ago
  TIL about the s method.. Thanks for the tip!
- enjo 10y ago
  Nothing useful to add other than thanks for the great library. I've been using Celery for years, it's made my life demonstrably better.
- mdomans 10y ago
  In all honesty .subtask() is more readable for me than .s() Also, I don't want Celery to replace asyncio and/or threading. I was trying to point out that the programming model can be back ported to Python as a way of approaching concurrency, which, without having to support "locks everywhere" is simpler to implement with GIL
  - asksol 10y ago
    I think it's more readable in isolation, but when you have workflows it distracts you from what it's actually doing, making it harder to see what the purpose is at a glance. An API can probably not have many shortcuts like this, but you learn what .s does once, and then you barely notice it. But that's my opinion and I have come to learn not everybody value succinctness in code, so you have the choice of both :o) Had I not been constrained by backward compatibility I may have made it so that task(arg) only defines the signature of a task invocation and you'd need to do task(arg).delay() to call it remotely, and task(args)() to call it as a function locally.
abotsis 10y ago
To all the haters: the GIL is just a constraint that can be worked around. Clever ways of starting more than one interpreter is a solid way to do that. If you can't think of a way to do that, then maybe you shouldn't be working on distributed systems. For example, I wrote a high-speed distributed database load tool, a perfect python use-case. It did some transforms, but nothing heavy. The database (Vertica) could generally keep up. For large files (multiple TB) I'd divide the file into a number of chunks and each interpreter would just seek into an offset, read to EOL, then load until it reached the next interpreter's offset. I built it in a few hours. Just trying to say that there are very few real-world workloads (were python is extremely useful) that the GIL really gets in the way. It certainly doesn't offset the time and ease of developing a solution.
- dang 10y ago
  > To all the haters This is a form of name-calling, which the HN guidelines ask you not to do in arguments. And the "if you can't think of... maybe you shouldn't be" is also a jab. Please don't do these things in HN comments. They add no information, just provocation. Consider how much better your otherwise fine comment would be without them. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html
  - ixtli 10y ago
    I worry that you would not have posted this response if the top comment had said "Dear dissenters,". However I think you're right about the extraneous and specious "maybe you shouldn't be working in distributed systems" claim.
    - scott_s 10y ago
      Perhaps not, but for good reason: "dear haters" and "dear dissenters" mean different things. As does, "To everyone who disagrees, please consider ..." The word "haters" is combative.
  - 10y ago
methodover 10y ago
Celery is neat. Something that sort of bothers me with it though is -- and this is maybe totally something that is my fault, I just can't figure it out -- sometimes workers will just stop processing things. For unknown reasons. I've set up some alerts to automatically slack me when this happens, so I can go in and see why they're stuck and restart them. Still haven't been able to figure it out. Happens maybe once every month or two.
- harel 10y ago
  Put a hard time out on the worker and it will be killed and restarted if hanging for X period of time.
  - jvoorhis 10y ago
    In my experience, celery takes some hand holding to stay alive. I generally configure it to recycle workers after N tasks, in addition to a hard timeout on tasks. I've also found in my configuration, using RabbitMQ as the bus, workers can become non-responsive after receiving too large a message.
  - methodover 10y ago
    Yeah, I've thought about it. I guess I've been holding out hope that I could figure out the underlying cause. What's wierd is, it always affects just one type of worker -- the workers that handle sending an email. And it happens on all workers simultaneously, on all servers, of that type. Just that type. Meh. I should just put a harakiri timeout like you suggest.
ploxiln 10y ago
> Do people use coroutines? Yes, but not in production code. Are you serious? Well, first, surely in all this talk of cooperative threading, you also include callback-oriented event-based programming. This is HUGE. Have you heard of nginx? But also, I've personally worked on very high volume api servicing and analytics, largely written in python using tornado (which is very similar to twisted) for a very popular web service: Bitly. There were also a couple significant components written in c using libevent, another similar event-ed i/o mini-framework, involving lots of callbacks. It can be done right and be readable and debug-able, and it can be super efficient, more efficient than spawning threads for hundreds or thousands of parallel i/o tasks (or managing a thread pool or forking processes or whatever). It's not for everything but it's definitely in serious production environments. The more serious, the more likely to use event-ed i/o.
- mdomans 10y ago
  Cooperative concurrency and event-based programming are different concepts. You need events for the coroutine since scheduler needs to base trampolining of something. Alas, you can do event based programming without coroutines. Yes, I used libevent and libev and I do love nginx :)
deleted 10y ago
[deleted]
btilly 10y ago
There is opinionated, and then there is stupidly opinionated. This article crosses that line then stays there. It offers one useful point, but that one is hardly news. For example we have claims like this, Do people use coroutines? Yes, but not in production code. This claim is trivially false. There is lots of production code that has the word "yield" in it. Heck, plenty of production code uses Tornado, which at its heart is nothing BUT coroutines. And most programmers have no trouble reading it. Moving on we see, Also, cooperative concurrency aka coroutines was abandoned a fair amount of time ago. I wonder why? I'm sure that people who use Redis and Node.js would be surprised to discover that cooperative concurrency has no place in our modern software world. He even concludes that people don't need concurrency. Which is obviously ridiculous - just look at any distributed system. But let's go back to why people move from cooperative concurrency to preemptive. Answer it will make it clear why we'll always have cooperative concurrency in the programming mix. Cooperative concurrency means that you don't need locks and don't have race conditions. However one poorly programmed routine can block your whole system. Preemptive concurrency means that a bad routine can no longer block your whole system, but now you have opened up the guarantee of complex race conditions. Programmers are really, really bad at understanding race conditions. So bad that almost all of our software has them. So bad that people who write automated code verification tools have learned the hard way to not report on them because programmers won't understand what you are talking about. (Sadly true. In http://web.stanford.edu/~engler/BLOC-coverity.pdf http://web.stanford.edu/~engler/BLOC-coverity.pdf look for the phrase, "...for many years we gave up on checkers that flagged concurrency errors.") Therefore if you want concurrency with no locking overhead, and no race conditions, then coroutines are great. But once your application becomes big and messy enough, it becomes better to accept the problems that come with preemption. Now if you're going to preempt, there is a ton of advice to be offered on how to do it well. For example you'll generally have fewer problems with coarse-grained processes than with threads. Queues and worker pools are a great solution for some types of problem, and I first did that about 15 years ago. It was hardly an original idea when I first did it - it was suggested to me by someone who had been using that technique since the 1980s. And it wasn't original with him either. However there are a ton of use cases that this doesn't cover. For example it doesn't do much to help you handle high volumes of traffic to your webserver, and it won't let you create a distributed data store for low-latency access to terabytes of data. So put celery in your tool bag, know what it is good for, and don't make the mistake of thinking that it solves all problems.

129 points by mdomans 10y ago

denom 10y ago

Celery's pretty nice, I've used it in several projects with rabbitMQ and it's useful for IO bound workloads. The main drawback with it is the overhead of dispatching the job and waiting for the result. With threads, there's no overhead of starting up a separate process. That said, I'd recommend celery to anyone where latency is not a problem.

kerkeslager 10y ago
Could you explain what you mean by "The main drawback with it is the overhead of dispatching the job and waiting for the result."?
- denom 10y ago
  Well there are a couple drawbacks. The author is describing celery as a solution to the GIL thread lock problem. However the time it takes to dispatch the job[1] and fire up the worker is much greater than the time it takes to create a thread. So while celery is _a_ solution to the problem, it's not an equivalent solution. Just something to be aware of. The type of computation you're modeling is limited by the async/dispatched nature of celery. Threads can communicate data structures to each other. With celery, that's awkward with the latency and async semantics. [1] I've used rabbitmq to dispatch jobs over to celery in the past, here's a good overview of latency in that process. TLDR requests that hit the wire are on the order of milliseconds: https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/ https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory...
  - kerkeslager 10y ago
    I guess I'm just not sure what people are expecting here. The point of Celery isn't to parallelize and synchronize, it's to fire off a task, probably on an entirely different machine, and forget about it. If you're trying to sync up afterward, you're going to run into problems because it's not designed for that.
- crucialfelix 10y ago
  The overhead is messaging to rabbit, having that create ids for it and queue it; then celery pulls the next job from the exchange, unpacks the job description with its arguments, and then calls the function (wrapped in exception handlers etc. etc.) Celery is suitable for "send an email" or "rebuild a value and put in memcache" but not suitable for "when the OS finally returns the file handle I requested, could you read the file into memory please" Totally different time scales and use-cases.
  - kerkeslager 10y ago
    I guess what I'm confused about is the "waiting for the result" in the parent comment. Celery is pretty much entirely fire-and-forget, and you're usually firing to an entirely different server. If this is being used to achieve parallelization instead of asynchronous processing, then it's no wonder that the parent is having trouble, because they're using Celery for a task for which it's wholly inappropriate.
    - crucialfelix 10y ago
      Agreed. There are a lot of bells and whistles with celery and I've tried them and now avoid them all. I don't nest or chain tasks. I don't use periodic tasks because they can pile up badly and block everything. I have my own simple loop that calls the most urgent periodic-task. I built an angular button that would call a task and would show spinning in-progress and then show when the task completes. Works on development, didn't work on production. why why why ? well, no time to fiddle with celery so I moved on. over the years I've come to mistrust its complexity.
      - kerkeslager 10y ago
        > I don't nest or chain tasks. Yeah, I think nested/chained tasks are pointless. The point of celery is to offload the task to an asychronous worker (probably on another machine) so if you're already on the worker there's no point sending another task to the same worker--it doesn't achieve anything above doing the task inline. > I don't use periodic tasks because they can pile up badly and block everything. This sounds like you're either running the periodic task too frequently, or you need more workers. I've seen a few cases where people tried running a 20-minute task every second on a single worker--of course this is going to pile up. Periodic tasks are among the most predictable, so they're easiest to prevent from piling up: simply make sure that the following relation holds true: w > t / p w = number of workers t = how long it takes for the task to run p = the period (a task that runs ever 20 minutes has period 20 minutes) The > should be greater than by an order of magnitude to be safe in case t increases.
brianwawok 10y ago
I think the idea is for many things you don't sit around and wait for Celery to complete. User clicks "send email"? You throw a send email task in Celery. If it takes .1 seconds or 12 seconds, the user doesn't have to wait for it, because they don't care. This is perfect Celery use. Now when you are loading a page to display to the user, and that page makes a bunch of celery tasks that you NEED To complete.. this is more iffy.. in most cases, you may just want to run those inline in the webrequest. Needing a bunch of tasks to complete to show a user a result is not an ideal use case of Celery.

MollyR 10y ago

Its an interesting use case the author solved with celery. I've used celery with a few django and flask projects. It's nice, but it added a lot of overhead for me. So I've used http://flower.readthedocs.io/en/latest/ http://flower.readthedocs.io/en/latest/ to help handle the monitoring. For some really small sites and very specific use cases, I actually wrote some small custom helper scripts to just have cron jobs. If you are using django, I found this library helpful. https://bitbucket.org/wnielson/django-chronograph/ https://bitbucket.org/wnielson/django-chronograph/

brianwawok 10y ago
The biggest overhead is really setting up RabbitMQ right? Because Django and Celery play nice, you can just deploy one codebase and run 1 Django process and 1 Celery + Celery Beat process pretty easy... Bummer you lose so many Celery features when you use Amazon or Google Cloud Queues instead of RabbitMQ....
- mdomans 10y ago
  AFAIK you can have RabitMQ SaaS these days. And yes, RabbitMQ is very good.
  - brianwawok 10y ago
    cloudamqp? I guess, not terribly cheap though.
    - mdomans 10y ago
      I think there are a few other options. But deploying RabbitMQ isn't that hard. I can actually write something about that. Is Linode cheap enough for you?
      - brianwawok 10y ago
        I mean last time I needed RabbitMQ I just used https://github.com/Mayeu/ansible-playbook-rabbitmq https://github.com/Mayeu/ansible-playbook-rabbitmq and it took 3 minutes. But I hear a lot of people saying not to use Celery because RabbitMQ is hard.. maybe people just have different standards for hard?

forgottenpass 10y ago

Design patterns are just that, design patterns. With upsides, downsides, suitable use cases and unsuitable use cases. I was originally going through pointing out all the nonsense that was outright wrong or just ruffled my feathers, but this whole article is punching above it's weight. And boils down to the following: The author doesn't like using concurrent programing techniques in a single-threaded process. The author does like a job queue that manages task execution across multiple processes. This has fuck all to do with the GIL beyond the fact the presence of the GIL might have contributed to the popularity of the former in python and that celery uses multiprocessing over multithreading. Designing a piece of software is more than cargo-culting the latest fads. Who cares if asyncio exists? Yes, we "really need" it to the extent that people want to design their programs like that. No, you don't "really need" to use it.

markbnj 10y ago
The title is off I agree, it's not like Celery fixes cpython's problematic data structures. I don't think it was a bad article, though.
- forgottenpass 10y ago
  I wouldn't have minded reading someones process of learning a new design pattern if it wasn't asserting so many things that are silly or flat out wrong during the process. That's why I originally wrote, "this is a bad article." I edified that bit out because it detracts from my point unless I take the time to go through and fisk the article.
  - mdomans 10y ago
    Well, I'm more about advertising the lockless programming pattern :) Point out the silly, please, I like it when someone actually makes honest effort to teach me something.
    - scott_s 10y ago
      The title, mainly. Most readers will take that to mean that you fixed the GIL itself - that is, removed it - rather than are presenting a pattern that avoids it. Your title sets up an adversarial relationship with such readers, because their attitude when they start is going to be "no, you didn't", which will cause them to miss your actual message.

aduitsis 10y ago

I have used Gearman for a long time and I like it a lot, very simple to use, lightweight, does its job and doesn't get in the way. It's one of the few cases where something literally just works. If one would just want a job queue to fanout jobs in a network transparent fashion, would moving to Celery provide any advantages? With regard to Celery's broker, which is most preferable, Redis or RabbitMQ?

mdomans 10y ago
RabbitMQ - it's simply better, I can elaborate but it'd take an article on it's own. Unfortunately I never really used Gearman in high load environment, so can't comment on advantages of Celery over Gearman

omni 10y ago

> In the last 10 years I've never met a case where someone pointed out that "this code needs to be concurrent, but single threaded". I wish I could have slept through the circa 2012 Node.js hype cycle like this person clearly did.

omni 10y ago
To follow up on my own snark: > Do people use coroutines? Yes, but not in production code. Now I'm wondering if I'm the one who's confused. Either the author has also never heard of Go or goroutines don't meet his/her definition of coroutines.
- mdomans 10y ago
  I know of goroutines and comparing Python's coroutines to goroutines is like saying that Porsche and Tata car's are same, since both have 4 wheels.
  - omni 10y ago
    Fair enough, I thought you were speaking more generally since the next sentence was: > I may be opinionated, but I've done concurrency in many languages and never ever have I seen anything less readable than coroutines.
- weberc2 10y ago
  I'm guessing it's the latter, and perhaps not without cause. Traditionally, coroutines require the programmer to explicitly yield. With Go, you don't have to think about that much; you can pretty much treat them as lightweight threads and trust the scheduler to do the right thing. Goroutines aren't pre-empted, but the yield points aren't explicit (though they are well defined and thus easy to reason about).
  - jerf 10y ago
    "Traditionally, coroutines require the programmer to explicitly yield." When terms move from academia into common usage, they inevitably blur. Academia had a specific definition of coroutine that includes that. Programmers now use "coroutines" in a way that I have a hard time distinguishing from "threads", Pythonic "generators", and in the worst cases "something vaguely concurrent". Any given author may have a clear idea of what they mean (and in particular, I'm not making any specific accusations about the author of this piece) but the term is rapidly approaching uselessness in the general programming community.

sametmax 10y ago

The TL;DR (tl;dr: this explains how Celery obsoletes GIL and coroutines) is only telling half of the story. Celery solves some cases for Python: - background tasks; - tasks queues; - cron tasks. And it's nice, I used it extensively. But in NO WAY it totally solves the GIL problem nor does it render coroutines obsolete. Here are some celery limits: - you need to calibrate the number of worker you have to match your workload. - You are limited to x number of blocking operations, where x is the number of workers. It means you cannot do massively parallel I/O such as network operations with it (e.g: a web server). - Tasks don't have access to your main process memory, and vice versa. - Tasks cannot communicate with each others; - You must juggle with the workflow of your tasks (is it ready ? it it dead ?). You can use await stuff() with a try/except; - Celery is an additional process to setup and start, with backends to choose from and tuning to do. It's a lot more work than just importing an async lib. Granted, celery is a very useful piece of software, but not the silver bullet this article depicts.

mdomans 10y ago
On Celery limits: - celery has autoscale which works _awesome_ - autoscale? also, having bounds isn't such a bad idea in reality :) In regard to what you've written about tasks - yes, and point I was trying to make is that Celery model is a good example of what can be done to enable concurrent programming in Python
deleted 10y ago
[deleted]
asksol 10y ago
> - You are limited to x number of blocking operations, where x >is the number of workers. It means you cannot do massively >parallel I/O such as network operations with it (e.g: a web >server). Celery can use eventlet/gevent instead of multiprocessing for executing tasks, so this should be possible (granted, not sure if using it as a web server is a great idea) >- Tasks cannot communicate with each others; This is not true, they can send messages to each other >- You must juggle with the workflow of your tasks (is it ready >? it it dead ?). You can use await stuff() with a try/except; If you have to juggle it means your workflow design is not good enough. I.e. you shouldn't wait for other tasks, you should have callbacks and errbacks. Also, note that a try/except does not guarantee the operation will be completed (e.g your asyncio app can be killed). I'm not sure there's much point in comparing these, they are wildly different concepts: Celery is a distributed system, asyncio is for async I/O, the GIL is only a problem if you can't start n instances of your app.

garethrees 10y ago

The article makes claims that are way too broad: "fixed Python's GIL problem", "obsoletes GIL and coroutines". Maybe the claims are reasonable when addressing the author's particular use case, but they are not true in general and phrasing them in this way seems designed to provoke. It's important to be clear about which use cases are addressed by a given system. For example, if you have compute-bound tasks that operate on non-shared data then it makes a lot of sense to distribute them via a multiprocessing system like Celery. But if you have I/O-bound tasks that operate on shared data then it doesn't — implementing the necessary locking and communication is very hard. In this case, coroutines are simple to program and reliable, and because you're I/O-bound you're not losing anything by being single-threaded.

mdomans 10y ago
Here's the deal, most tasks I've seen are both CPU and I/O bound, only in different part of the same task. Of course you can very aggressively optimise the design, but sometimes it becomes so unreadable, the cost you incur through complication makes it not worth the effort. What I'm trying to advocate is that the distinction of CPU bound and I/O bound makes sense in CS classroom, but most of the tasks in production code are a mix of both. Therefore, the distinction is really between blocking and non-blocking, and Celery enables "almost readable" non-blocking coding
- garethrees 10y ago
  Thinking about where the computation is bound is only one half of the space of use cases: you also have to consider the shared/non-shared axis. There's a 2x2 matrix of task types: non-shared shared compute-bound multiprocessing ??? I/O-bound either coroutines I put ??? in the upper right because as far as I know Python doesn't have any general solutions here. Also, if "most tasks you've seen" are compute-bound at some points, then of course it makes sense to use Celery! But you should bear in mind that other people may have seen other kinds of task.
  - mdomans 10y ago

armitron 10y ago

Terrible post, worst I've read today by far. Not only does the author completely NOT understand why the GIL is a problem, but he also misunderstands asynchronicity which is hilarious given his recommendations.

mdomans 10y ago
And why GIL is the problem, in your opinion? Honest question, explain your point of view.
- jerf 10y ago
  In a nutshell, if your thesis was true, it would be true. If Celery was the solution to the GIL, nobody would still be talking about the GIL. Celery may be slick and well-developed, but sharing stuff out to processes has been well-established tech for a long time. If that were all people wanted, the problem would be solved. Since it's not, people must want different things. It's great that you solved your problems with it. Many other people have too. Many others have problems that are less amenable to this approach.
  - mdomans 10y ago
    Assuming that people will stop talking about something because it's not a problem or it isn't real is a bit ... uninformed :) The main point I was trying to make is that if you can force the programmer to guarantee safety, you don't need to rewrite GIL. And approach Celery enforces you to use, can be, with some extra syntax sugar, back ported to Python to have nice threading API
    - jerf 10y ago
      Very well, then I'll also point out that among the people still talking about the GIL are those who know enough about Python to successfully take steps to remove it, which is pretty much the highest possible level of knowledge of Python you can ask for. This is probably a topic "in the air" precisely because of the story of someone's most recent attempt/implementation of this a couple of days ago. It is not the case that only "people who don't really know Python" are talking about the GIL and considering the problem unsolved. If you were right, these people would not be talking about it. I would strongly, but politely, suggest that you should consider the possibility that there either things you don't know about the issue or problems that others have that you don't. It's not like Celery is the only obscure library nobody's ever heard of that has this basic approach; "multiprocessing" is built into the standard library now, for instance.

zzzeek 10y ago

> this explains how Celery obsoletes [emphasis mine] GIL and coroutines > Celery is a whole project and there's tons of coordination before you fire your first task - it's true. well that's why it's not really something that "obsoletes" threading (or your fancy-pants you-kids-today concurrency system of choice). Geez can people no longer write without hyperbole as a default position?

coldtea 10y ago
>Geez can people no longer write without hyperbole as a default position? Oh, the irony. (One or a few articles are hyperbolic != "people can no longer write without hyperbole as the default")
- JustSomeNobody 10y ago
  One or a few? Come on. Most articles to day are this way. And programming today is hyperbole induced cargo-cultism if it's anything at all.

JustSomeNobody 10y ago

Python doesn't have a GIL "problem". Just stahp.

mdomans 10y ago
Have you seen a program of any Python conference :D
- JustSomeNobody 10y ago
  So? Calling it a "problem" is crap. The GIL was a decision that has repercussions on certain types of programming, sure, but it's not a "problem".
  - mdomans 10y ago
    It is for the community. Personally I consider GIL to be an awesome idea and it'd work if we went with a concurrency model that avoids implicit locking

FunctionalGuy 10y ago

Or you can use a language like Erlang which has first-class support for this sort of architecture.

TheGuyWhoCodes 10y ago

Celery is great and works well but it definitely doesn't fix the GIL problem. Having shared memory is a must for some workloads, or can save you a ton of money, especially when taking about caching. Sometimes caching per process isn't feasible but caching for all threads is, even using multi processing isn't good enough because you can only use the main process memory if it's simple arrays.

mdomans 10y ago
Talking about shared memory benefits and overheads in Python is a pretty abstract problem, if you account for how much objects Python interpreter creates and what OS does under certain load levels. I'm merely advocating a different model for concurrency in Python :)

sciurus 10y ago

Celery boils down to a nice abstraction over forking another process to do your work. You still have the scalability problems that might make you want to write concurrent single-threaded code, just at a different layer of your stack. Take the author's example of scraping 1000 webpages. Let's say their computer running celery has enough memory to run 50 celery processes. This means they can request 50 web pages concurrently. Most of the time those celery processes are going to be sitting idle, waiting on the remote web host to respond. This is terribly inefficient compared to coroutines with asyncio/aiohttp, or even using a thread pool since urllib will release the GIL and let another thread run when it's blocked on network i/o. You could have a single process performing the work just as fast as 50 celery processes.

asksol 10y ago
Celery can use async I/O too! Using the gevent pool you can have e.g. 4 celery worker instances with 1000 threads each.
- sciurus 10y ago
  Right, but as you mention elsewhere in the thread there are caveats, like discussed at http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html#concurrency-eventlet http://docs.celeryproject.org/en/latest/userguide/concurrenc... . At this point the blog post author's contention that celery is an better alternative to doing async i/o breaks down.

dikaiosune 10y ago

I recently opted to use Celery for a similar-ish task to the author's update_metrics job. Having generally been OK with Python's single-threaded performance and simplicity, I was not particularly pleased with a solution that involved an additional backend service (RabbitMQ), separate logfiles, fighting with the Python module system during setup, and ultimately a lot less available documentation than for items which live in the standard library. All of this just to kick off a background process. I haven't had my coffee yet and am feeling a bit grumpy this morning, but I'm really not sure I would be willing to use it again for a project where I had to manage the code and the deployment. Regardless of my mood, celery definitely isn't a panacea for these problems -- it just happens to ameliorate ones where there's already a larger deployment to manage.

wacowz 10y ago

Every time I read an article like this, all I can think is, this is what Elixir and Erlang are built for. I love Python and use it on a daily basis, but developing concurrent systems in Python is beginning to look like a questionable choice, while there exists tools better suited for this. Celery works great, and the integration with django is superb. However, it's consistently been a memory hog for us. Hell, we had to write a shell script to regularly check for memory use by celery and take necessary precautionary steps.

brianwawok 10y ago
What level of concurrency is required though? 99.9% of web request only needs concurrency at the user level, which standard python + fork will do just nicely. Making a realtime system to crunch some massive numbers in 50 micros? Then you might want some C or Java concurrency. But much of the world is not coded at that level.
- wacowz 10y ago
  It's often the case that some work needs to be done outside of the request/response lifecycle, eg external api calls, data aggregation, sending notifications etc. Most of it is essentially IO bound. We happen to have lots of such tasks and therefore use celery, but there's only so much celery can do, given it's concurrency level is bound by the number of cores available since it uses multiprocessing. An actor based approach or sth like Go's goroutines would be a better fit.
bitwize 10y ago
Developing concurrent systems in Python on Unix is a questionable choice. On Windows it's a doddle, because Windows supports true async I/O. See: http://pyparallel.org http://pyparallel.org
- wacowz 10y ago
  The numbers for pyparallel are quite impressive.

cyberpanther 10y ago

Celery does solve the GIL problem and in some use cases is better suited even if the GIL wasn't a problem. However, it is an additional "system" that you have to add to a code base. If you're writing a small program that is easily distributed and needs "non-blocking" features, Celery brings a whole host of extra requirements. This becomes additionally problematic for Python when other languages implement these "non-blocking" features into the language. Thus you can implement these things in a few lines of code, where with Python you must learn a new framework and have a complicated install to use a simple feature. If Python wants to compete with other languages in the future it will have to keep up with these "non-blocking" features without a huge system overhead expense. I see this same problem with Django Channels which lets you implement web sockets and such with Django. Django Channels relies on Redis and solves more problems than just web sockets. However, if I just want a web socket, doing it in Django Channels produces too much overhead. Writing a web socket in Tornado can be done in few lines of code without the extra "system" overhead.

seomis 10y ago

Why is it that Python's single-threadedness is always singled out over every other language in the general family? Ruby, PHP, Perl, and JS are all single-threaded (Perl ithreads and JRuby/Jython caveats notwithstanding). Celery itself is just a framework for fork management, something that has been available to these languages since their inception.

nawitus 10y ago
JavaScript's "logic" may be single-threaded, but it's asynchronous event loop solves a lot of problems that a single-threaded synchronous model has (like what Python has by default).
- seomis 10y ago
  Asynchronous event loops aren't a core part of the JavaScript: They're from the DOM API and Node standard library. Though it is significant that nothing comparable exists for Python that has reached the same robustness and support.
Rafert 10y ago
And as a Rails dev I was thinking: "hey, somebody is taking a shot at Python's single-threadedness for a change". I'm not familiar with Celery, I gather it is similar to Sidekiq/Que/Resque/DelayedJob/etc?

fideloper 10y ago

So if I'm reading this correctly, the author is using a queue system (Celery, presumably with a backend such as RabbitMQ?) to run background jobs, but then waiting for them all to complete before finishing a task. In effect, you sorta get concurrency (assuming multiple workers are processing jobs). Or does Celery also provide a way to do threading without using a backend service (e.g RabbitMQ, beanstalkd, sqs)? Basically, I'm confused.

brianwawok 10y ago
You could totally do concurrency through Reddis or something super lightweight right on your webserver... could make sense. Really seems better to throw your task back to the backend pool though, and let a web request finish.

asksol 10y ago

Author of Celery here (not the article). Celery solves a different problem: it's a distributed system that will help you run your tasks on multiple machines, not an alternative to asyncio/twisted/tornado etc. The GIL is often easily worked around by starting N instances of your app, but that doesn't work for all applications (games, video, audio). Celery won't help you in those cases, but you can write C extensions. 99.99% of the time you shouldn't even consider using posix threads in Python, as most libraries (including popular ones) are not thread-safe, resulting in spending precious time fixing tricky bugs. Celery is also not in contrast to coroutines, actually it's quite common to use Celery as a distributed layer on top of async I/O frameworks (top tip: routing CPU-bound tasks to a prefork worker means they will not block your event loop). Another thing, the article says `scrape_url.subtask(args=(url,)),` is not very readable, but the idiomatic way to write this is: `scrape_url.s(url)` (yup we have a one letter method name, Django has Q we have .s). See more examples here: http://docs.celeryproject.org/en/latest/userguide/canvas.html http://docs.celeryproject.org/en/latest/userguide/canvas.htm...

brianwawok 10y ago
TIL about the s method.. Thanks for the tip!
enjo 10y ago
Nothing useful to add other than thanks for the great library. I've been using Celery for years, it's made my life demonstrably better.
mdomans 10y ago
In all honesty .subtask() is more readable for me than .s() Also, I don't want Celery to replace asyncio and/or threading. I was trying to point out that the programming model can be back ported to Python as a way of approaching concurrency, which, without having to support "locks everywhere" is simpler to implement with GIL
- asksol 10y ago
  I think it's more readable in isolation, but when you have workflows it distracts you from what it's actually doing, making it harder to see what the purpose is at a glance. An API can probably not have many shortcuts like this, but you learn what .s does once, and then you barely notice it. But that's my opinion and I have come to learn not everybody value succinctness in code, so you have the choice of both :o) Had I not been constrained by backward compatibility I may have made it so that task(arg) only defines the signature of a task invocation and you'd need to do task(arg).delay() to call it remotely, and task(args)() to call it as a function locally.

abotsis 10y ago

To all the haters: the GIL is just a constraint that can be worked around. Clever ways of starting more than one interpreter is a solid way to do that. If you can't think of a way to do that, then maybe you shouldn't be working on distributed systems. For example, I wrote a high-speed distributed database load tool, a perfect python use-case. It did some transforms, but nothing heavy. The database (Vertica) could generally keep up. For large files (multiple TB) I'd divide the file into a number of chunks and each interpreter would just seek into an offset, read to EOL, then load until it reached the next interpreter's offset. I built it in a few hours. Just trying to say that there are very few real-world workloads (were python is extremely useful) that the GIL really gets in the way. It certainly doesn't offset the time and ease of developing a solution.

dang 10y ago
> To all the haters This is a form of name-calling, which the HN guidelines ask you not to do in arguments. And the "if you can't think of... maybe you shouldn't be" is also a jab. Please don't do these things in HN comments. They add no information, just provocation. Consider how much better your otherwise fine comment would be without them. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html
- ixtli 10y ago
  I worry that you would not have posted this response if the top comment had said "Dear dissenters,". However I think you're right about the extraneous and specious "maybe you shouldn't be working in distributed systems" claim.
  - scott_s 10y ago
    Perhaps not, but for good reason: "dear haters" and "dear dissenters" mean different things. As does, "To everyone who disagrees, please consider ..." The word "haters" is combative.
- 10y ago

methodover 10y ago

Celery is neat. Something that sort of bothers me with it though is -- and this is maybe totally something that is my fault, I just can't figure it out -- sometimes workers will just stop processing things. For unknown reasons. I've set up some alerts to automatically slack me when this happens, so I can go in and see why they're stuck and restart them. Still haven't been able to figure it out. Happens maybe once every month or two.

harel 10y ago
Put a hard time out on the worker and it will be killed and restarted if hanging for X period of time.
- jvoorhis 10y ago
  In my experience, celery takes some hand holding to stay alive. I generally configure it to recycle workers after N tasks, in addition to a hard timeout on tasks. I've also found in my configuration, using RabbitMQ as the bus, workers can become non-responsive after receiving too large a message.
- methodover 10y ago
  Yeah, I've thought about it. I guess I've been holding out hope that I could figure out the underlying cause. What's wierd is, it always affects just one type of worker -- the workers that handle sending an email. And it happens on all workers simultaneously, on all servers, of that type. Just that type. Meh. I should just put a harakiri timeout like you suggest.

ploxiln 10y ago

> Do people use coroutines? Yes, but not in production code. Are you serious? Well, first, surely in all this talk of cooperative threading, you also include callback-oriented event-based programming. This is HUGE. Have you heard of nginx? But also, I've personally worked on very high volume api servicing and analytics, largely written in python using tornado (which is very similar to twisted) for a very popular web service: Bitly. There were also a couple significant components written in c using libevent, another similar event-ed i/o mini-framework, involving lots of callbacks. It can be done right and be readable and debug-able, and it can be super efficient, more efficient than spawning threads for hundreds or thousands of parallel i/o tasks (or managing a thread pool or forking processes or whatever). It's not for everything but it's definitely in serious production environments. The more serious, the more likely to use event-ed i/o.

mdomans 10y ago
Cooperative concurrency and event-based programming are different concepts. You need events for the coroutine since scheduler needs to base trampolining of something. Alas, you can do event based programming without coroutines. Yes, I used libevent and libev and I do love nginx :)

deleted 10y ago

[deleted]

btilly 10y ago

There is opinionated, and then there is stupidly opinionated. This article crosses that line then stays there. It offers one useful point, but that one is hardly news. For example we have claims like this, Do people use coroutines? Yes, but not in production code. This claim is trivially false. There is lots of production code that has the word "yield" in it. Heck, plenty of production code uses Tornado, which at its heart is nothing BUT coroutines. And most programmers have no trouble reading it. Moving on we see, Also, cooperative concurrency aka coroutines was abandoned a fair amount of time ago. I wonder why? I'm sure that people who use Redis and Node.js would be surprised to discover that cooperative concurrency has no place in our modern software world. He even concludes that people don't need concurrency. Which is obviously ridiculous - just look at any distributed system. But let's go back to why people move from cooperative concurrency to preemptive. Answer it will make it clear why we'll always have cooperative concurrency in the programming mix. Cooperative concurrency means that you don't need locks and don't have race conditions. However one poorly programmed routine can block your whole system. Preemptive concurrency means that a bad routine can no longer block your whole system, but now you have opened up the guarantee of complex race conditions. Programmers are really, really bad at understanding race conditions. So bad that almost all of our software has them. So bad that people who write automated code verification tools have learned the hard way to not report on them because programmers won't understand what you are talking about. (Sadly true. In http://web.stanford.edu/~engler/BLOC-coverity.pdf http://web.stanford.edu/~engler/BLOC-coverity.pdf look for the phrase, "...for many years we gave up on checkers that flagged concurrency errors.") Therefore if you want concurrency with no locking overhead, and no race conditions, then coroutines are great. But once your application becomes big and messy enough, it becomes better to accept the problems that come with preemption. Now if you're going to preempt, there is a ton of advice to be offered on how to do it well. For example you'll generally have fewer problems with coarse-grained processes than with threads. Queues and worker pools are a great solution for some types of problem, and I first did that about 15 years ago. It was hardly an original idea when I first did it - it was suggested to me by someone who had been using that technique since the 1980s. And it wasn't original with him either. However there are a ton of use cases that this doesn't cover. For example it doesn't do much to help you handle high volumes of traffic to your webserver, and it won't let you create a distributed data store for low-latency access to terabytes of data. So put celery in your tool bag, know what it is good for, and don't make the mistake of thinking that it solves all problems.