18 ms·
Why do we need Flask, Celery, and Redis? (2019)
- pupdogg 7y agoWith all due respect, my recent experiences ordering at McDonald's have been nothing short of horrible! As nice as the kiosk is, it has taken the accountability factor for your order out of the picture. I vividly recall getting blank stares from employees when asked "how much longer until my order is complete?". My past few visits (11 to be specific from 11/19 thru 2/20) have yielded 8 minutes of wait time on average. This is ordering inside the facility and at 5 different locations. Last I recall, it used to be a lot faster...I think between 1-3mins tops! I can't say the same for drive-thru though...seems like any orders from the drive through are always tagged with a higher priority. I do remember this since I was in my teens in the late 90s. Though it looked like a mainframe system, McDonald's did have a countdown timer that would initiate on orders. I'm all for tech and automating queues...but humans are complex beings and until the gap is bridged, I think we have a lot more room for improvement!
- jessaustin 7y agoDrive-through is higher priority at most restaurants, because the customer can drive away after ordering but before paying.
- procinct 7y agoOne aspect of this set up I’ve never been able to understand is how the application then gets the result from the worker? If it’s polling for status from the backend doesn’t that defeat the purpose of having a worker to begin with? Or does this set up only work for tasks that don’t need to have the backend notified about the result so the front end can just poll for the result via the application?
- leowoo91 7y agoIn case of a polling requirement from user perspective (e.g. waiting for a notification) it's true it doesn't matter. However, it allows application to respond to 'other' users. Whole point is to serve multiple users at the same time.
- procinct 7y agoWhat I mean is that if your backend is polling for a task to finish, that is also taking away time from other users. For some tasks that won’t matter because the backend won’t need to be aware of the outcome of the task but there could be some longer jobs that the backend needs to be aware of the outcome right away. You could get it so the polling is done on the front end and then passes the outcome to the backend but that obviously isn’t a good idea because then the backend is trusting outcome data from the front end.
- leowoo91 7y agoI see your concern is focused on polling case (e.g a chat room). As goes with the McDonalds example, a single cashier can reply the question "is my order ready?" rapidly. While it takes seconds in real world, a client polling request would take milliseconds to complete, so it can still serve hundreds of clients in a given second. If you'd need more for that part, there can be more cashiers/apps to make "asking" part scale indefinitely (even you'd have 1 worker only).
- WJW 7y agoIf you really need the answer immediately to show to a user on a page, background workers (usually) don't help. However, there are a bunch of situations where there is no feedback to the user other than "we received your request in good order and will see that it's done". For example, sending a mail can take a while because mailservers have queues and whatnot. If you send a mail after signing up a user for them to verify their email, you don't have to wait until the mail is "really" sent before letting them know that their signup has been processed. You can tell a background worker to send the mail and return to the user much faster. For another example, in a previous job I worked for a big file sharing service. If user wanted their files deleted that caused all sorts of calls to AWS to actually delete the files, which could take a while. However, from the user perspective it was pretty fast because all we had to do was set the file in the database to "delete in progress" state and tell a background worker to delete the files. Then we could show the user that their files were being deleted within a couple dozen milliseconds instead of having to wait for all the AWS calls to complete.
- 1337shadow 7y agoNot to mention the case where the mailserver is down or denies service, which will also happen at some point even if you have HA mailserver: be it with AWS emails, mailjet and whatnot. One day it'll fail everyone. Then, what's it going to be ? Return an HTTP 500 to the user rolling back the transaction ? Spooling emails removes that failure spot at all.
- masklinn 7y agoI would agree, generally a task queue makes sense for jobs which are not needed to respond to a query (e.g. generating reports or sending emails or whatever), otherwise you're just adding delays in the response chain. Although this delay chain might be considered worth it you don't want to scale the frontend to multiple workers for some reason e.g. single-threaded evented runtime, or GIL runtime (python, ocaml), or if you want to avoid CPU-hard tasks being executed on your frontends. In that case, it might be valuable to transform CPU waits into IO waits by moving the CPU work to a jobs queue, possibly running its workers on a different set of machines entirely.
- adamcharnock 7y agoYou’re spot on really. Having the front end wait for a background task to complete broadly defeats the purpose. There are some caveats though: if you’re using an async/threaded web server then it may not matter that you have a pending request hanging around as your web server is free to continue serving other requests. It also may be that you need to run the task on specialist hardware for some reason. Really though, I think a lot of people use celery for offloading things like email sending and API calls which, IMHO, isn’t really worth the complexity (especially as SMTP is basically a queue anyway). Of course, YMMV depending on your use case. However, I find it is often more worthwhile for: 1. Tasks which take a long time to run 2. Tasks which need to happen on a schedule, rather than in response to a user request. There is an option 3 too, which is for inter-software communication. Eg events or RPCs, but I found Celery to be very much a square-peg-round-hole for this, which is why I developed Lightbus (http://lightbus.org http://lightbus.org). Lightbus also supports background tasks and scheduled tasks. /plug
- doliveira 7y agoIt's not just about the time the operation takes, it's about reliability. Even if sending an email synchronously doesn't usually take more than a few milliseconds, you still need to handle cases like servers failing in the middle of the request, temporary upstream unavailability, some expired API, account limits reached, etc... Honestly, I think that mostly anything that doesn't depend directly in the current state of your infrastructure should be done asynchronously. I've had a lot of issues with systems that start up doing everything synchronously: you'll probably need to refactor it to be asynchronous in emergency mode during a crisis.
- marcosdumay 7y agoFor email, that's why you set a relay within your control, that will accept messages without a fuss and send them around following SMTP conventions. But anyway, how is your application supposed to respond after any of those failures? Is it just supposed to ignore the failure and thread on like if nothing happened, leaving your users on the dark? Is it supposed to reliably log every task so that it can retry anything that fails and in the worst case feed failures into some monitoring system/process? Or is it supposed to inform the user of any success or failure before the user can move on? Queue software is only a good match for the first. For the second you will need to roll your own interface with the monitoring system anyway, so it's much easier to roll your own queues and get control of everything. The third one is best done synchronous, it doesn't matter the nature of the process or how long it takes. But funny thing is, I have never seen the first situation on the wild.
- golergka 7y agoRight now, I'm designing a service that's very similar to OP, with workers waiting for an external API (or APIs) to answer, which can be slow sometimes. Looks like we're going to use websockets or polling to update the client with all the delivered data, and introduce yet another service for it, which would actually play the role of "LED screen" and ready order station in this analogy. But as we decided to implement an MVP with a single HTTP request from the client, this whole separation doesn't make any sense, exactly as you noted.
- notyourday 7y agoIn my experience a lot of applications with workers that require a result to be furnished to a client are incorrectly engineered. The correct pattern should be a client submits a request to get something done to a thin layer, receives a ticket that allows it to claim the result and goes away to either check for the result via polling for the ticket or receives a call back.
- doliveira 7y agoChecking the result is just doing a SELECT in your database.
- procinct 7y agoI was more referring to how it knows once it’s finished since the task is asynchronous. So for the backend to find out, it could do the select you mentioned and find out it’s still in progress. Then it checks again next second and still in progress. Then checks again and it’s done. But now your backend has been polling for an update and so you might as well have performed the task in the application because it is still being used up for the duration of the task.
- turtlebits 7y agoHave your worker POST back to your web server when it’s done. Use websockets to notify the user.
- jb3689 7y ago> If it’s polling for status from the backend doesn’t that defeat the purpose of having a worker to begin with? It depends. If you needed a coordination layer or you needed to isolate certain types of traffic then it makes more sense. I assume your alternative here is "why not just have a new client tier doing work" which is a reasonable architecture too > One aspect of this set up I’ve never been able to understand is how the application then gets the result from the worker? Often they don't in these architectures. I found it a little strange that a job queue was being used to serve (what seems like) synchronous traffic. Usually I see job queues in the wild used for async/send-and-forget workloads Personally I would rather chain multiple synchronous service calls if I needed a synchronous workflow. It's just simpler to me to stick web workers behind a load balancer and scale that. This is less elegant when things take a very long time though or are prone to retries. Either the client or server needs to be responsible for queuing/message persistence/retries - with services the client does it, with job queues the server does it
- alanfranz 7y agoAnd, of course, you need multiple separate components because Python/Flask has no central "application" concept,there are multiple, stateless processes. Had you got e.g. a Java app running with a multithread application server model, you could serve and process everything within a single process. No Celery, no Redis, no MQ. This doesn't mean that the above stack has no use. But, whenever picking a tech, you should understand the use case. The "simpler" Python/Flask solution has an increased complexity when the task at hand is not simple anymore.
- twhitmore 7y agoA rather vague and uninformative article. I liked the cartoons, but regard McDonalds as a negative corporation serving low-quality food to the lowest denominator of people.
- deleted 7y ago[deleted]
- WJW 7y agoWhen the task complexity rises further, splitting out the message queue and workers into separate processes makes sense again. That way, you can easily scale worker capacity up and down and you can restart your web server processes without having to worry about background job persistence since that is handled by the message queue.
- alanfranz 7y agoSure. You can do the same with the application server model. But you haven't to as long as you don't want to.
- polote 7y agoIt has nothing to do with Python, there a plenty of async web python framework.
- alanfranz 7y ago
- doteka 7y agoThey mostly need Celery and Redis because in the Python world concurrency was an afterthought. In most other languages you can get away with just running tasks in the background for a really long time before you need spin up a distributed task queue. In Python I’ve seen Celery setups on a single machine.
- mattbillenstein 7y agoYou still should use some sort of work queue - your application process may need to restart (deploys), or for a period of time, work could overflow the amount of available resources (bursts), so having some place to put the task before it goes onto processing is useful regardless of the underlying concurrency primitives of the language.
- rollingbarreler 7y agoTransactionality is enough for most systems. Your order either succeeds or fails, it never stays in an incomplete state. Queues are not a panacea, and introduce their own problems like obscuring problems by delaying them long enough to bring everything to a halt.
- mattbillenstein 7y agoYes, you wouldn't do all work in the task queue - commonly, we make some change in the database which can happen pretty fast, and after that transaction commits, we might defer a task that sends a notification, email, whatever.
- acjohnson55 7y agoIt may not make sense to retain jobs across deployments. What if the contract of the job is changed by the code being deployed? Might be easier to keep it all in-process, letting queues drain in a graceful shutdown.
- sateesh 7y ago
- andybak 7y agoI wish there was a paragraph up the top that made two points: 1. Quite often you don't (I've built dozens of websites without needing Celery) 2. Even if you think you do there's often a much simpler solution that is enough for most needs (Use cron, spawn a process etc) Celery is a big, heavy lump of code to add to most websites and it increased the deployment complexity.
- mattbillenstein 7y agoI prefer rq - Celery is too complex imho.
- ptype 7y agoAlso worth checking out nameko, I’ve had good experiences with it
- cepp 7y agoHuey is also worth a shot.
- mattbillenstein 7y agoI like it - uses redis as a broker - supports crontab style periodic tasks. I've usually had to build a small python cron runner using croniter in previous systems - which I think is a pretty clean solution - it just deferred tasks to rq workers. But having direct support in the lib might be nice.
- mattbillenstein 7y ago* depends on rabbitmq
- danpalmer 7y agoIt’s worth scoping out what your site will need to do up front to some extent. Spending a couple of days setting up a basic background processing system (whether that’s celery, rq, or a home grown system) makes it easy to make great engineering decisions later down the line. Questions like: should I send this email in-line in the web request? Get a very easy answer: no, just stick do it later. Sure, sending an email is probably fine to do in-line for now, but months in you may realise that things are slow, that you’re sending emails and rolling back transactions later, or committing the transaction but losing the email that needed to be sent, or all manner of other annoying edge cases. Queues don’t solve everything, but they can be an ok answer to a lot of stuff for a long time. For basic sites, yeah maybe not necessary, but a reliable background processing system has always been a significant accelerator in my projects.
- waterside81 7y agoWhat's interesting is that McDonald's wait times have actually gone up since they moved to kiosk ordering, mobile app ordering, Uber eats etc. They've increased their ability to take orders, but haven't been able to keep up on the supply side. The old way was almost better in that it introduced a natural bottleneck so while it took longer to place your order, once you did, the queue in front of you was shorter. https://www.businessinsider.com/mcdonalds-spending-millions-on-drive-thru-2019-10 https://www.businessinsider.com/mcdonalds-spending-millions-...
- xenonite 7y agoMoreover, one could just look at the shelf of already prepared burgers and buy one of them. So quick!
- nebulous1 7y agoThat seems to be looking at "speaker to order window" instead of "arrival to order window" times. This seems odd to me, as personally I would want to minimize the latter not the former.
- forgone 7y agoanyone knows why they are not doing the same in UK? Most restaurants stopped being available in Uber eats and deliveroo
- formercoder 7y agoWouldn’t that be working as intended? More revenue
- DeonPenny 7y agoThat seems like a good problem cause before they had issues with demand. I think based on McDonald history of optimization they feel like they can solve that supply side issue more easily than the demand side one they were contending with
- punnerud 7y agoCould it be that the reduced stress on the buyers result in reduced speed on the staff?
- tchaffee 7y agoMcDonald's is very often a love or hate divide as you can already see from some of the comments here. You could prevent that distraction by using a generic food takeout store and ask people to imagine their favorite. The article will resonate with a larger audience, and the comments will be higher quality.
- ljvmiranda 7y agoHi author here, definitely didn't intend and expected that! I'm mostly drawing from my own experience and that insight while ordering food inside Mcdo.
- tchaffee 7y agoAs someone who also does technical writing, I agree you should draw from your own experience. It can be hard to find the right analogy. If the subject of the analogy is enough of a hot topic, it will get attention itself. As you can see, you've got comments in here even talking about how McDonald's isn't faster with their new system, or how they could have better optimized for customers, links to articles about McDonald's business model etc. Some of the earlier negative comments about McDonald's were deleted - probably due to downvotes. Since my advice to other writers was sincere and I believe useful, I'm keeping my comment. For sure I'm not expecting you to change your article. Just hoping that my tip might help you with future technical writing. If not, no worries.
- ljvmiranda 7y agoHi, author here! Pleasantly surprised I saw this on HN, thanks for posting feross! Sorry for the Mcdonalds analogy, it's just that it's really near our office and I got that insight while ordering McNuggets! Didn't expect it will cause some divide Agree, Mcdonalds has definitely upped their ordering game recently. Thank you and I appreciate all the helpful comments!
- throwaway888abc 7y agoHey, Nice article format for actual humans with easy to digest flow. Will share with devs. Great work! Thanks
- nickjj 7y agoIf anyone is looking for another Celery post that goes over common web development use cases for using Celery there's: https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-celery-in-a-flask-application https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-c... The above post walks through sending emails out with and without using Celery, making third party API calls, executing long running tasks and firing off periodic tasks on a schedule to replace cron jobs. There's links to code examples too in an example Flask app which happens to use Docker as well.
- sandGorgon 7y agoI see a lot of comments that are talking about how python does not have a go-like concurrency story. Fyi - python ASGI frameworks like fastapi/Starlette are the same developer experience as go. They also compete on techempower benchmarks. Also used in production by Uber, Microsoft,etc. A queue based system is used for a very different tradeoff of persistence vs concurrency. It's similar to saying that the usecase for Kafka doesn't exist because go can do concurrency. Running "python -m asyncio" launches a natively async REPL. https://www.integralist.co.uk/posts/python-asyncio/#running-async-code-in-the-repl https://www.integralist.co.uk/posts/python-asyncio/#running-... Go play with it ;)
- pinkbeanz 7y agoI think a hard part with lots of these “what do I use x for” examples is it starts with the tool and the discusses the problem that it solves. I find it more helpful to start with a problem, and discuss the various tools that address it, in different ways. Forget email, say you have a app that scans links in comments for maliciousness. You rely on an internal api for checking against a known blacklist, which follows shortened links first, and an external api from a third party. You want the comment to appear to submit instantly to the poster but are comfortable waiting for it to appear for everyone else. What are your options? You could certainly use message queues and workers. If you’re cloud native maybe you leverage lambdas. Maybe you spin up an independent service that does the processing and inserting into the database in the background, and all you need to do is send a simple HTTP request on an internal network. Your solution depends on your throughout requirements, the size of your team and their engineering capabilities, what existing solutions you have in place. Everything has its pros and cons. Pretending that celery/redis is useless and would be solved if everyone just used Java ignores the fact that celery and redis are widely popular and drive many successful applications and use cases.
- StavrosK 7y agoWhile I agree with the rest of your comment, the sentence "if you’re cloud native maybe you leverage lambdas" made me irrationally angry.
- trboyden 7y agoExcellent craftmanship of a helpful blog. Very reminiscent of the style used by the Head Rush Ajax (http://shop.oreilly.com/product/9780596102258.do http://shop.oreilly.com/product/9780596102258.do) book O'Reilly published back in 2009 and the rest of the Head First series.
- 1337shadow 7y agoActually with uWSGI, be in on Flask, Django or else: I don't need neither Celery nor Redis. uWSGI has a built-in celery-ish spooler, cron-ish task scheduler, memcache-ish key-value store, along with plenty of other toys that I love from the deepest of my heart ... And I've been in this (great) situation for years, not planning to move out to more complicated stacks. I would highly recommend uWSGI over of Celery and Redis, which I used in the past, prior to doing it all in uWSGI, unless you have a really good reason which I'm eager to read about. And now that uWSGI supports a lot of languages, even if I have some PHP or whatever to deploy I'll go for uWSGI, one of the most beautiful piece of software I have the chance to use.
- d33 7y agoCould you elaborate, e.g. link to documentation, subprojects or example code? I'd love to get rid of Celery because of how difficult it is for me to tweak it for good performance.
- dzonga 7y agoyou can also use dramatiq
- darkerside 7y agoWould love to see more detail as well because I'm a bit skeptical. From uWSGI documentation, it looks like to coordinate cron across anything more than a single server setup, you'd need to configure a Legion, which means you're then integrating uWSGI's orchestration framework with whatever you're already using (k8, ECS, etc). I like minimalism, but sometimes batteries are included for a reason.
- GlennS 7y agoI suspect they are using a single server. I found uwsgi's mules and cache2 very useful in that situation. If someone finds that Redis and Celery are more complication than they need for a given task, then I think they're probably not using an orchestration framework.
- 7y ago
- bryanrasmussen 7y agoI think the McDonald's order by touch screen also has the benefit of seeming to take less time when waiting because you are not standing in line behind someone, even if it takes the same amount of time psychologically it seems less.
- quezzle 7y agoI only use celery for sending out emails. It’s overkill. I wonder how many other people have celery just for email.
- dralley 7y agoDoes anyone have enough experience with alternatives to Celery to give a good comparison of Celery vs. Dramatiq vs. RQ?
- nurettin 7y agoCelery is good for distributed and persistent message queues which can be monitored. If you just need multiprocessing, use a multiprocessing pool, it comes with python.
- kantye 7y agoVery noice! I am not normally not a huge fan of stick figures (a la waitbutwhy), but this was very pleasant to read :D And noice work picking an example I can relate to
- memco 7y agoI think it’s really important to understand task queues and workers but my experience working with these particular tools isn’t exactly fun. I inherited a system built on celery, rabbitmq and nameko I’d be interested to hear how people setup their systems to debug and test new tasks. I’m currently using manually added psb.set_trace to telnet in to a debugging session, but I’d prefer to use an IDE so I can modify code while debugging. Anyone have any tips? One thing I would caution against is putting the business logic in the task logic. This is obviously up to whoever set up the tasks but it seems like most of the tutorials don’t mention how painful this can make testing especially once you start making chains, chords and sub tasks.
- rukittenme 7y agoFor testing you can set "CELERY_ALWAYS_EAGER" to "True" in your config.
- btcnews 7y agoKiller news Bitcoin vs Ethereum https://coinmarket01.blogspot.com/2020/04/bitcoin-vs-ethereum.html?m=1 https://coinmarket01.blogspot.com/2020/04/bitcoin-vs-ethereu...
- deleted 7y ago[deleted]
- timwis 7y agoCool article! But why do you need a database _and_ a message queue? I would think a message queue is the main thing, and a database is only necessary if you want long term persistence. Or you could just use a database as a message queue.