11 ms·
McDonalds Event Driven Architecture
- alanhaha 4y agoI still remember my teacher uses McDonald to explain Instruction pipelining...
- achrono 4y agoWhile this satisfies some idle curiosity, I don't really get what justifies spending the resources in posting such stuff. Is this a recruitment tactic somehow for such companies?
- joezydeco 4y agoI have a hunch that's part of it. I also think they're trying to show that they're as technically competent as Chik-fil-A, who has been doing this for a while now. https://medium.com/chick-fil-atech https://medium.com/chick-fil-atech
- mbg721 4y agoThinking about it, that makes sense, given how gargantuan Chick-Fil-A is, but the two chains leave very different impressions to me. Chick-Fil-A feels like they're going out of their way to make it seem like there are friendly humans running everything, to the point of having employees with tablets stand outside at the drive-thru to take your order; lately, they've even been laying off the robotic "my pleasure" thing. McDonald's, on the other hand, feels like a clumsy extraterrestrial sidling up and saying "What is up, my fellow human, amirite?" The one by me is obsessed with pushing the little how-are-we-doing survey on the receipt; I don't think I've ever been there and not been given one. When I see their TV commercials, I keep expecting a flurry of tentacles to burst out of the smiley-person suits. But under the hood, they're both furiously optimizing everything they can.
- filmgirlcw 4y agoFor better or worse, Chick-fil-A is very obsessive about its standards and its quality. I grew up in Atlanta (where CFA was founded/based) and have had lots of friends who have worked for CFA over the years in various positions. Even in high school, CFA was considered the best fast-food job you could get (and better than some retail jobs) because it paid better than almost anyone else. The downside was there was a strict dress code (no visible piercings or tattoos, short hair for men, etc.) and an understanding that everyone had to be very well-behaved, but like, that comes through in the service. Beyond that, with a handful of exceptions (I think mostly airports and college campuses), CFA owns all their restaurants, unlike the typical franchise arrangement that McDonald's popularized. You still have independent operators, but they don’t own the restaurant and have to go through a rigorous selection process to even get one. And then, the franchisee only pays like $10,000 upfront, and CFA pays the rest of the costs to build the restaurant. Individual restaurants control their own hiring and marketing and whatnot, but this is all a very top-down approach that needs to align with corporate. McDonald's outright owns some of its restaurants, but most are owned and run by franchisees, and usually a franchisee owns more than one location. They license the brand, food, and process, but a lot of the day to day standards are decided by the franchise groups. Whereas CFA operators have less leeway for such things.
- joezydeco 4y agoIt's my understanding that CFA franchisees are only allowed to own one store (unlike MCD), and must work in their restaurant full-time. That's a significant difference from how MCD stores are operated and it shows.
- bitshiftfaced 4y agoI would never had guessed that Chic-fil-A was considered a contender to McDonald's in technical competency. I drive up and a person standing outside takes my order and asks my name. Then I drive up some more and another person asks me my name and punches something into a tablet, and then I drive up some more and get to the window. This is 2022. Businesses already have queueing software that automatically takes images of the vehicle to match with your order. I get that some customers like having the "human factor," but there's definitely room for improvement.
- bavell 4y agoMine has never asked for the name twice but they recently started slipping numbered cards under your wipers after placing your order. I remember CFA posting some articles a few years back on their k8s setup they use for store and inventory management.
- hamandcheese 4y agoChick fil an optimizes the process, not the tech, and it very much shows. Which is what any good business should be doing, IMO.
- filmgirlcw 4y agoThey have humans do it because it is faster. Significantly faster, in fact, to go car to car, rather than have each car drive up to a window to order. It also allows them to have multiple lines running at once. And it is probably just as fast to have to person handing you food just ask your name so they can make sure to pass it off efficiently. Chick-fil-A is the only chain where you will reliably see a massive line (often backing up to the freeway) on any given lunch hour (or a massive in-person line in New York City) that will still get you out and get you served more efficiently than another drive-thru (or walk up shop) that is 1/4 as busy.
- joezydeco 4y agoCFA has found lots of small incremental improvements that speed up the drive thru process significantly. MCD did the double drive thru ordering first, but my local CFA now has two pickup lines as well, with a heated covered roof. The window has been replaced with a wide open door and employees walk the orders to each car. Simple and it works great.
- itisit 4y agoThat's all it is. These EDA transformation journey blog posts are de rigeur for any company looking to have its engineering culture perceived in a modern light, at least in the eyes of a junior-to-intermediate developer.
- annoyingnoob 4y agoCan I get promoted to Manager if I get a degree in Hamburger-data-ology? From what I can tell the career path is into Corporate hell. https://www.uopeople.edu/blog/hamburger-education-inside-mcdonalds-university/ https://www.uopeople.edu/blog/hamburger-education-inside-mcd...
- greedo 4y agoIf you want to be a franchisee, the easiest way (other than having $3-$10m lying around) is working for McDonalds, taking AOC at Hamburger U, and then joining their franchise program where they "lease" you everything for a larger take of the annual sales than if you pony up the franchise fee and buy a franchise location.
- ripley12 4y agoWhat's AOC in this context?
- bckygldstn 4y agoAdvanced Operations Course (Found by googling "AOC Hamburger University -cortez")
- dbetteridge 4y agoIt is often an OKR for senior/staff developers to write blogs or tech articles to promote the companies engineering culture, especially for organisations that are not 'traditional' tech firms.
- cco 4y agoFrom the company's perspective, the hoped for result is mostly recruiting oriented, a little bit is related to offering career advancement (public visibility of your work) to individuals that write or are featured in the blog.
- 1123581321 4y agoRecruitment but also retention. Some engineers want to publicly communicate their work at a recognizable company, or aspire to become the kind of company good enough to have a well-liked engineering blog. Providing that opportunity doesn’t cost much and means a lot.
- bee_rider 4y agoProbably, yeah. But I’m always shocked for some inexplicable reason to hear that places like McDonalds and Walmart Labs are interested in solving tech problems. But I mean obviously, of course they are. Good to be reminded now and then that there are alternatives to contributing to the ad/social media panopticon, right?
- rjh29 4y agoYeah, you can work on making people fat or destroying labour unions! /s
- j-bos 4y agoIt feels so dissonant to see a giant company posting it's technical blog on Medium.
- keepquestioning 4y agoYou mean discordant
- j-bos 4y agoThanks :)
- weeniehutjrdev 4y agoI think both work, but it depends on what they mean?
- bcjordan 4y agoWanted to learn more about the difference so asked ChatGPT for a rundown with the context, pretty neat in-depth analysis (assuming it's accurate): > It's possible that either word could be used in the context you've provided. Both "discordant" and "dissonant" can refer to things that are unpleasant or conflicting, so either word could be used to describe the feeling of seeing a large company using Medium for its technical blog. > However, there is a subtle difference between the two words. "Dissonant" typically refers to things that are in conflict because of their individual qualities, while "discordant" typically refers to things that are in conflict because of their relationship to each other. In the context of your sentence, "dissonant" might be a slightly better fit because it emphasizes the individual qualities of the company (i.e. its size) and the platform (i.e. Medium) that are in conflict.
- keepquestioning 4y agoUgh holy fuck
- deleted 4y ago[deleted]
- b20000 4y agowhy does everything have to be microservice oriented and what's wrong with a simple monlithic application that runs the whole thing? also, when are those ice cream machines going to get fixed?
- toomuchtodo 4y agoTo your point, Shopify's monolith was handling 1.27M requests/sec on Black Friday: https://twitter.com/ShopifyEng/status/1597983918900510720 https://twitter.com/ShopifyEng/status/1597983918900510720 Thread: https://threadreaderapp.com/thread/1597983918900510720.html https://threadreaderapp.com/thread/1597983918900510720.html
- gscho 4y agoBut ruby is too slow!
- pxue 4y agoLots of their code is in Go
- ecshafer 4y agoWay way way more of our code in Ruby. Go (and now Rust) is really only used for very low level services with a high SLA (Like Infrastructure). Almost all business logic is Ruby + Rails.
- vips7L 4y agoTrue but you’re still putting significant investment into ruby to speed it up. Ruby is slow. I don’t think any other company could do that.
- tiffanyh 4y ago> averaging 3 Terabytes per minute of egress traffic across our infrastructure. That’s 4.3 Petabytes per day! I’d hate to get that AWS bandwidth bill.
- lokar 4y agoI was expecting a parody
- marcosdumay 4y agoI'm still not sure you were wrong.
- itake 4y agoTheir event driven infra has race condition exploits.
- rbosinger 4y agoBoth my partner and I always have trouble with the McDonald's app (for ordering). I use Android, she uses iOS. As a developer, I've said to myself "This feels like a React Native app that's calling into a mess of microservices" (having worked on that type of project more than once myself). Anyway, I only skimmed the article, but I had a chuckle seeing the title of this article pop up on HN at all.
- tharkun__ 4y agoCompletely unrelated probably but who knows but this reminded me of the time when my McDonald's receipt printout from that self-ordering terminal started with a bunch of XML, then the regular receipt part and then ended in a bunch of XML. That was fun.
- russelg 4y agoI can't speak for other region's McDonald's apps, but the mymacca's app (Australia) is entirely native Java on Android, and still runs like absolute crap.
- geysersam 4y agoI wished more people used a regular website for these kinds of applications. It's a form. What's the problem?
- quickthrowman 4y agoA web form cannot track your precise location 24 hours a day, but an app can.
- halfer53 4y agobackend of McDonald's app is handled by a third party new Zealand company plexure https://www.plexure.com/ https://www.plexure.com/ not sure what their architecture is
- rsstack 4y agoPlexure are only used in select McDonald's markets. Almost all Hacker News readers are in markets where the McDonald's app is developed in-house by McDonald's.
- campbel 4y agoITT microservices vs monoliths dogma.
- foolfoolz 4y ago> Standby Event Store: To avoid loss of messages in the event the MSK is unavailable, the platform is wired with a standby data store, where it writes events onto a database . The architecture provides tools and utilities to read messages and publish them back onto MSK, once it’s available. on one hand aws msk is good enough for an enormous application like mcdonald’s. on the other they need a backup database just to get around it not being available? what’s the real story here. interested to see where this goes
- chiph 4y agoProbably nothing more than a requirement for resiliency. With thousands of restaurants, chances of a couple of them losing connectivity each day are going to be pretty good, through traditional interruptions like backhoes digging up cables and drunk drivers taking out telephone poles. If those messages are discarded because the store can't talk to MSK (or MSK is unavailable), then things like automatic replenishment based on order volume couldn't happen. The store manager would have to do a daily physical inventory count to know how many bags of fries, boxes of drink straws, etc. to reorder.
- KptMarchewa 4y agoBut it's more likely that they'll lose connectivity to AWS altogether rather than MSK going down itself. I'd not try to write from each store network directly, but through some gateway that would take care of writing to Kafka and that separate store if we still have availability issues.
- hamandcheese 4y agoWe use kinesis a lot at work, and some services are architected to write events to a Postgres table which eventually gets dumped to kinesis, while other services write directly to kinesis. Guess which services fared better during the last kinesis outage?
- cgio 4y agoWhat's the point of using kinesis in this case? Why can you not consume from postgres. My understanding would be that you use kinesis for impedance matching on writes. If you can already write everything on Postgres reliably I cannot see any immediate use case for Kinesis. I could see the other way around, i.e. having kinesis as a resilience backup for postgres being a more valid concern.
- guyzero 4y agoSerious question, isn't think just rebuilding the same thing that commercial cloud pubsub offerings already provide?
- BoorishBears 4y agoI mean I'd expect if they're already all-in on AWS they'd just use Firehose with some deduplication instead of whatever home-brew fallback solution they described, but other than that it doesn't seem like they built much? What's impressive to me is that they need all that architecture. Mcdonalds sells under 100 burgers a second from what I can find, their order load is probably bursty, so assume maybe all the orders come in the same third of the day, so 300 per second, and every burger is it's own order... that's still not that much. One order is more than one operation when you're dealing with everything a place at McDonald's scale, but even if you multiply by a factor of 10 to account for analytics, compliance, etc. 3,000 operations per second? Does that really require an entire Kafka-driven event architecture?
- ehnto 4y agoI think you are underestimating how liberal some applications are, especially as analytics is one of their requirments. It's probably multiple events per thing you do. I wouldn't be surprised by 100+ events before even placing an order.
- BoorishBears 4y agoForest for the trees, I already practically doubled my numbers and assumed McDonalds gets all their orders in the same 8 hour period! And even then multiplying them by 100 doesn't get you into the realm of "we couldn't build this on a monolithic horizontally scaling application". If anything, if you're at McDonalds scale and still can't find the engineering skill to build a monolith that can handle 30k operations per second, you're playing with fire building a distributed system. (if you're a nascent startup, then by all means stand on the shoulder of giants and don't sweat that you don't have a full blown cloud engineering org, but that's definitely not where McDonalds should be...)
- georgeburdell 4y agoCall me a luddite but I still find it easier to tell my order to a cashier (I usually remove a few items like the godawful special sauces)
- lleb97a 4y agoI'm with you. I'll use the self service in shop but I never use the app.
- switch007 4y agoThey penalise you with enforced waiting time here in the UK (if it’s not per training to ignore you for a set amount of time, I’d be very surprised) Still worth it though. Their machines are awful - way too big and bad at input
- Rebelgecko 4y agoMcDonalds by you still have human cashiers?
- yurishimo 4y agoMost of the ones in the US have humans. They also have kiosks, but if you really want to talk to a person, you can.
- andy800 4y agoI am, for better or worse, a "heavy user" of the McDonald's app. And unfortunately, all this architecture somehow still misses so many usability issues. The most annoying one, is that the app always defaults to the closest location "as the bird flies." Despite the fact that I have "favorited" a different store, and also my last 5 or 10 orders were placed at this favorite store, the app ALWAYS defaults to the first store, which may be a tenth of a mile closer but has three additional traffic lights and twice the traffic. The app is also wildly inaccurate with store open/close times (in both directions -- sometimes it tells you a 24-hour store is "currently closed"), and the in-store employees are often confused how to ring up or serve a mobile order. But the deals are good. BOGO QPCheese just about every day.
- tstrimple 4y agoThe one and only time I've used the app was when I thought it would help them during the pandemic when drive thru's were actually slammed with demand while they didn't have enough workers. After waiting in the parking lot for 30 minutes while literally dozens of cars went through the drive through I went in to check on the order and the employees literally couldn't look up the status of my order without an order number which the app didn't give me at the time. An incredibly frustrating experience which turned me off to the concept altogether.
- wincy 4y agoI also use it a lot (I have a disabled kid who went from feeding tube to chicken nuggets and LOST weight, whatever she wants she gets). After you order and drive away, I’ll find the app hours later still tracking me and using GPS. I almost always have to force close the app. Then the next time I use it the previous order won’t have cleared out despite me picking it up, and I’ll have to go in and “cancel” the order. The coupons are good though. For a week or two it was giving 30% off, and almost always has a 20% off coupon.
- andy800 4y agoVery similar experience, I often open the app and it is still hung up on my previous (completed) order and I have to clear out the cart. And like you said, the "offers" are good (actually McD has raised prices quite a bit, now with the offers I'm paying about the same as face value was only a few years ago). I will also say, despite the issues, it's better than most competitors. BK and Wendy's apps are both full of problems as well. I have had multiple accepted and paid orders on the Burger King app, when I got to the store they didn't have the item, weren't accepting mobile orders, or were totally closed.
- spacehunt 4y agoIs it really scalable? Every time the local McDonald's have some cross-promotion going on with a certain boy band (for example this week's collectable cards), their app's backend always crash right at 11am when the promotion starts.
- tom_walters 4y agoDid AWS just pay them to write this? I was hoping for an interesting article exploring the complexities of a massively distributed and high-throughout system, but I just read “we connected these managed AWS services together and it’s cool” Hopefully further instalments might actually talk about the problems they faced building this out, and their unique challenges.
- toyg 4y ago> Did AWS just pay them to write this? This is part of their technical blog ( https://medium.com/mcdonalds-technical-blog https://medium.com/mcdonalds-technical-blog ), which looks like an outreach attempt to help with recruitment (hackathons etc). Probably someone went to a PM saying "we need to talk about our stack, how do we make it sound sexy and cool?" and this is what they got back.
- mark_sz 4y agoUnfortunately McD app doesn't make a good impression. It's quite buggy (iOS and Android) and has some obvious usability issues. So it doesn't matter what architecture is behind McD systems if customer facing software doesn't work correctly.
- nycdotnet 4y agoI find this blog post to be a funny snapshot of how event driven architecture initiatives can be sold internally. “Oh it’s going to give us all these great advantages and be super robust and performant. Here’s my diagram of producers and consumers with our new event gateway project in the middle. How does it actually work? Um… I’ll tell you later”
- manv1 4y agoReally, their app is a mess, but from a delivery point of view their stuff is pretty good. Just yesterday I was going to redeem a deal and the store was closed to walk-ins, so I had to use the drive thru. I couldn't use the deal anymore because I had used the code already, and you can't apparently switch between walk in/drive thru mode. Luckily the drive-thru was so slow I was able to use the code (there's a timer on the code). That said, the backend worked great; I ordered on the mobile in-line and the order was in-store once I got to the drive thru order speaker. People forget how hard and expensive it was 10 years ago to do a realtime architecture. Today, McD gets information from your phone to wherever and down to the stores with maybe a few seconds of delay, so you can use your code at the in-store kiosk. And it has to integrate with their in-store order and payment systems. I'm disappointed at the article, because it doesn't talk about any of this stuff; it's just a laundry list of AWS services. That isn't the important stuff; the important stuff is really how they got all this legacy (in-store) and new stuff to work together.
- barbarbar 4y agoI have not experienced any improvements with event driven architecture despite it was advertised with the same wording as this.