4 ms·

Serious question, isn't think just rebuilding the same thing that commercial cloud pubsub offerings already provide?

BoorishBears 4y ago
I mean I'd expect if they're already all-in on AWS they'd just use Firehose with some deduplication instead of whatever home-brew fallback solution they described, but other than that it doesn't seem like they built much? What's impressive to me is that they need all that architecture. Mcdonalds sells under 100 burgers a second from what I can find, their order load is probably bursty, so assume maybe all the orders come in the same third of the day, so 300 per second, and every burger is it's own order... that's still not that much. One order is more than one operation when you're dealing with everything a place at McDonald's scale, but even if you multiply by a factor of 10 to account for analytics, compliance, etc. 3,000 operations per second? Does that really require an entire Kafka-driven event architecture?
- ehnto 4y ago
  I think you are underestimating how liberal some applications are, especially as analytics is one of their requirments. It's probably multiple events per thing you do. I wouldn't be surprised by 100+ events before even placing an order.
  - BoorishBears 4y ago
    Forest for the trees, I already practically doubled my numbers and assumed McDonalds gets all their orders in the same 8 hour period! And even then multiplying them by 100 doesn't get you into the realm of "we couldn't build this on a monolithic horizontally scaling application". If anything, if you're at McDonalds scale and still can't find the engineering skill to build a monolith that can handle 30k operations per second, you're playing with fire building a distributed system. (if you're a nascent startup, then by all means stand on the shoulder of giants and don't sweat that you don't have a full blown cloud engineering org, but that's definitely not where McDonalds should be...)
    - ehnto 4y ago
      Architecture reflects organization structure more often than not. What I see is not an attempt to handle requests rates, but an attempt to service a widely distributed set of applications and handle the inevitable churn of client applications requirements. I am on team monolith, but I also don't see any issue with this approach if you are happy accepting it's caveats and vendor lock in, which they it seems they were.
- andy800 4y ago
  I'd guess that 100x database interactions per order may be more like it. Upon startup, there's a whole login, check app version, payment cards still valid, geo query sequence. Check user's point total, rewards, custom offers. Load menu based on chosen store availability and prices. Every menu item has multiple options (mcnugget sauce, burgers without tomato, type of soda). Add to cart. Remove from cart. Add something different to cart. Repeat. Ok, order ready? Check taxes in local jurisdiction. Adjust total based on offers/rewards. Delivery or pickup? Pickup in-store, curbside, drive-thru? Communicate order to store. Update order status based on customer arrival. Send code to customer. Update status upon pickup/delivery. Lots more in-between I skipped, and that's just user-facing, nothing about analytics, accounting, loyalty club, etc. I'm not defending McDonald's or it's architecture -- as I stated elsewhere, the app is far from perfect, and an entirely different architecture could very easily work much better. But I do think you are severely downplaying the number of interactions or transactions required to run an app like theirs.
  - BoorishBears 4y ago
    I mean you wrote all this justification, but 30k per second is still practically nothing compared to the complexity described in the article? Taking something like Postgres and sprinkling in some strategic use of Redis would handle their usecase with horizontal scaling pretty reliably... What it wouldn't do is let you add Kafka to your resume.
    - andy800 4y ago
      To be clear, I was not justifying the current architecture, I specifically wrote "an entirely different architecture could very easily work much better." I was pointing out, however, that, as is often the case, the initial estimates in a typical "why do they need all this stuff" post, likely underestimated the transaction volume by possibly 10x. Perhaps 3000 or 30000 transactions per second could run on the same system -- I'm not an expert at that scale. But I doubt you'd find any Fortune 100 company relying solely on Postgres and Redis.
      - BoorishBears 4y ago

I mean I'd expect if they're already all-in on AWS they'd just use Firehose with some deduplication instead of whatever home-brew fallback solution they described, but other than that it doesn't seem like they built much? What's impressive to me is that they need all that architecture. Mcdonalds sells under 100 burgers a second from what I can find, their order load is probably bursty, so assume maybe all the orders come in the same third of the day, so 300 per second, and every burger is it's own order... that's still not that much. One order is more than one operation when you're dealing with everything a place at McDonald's scale, but even if you multiply by a factor of 10 to account for analytics, compliance, etc. 3,000 operations per second? Does that really require an entire Kafka-driven event architecture?

ehnto 4y ago
I think you are underestimating how liberal some applications are, especially as analytics is one of their requirments. It's probably multiple events per thing you do. I wouldn't be surprised by 100+ events before even placing an order.
- BoorishBears 4y ago
  Forest for the trees, I already practically doubled my numbers and assumed McDonalds gets all their orders in the same 8 hour period! And even then multiplying them by 100 doesn't get you into the realm of "we couldn't build this on a monolithic horizontally scaling application". If anything, if you're at McDonalds scale and still can't find the engineering skill to build a monolith that can handle 30k operations per second, you're playing with fire building a distributed system. (if you're a nascent startup, then by all means stand on the shoulder of giants and don't sweat that you don't have a full blown cloud engineering org, but that's definitely not where McDonalds should be...)
  - ehnto 4y ago
    Architecture reflects organization structure more often than not. What I see is not an attempt to handle requests rates, but an attempt to service a widely distributed set of applications and handle the inevitable churn of client applications requirements. I am on team monolith, but I also don't see any issue with this approach if you are happy accepting it's caveats and vendor lock in, which they it seems they were.
andy800 4y ago
I'd guess that 100x database interactions per order may be more like it. Upon startup, there's a whole login, check app version, payment cards still valid, geo query sequence. Check user's point total, rewards, custom offers. Load menu based on chosen store availability and prices. Every menu item has multiple options (mcnugget sauce, burgers without tomato, type of soda). Add to cart. Remove from cart. Add something different to cart. Repeat. Ok, order ready? Check taxes in local jurisdiction. Adjust total based on offers/rewards. Delivery or pickup? Pickup in-store, curbside, drive-thru? Communicate order to store. Update order status based on customer arrival. Send code to customer. Update status upon pickup/delivery. Lots more in-between I skipped, and that's just user-facing, nothing about analytics, accounting, loyalty club, etc. I'm not defending McDonald's or it's architecture -- as I stated elsewhere, the app is far from perfect, and an entirely different architecture could very easily work much better. But I do think you are severely downplaying the number of interactions or transactions required to run an app like theirs.
- BoorishBears 4y ago
  I mean you wrote all this justification, but 30k per second is still practically nothing compared to the complexity described in the article? Taking something like Postgres and sprinkling in some strategic use of Redis would handle their usecase with horizontal scaling pretty reliably... What it wouldn't do is let you add Kafka to your resume.
  - andy800 4y ago
    To be clear, I was not justifying the current architecture, I specifically wrote "an entirely different architecture could very easily work much better." I was pointing out, however, that, as is often the case, the initial estimates in a typical "why do they need all this stuff" post, likely underestimated the transaction volume by possibly 10x. Perhaps 3000 or 30000 transactions per second could run on the same system -- I'm not an expert at that scale. But I doubt you'd find any Fortune 100 company relying solely on Postgres and Redis.
    - BoorishBears 4y ago