7 ms·

Chick Fil A's Edge Enterprise Architecture

93 points by freshrap6 4y ago

ramesh31 4y ago
But why?
- Rebelgecko 4y ago
  The previous article linked in the post goes more into the "Why" https://medium.com/chick-fil-atech/enterprise-restaurant-compute-f5e2fd63d20f https://medium.com/chick-fil-atech/enterprise-restaurant-com...
  - tracker1 4y ago
    https://medium.com/chick-fil-atech/edge-computing-at-chick-fil-a-2621f4b5a969 https://medium.com/chick-fil-atech/edge-computing-at-chick-f...
- deleted 4y ago
  [deleted]
- mike_d 4y ago
  When I was at Walmart the holy grail was edge compute in every store (which I think is now live). There is a mainframe in each store that powers point-of-sale, inventory, etc. But in terms of building modern apps, you couldn't assume 24/7 local connectivity was a thing or that it would be fast enough for what you wanted to do. Making each location its own failure domain was also a huge win. Imagine a cloud outage taking out hundreds of stores.
  - sithadmin 4y ago
    >There is a mainframe in each store Nope. vSphere clusters in every store. Possibly the biggest ‘field’ footprint of vSphere that exists.
    - mike_d 4y ago
      That might be true, but the heart of every store is an AS/400 or whatever IBM now calls the iSeries replacement.
- cdirkx 4y ago
  It sound a lot better than what we had when I worked at a fastfood restaurant: every location remote desktoping to some overloaded server in the regional office, so simple things like printing an inventory list would involve a lot of waiting.
- KennyBlanken 4y ago
  Justifying the most expensive franchise 'take' of any chain in the nation, apparently. They're basically the mob of the fast food industry. They only want $10k from you to start your franchise; they cover all the costs of starting the business. The tradeoff they have the highest percentage take of any franchise.
  - annadar 4y ago
    Almost like they gave you a bunch of capital and they want their money back. Weird huh? "mob of the fast food industry" That's funny.
- shrubble 4y ago
  I remember 10 years ago when my friend asked me to go look at his restaurant client's computer and diagnose/fix if possible. The 486 based PC had a mix of grease and lint/dust on every possible surface including the power supply fan, all cabling and the entirety of the motherboard. It had been placed on a shelf near one of the deep fryers and had run without problems for years. Certainly the other end of the 'long tail' of computing!
- 2d8a875f-39a2-4 4y ago
  HA design for stores is interesting. You want the store to stay up (dispensing goods and accepting payments generally, additionally preparing food in this case) as much as possible. At the same time you want a solution that is cheap to deploy and maintain. Making everything in the store a dumb client is cheap and easy, but also fragile. Doing as much computing in each store (and even on each POS) as possible is great for HA but now you have more complicated hardware and software deployment problems. Different merchants trade these off in different ways. CFA seem to have gone for a lot of computing in the store, and the rest of the design is about mitigating those deployment and maintenance problems and costs. I like the NUC cluster, Gitops, API and support team stories. Am less keen on the K3S deployment per store, seems like a questionable choice of orchestration engine for this scenario but maybe there are details of rest of their store architecture that I'm missing.
- koz1000 4y ago
  Why do all the tech? The goal is to have the ultimate model of just-in-time manufacturing: to never run out of supplies and, without sounding creepy, to know precisely what the customer plans to order and have it in a bag in and in their hands the moment they arrive at the store. There are upstream benefits for marketing to see the feedback of their campaigns in real-time, but this is mostly about keeping inventories low and service times lower. That equates directly to profit. CFA isn't the only chain working on this, but they're the most open about how they're implementing the infrastructure.
silisili 4y ago
[flagged]
- quanticle 4y ago
  It's easy to have a low-effort cynical take about Chick-Fil-A's use of Kubernetes in its restaurants, but the actual post outlining why they chose to provision each restaurant with a 3-node Kubernetes cluster [1] makes a fairly compelling argument. Simply put, they needed to ensure that their point-of-sale terminals had bulletproof reliability, and they wanted to do it in a way that leveraged commodity hardware. The solution they came up with, Kubernetes clusters running on consumer-grade Intel NUCs is a fairly elegant solution to that problem. It reminds me of how Google chose to run on "commodity" Intel servers, rather than "enterprise-grade" hardware from IBM or Sun, realizing that, at scale, failure is guaranteed, no matter how overdesigned your hardware is, and it's more important to design for redundancy, easy provisioning and replacement than it is to design for absolute uptime. Chick-Fil-A has come to a similar realization for their point of sale systems. [1]: https://medium.com/chick-fil-atech/edge-computing-at-chick-fil-a-2621f4b5a969 https://medium.com/chick-fil-atech/edge-computing-at-chick-f...
  - silisili 4y ago
    Low effort is pointing out just one thing that may or may not even help. This doesn't address the machine learning pieces, 100k sensors per restaurant, fryer sensors, everything else that is listed. It's a chicken sandwich restaurant. How much will all of this really buy over typical weekly/monthly/yearly sales trends? It's not just CFA. I for some reason watched a Taco Bell video on LinkedIn that layed out one small problem regarding the ability to accept orders, and how a -completely- new k8s microservice architecture addressed it. This smells a lot of the same, mixed in with more tech overoptimism. GE had the same spiels 10 years ago, with better use cases, and realized most of it was a dead end.
    - llanowarelves 4y ago
      I don't know how Chikfilas are near you, but with the ones I've been to there are often lines of cars wrapping around the building a couple times. And when that happens, there are several Chikfila employees walking car-to-car with tablets, pre-emptively taking and charging orders from customers. Allowing them to batch-make orders from cars that would otherwise have to wait for someone else to get their food to place the order. Not having to have the stores talk to a remote server to record an order during peak load could be a win. Not sure if it justifies "edge computing" but if a fast food restaurant needed it, it'd be them. I've never seen anything like that level of congestion at McDonalds or Taco Bell.
almost_usual 4y ago
Year 2023, FAA systems fail nationwide due to a “corrupt file” and Chick Fil A is highly redundant with a Kubernetes cluster running every store.
- decremental 4y ago
  The private sector will almost always be more efficient than the public sector, but that said, the last time that system in question was in the news it was because it had been given a more "inclusive" name. It speaks to the priorities of the regime.
- deleted 4y ago
  [deleted]
- MBCook 4y ago
  Isn’t NOTAM run by a private company already? I don’t think the FAA runs the actual hardware.
billllll 4y ago
> The goal of the Restaurant Edge Compute platform was to create a robust platform in each restaurant where our DevOps Product teams could deploy and manage applications to help Operators and Team Members keep pace with ever-growing business, whether in the kitchen, the supply chain, or in directly serving customers. > (Previous article) Our hypothesis: By making smarter kitchen equipment we can collect more data. By applying data to our restaurant, we can build more intelligent systems. By building more intelligent systems, we can better scale our business. I must admit, from an outsiders perspective, it really sounds like a bunch of buzzwords justifying a solution in search of a problem. Their examples of forecasting waffle fries reminds me of a failed startup that forecasted how many checkout lines to open via computer vision (which I can't find on Google). In the end, it turned out it was a lot easier for a human manager to simply open a new line when required, and the computer vision provided the wrong forecasting to be accurate. I wonder what CFAs success criteria and metrics are for this project. Tech-wise, wouldn't it be a lot simpler to do a single node, single application that gets updated via something like RAUC? Especially if you have a small team (which they emphasized), it seems to me like adding a Kubernetes cluster at the edge adds complication without much benefit, other than "redundancy" (how redundant is a single rack with the same power source anyways?). Also, how would they get an important security update to the host, if it becomes necessary? It's a lot of nitpicks, but the project overall is very cool. Sounds like they solved a lot of hard tech problems and executed well on the ops.
- FooHentai 4y ago
  Fundamentally what they've deployed here is something a lot of organisations struggle with or don't even recognize as worth having - A reliable edge which you can trust sufficiently to form the functional core of a site. For a restaurant chain this is something worth putting the development effort into because once you've figured it out and ran it for a few years to demonstrate it's reliability you can pitch the shift from a network-optional edge at each site to a network-dependent site with intelligent components hanging off it and depending on it. That's a pathway to having a major competitive advantage in the medium term that your competitors won't be able to put into place overnight once they realize you've left them behind. You can't get there with the amount of effort often put into untrusted edge sites like this - aka a pc in a cupboard. You also can't get there with cloud when the weakest element in the chain is unrealiable site connectivity. They could have done it in a lot of different ways, but going with cheap commodity hardware and avoiding expensive cluster license nonsense (vSphere etc) were smart choices. Spend that money on a compenent centralized tech team rather than vendor shinyness, and you can do a hell of a lot more (and often move faster, to boot).
greatpostman 4y ago
People laughing at chick fil a should look at dominoes stock price. Which has absolutely exploded due to technical innovations (and a new recipe)
pm90 4y ago
They talked about this at a Kubecon iirc. I wasn’t sure if it was an elaborate prank but they were seriously smart folks who patiently explained why they needed to do this and I remember being very impressed.
wcerfgba 4y ago
[flagged]
- deleted 4y ago
  [deleted]
- exogeny 4y ago
  [flagged]
recuter 4y ago
Our hypothesis: By making smarter kitchen equipment we can collect more data. By applying data to our restaurant, we can build more intelligent systems. By building more intelligent systems, we can better scale our business. As a simple example, imagine a forecasting model that attempts to predict how many Waffle Fries (or replace with your favorite Chick-fil-A product) should be cooked over every minute of the day. The forecast is created by an analytics process running in the cloud that uses transaction-level sales data from many restaurants. This forecast can most certainly be produced with a little work. Unfortunately, it is not accurate enough to actually drive food production. Sales in Chick-fil-A restaurants are prone to many traffic spikes and are significantly affected by local events (traffic, sports, weather, etc.). However, if we were to collect data from our point-of-sale system’s keystrokes in real-time to understand current demand, add data from the fryers about work in progress inventory, and then micro-adjust the initial forecast in-restaurant, we would be able to get a much more accurate picture of what we should cook at any given moment. This data can then be used to give a much more intelligent display to a restaurant team member that is responsible for cooking fries (for example), or perhaps to drive cooking automation in the future. Goals like this led us to develop an Internet of Things (IOT) platform for our restaurants. To successfully scale our business we need the ability to 1) collect data and 2) use it to drive automation in the restaurant. The football game next door is over and the home team won? Start extra burgers in anticipation of hungry fans - great. I buy that. The whole thing can be one app running on an iPad with multiple redundant data plans enabled, esims from AT&T and Verizon or whatever. You're going to need a touchscreen tablet for the POS anyway, no need for additional hardware or Kubernetes.
- hgsgm 4y ago
  Do people go out to eat more after wins than losses?
  - recuter 4y ago
    No idea, but I'm sure a super fancy machine learning big data model can run on the iPad/POS itself instead of 2000 Kubernetes clusters.
- 4y ago
jwsteigerwalt 4y ago
Refreshing to see dynamic and nimble solutions from a large organization.
- psychphysic 4y ago
  It's how they dominate. It's all the more important as walk ins and drive thrus reduce while delivers continue to rise.
lovetocode 4y ago
Could we have replaced the EdgeCommander with infrastructure tools like Puppet or Ansible?
rajnathani 4y ago
HN discussion on their initial 2018 blog post: https://news.ycombinator.com/item?id=17820626 https://news.ycombinator.com/item?id=17820626 (570 points, 392 comments)

93 points by freshrap6 4y ago

ramesh31 4y ago

But why?

Rebelgecko 4y ago
The previous article linked in the post goes more into the "Why" https://medium.com/chick-fil-atech/enterprise-restaurant-compute-f5e2fd63d20f https://medium.com/chick-fil-atech/enterprise-restaurant-com...
- tracker1 4y ago
  https://medium.com/chick-fil-atech/edge-computing-at-chick-fil-a-2621f4b5a969 https://medium.com/chick-fil-atech/edge-computing-at-chick-f...
deleted 4y ago
[deleted]
mike_d 4y ago
When I was at Walmart the holy grail was edge compute in every store (which I think is now live). There is a mainframe in each store that powers point-of-sale, inventory, etc. But in terms of building modern apps, you couldn't assume 24/7 local connectivity was a thing or that it would be fast enough for what you wanted to do. Making each location its own failure domain was also a huge win. Imagine a cloud outage taking out hundreds of stores.
- sithadmin 4y ago
  >There is a mainframe in each store Nope. vSphere clusters in every store. Possibly the biggest ‘field’ footprint of vSphere that exists.
  - mike_d 4y ago
    That might be true, but the heart of every store is an AS/400 or whatever IBM now calls the iSeries replacement.
cdirkx 4y ago
It sound a lot better than what we had when I worked at a fastfood restaurant: every location remote desktoping to some overloaded server in the regional office, so simple things like printing an inventory list would involve a lot of waiting.
KennyBlanken 4y ago
Justifying the most expensive franchise 'take' of any chain in the nation, apparently. They're basically the mob of the fast food industry. They only want $10k from you to start your franchise; they cover all the costs of starting the business. The tradeoff they have the highest percentage take of any franchise.
- annadar 4y ago
  Almost like they gave you a bunch of capital and they want their money back. Weird huh? "mob of the fast food industry" That's funny.
shrubble 4y ago
I remember 10 years ago when my friend asked me to go look at his restaurant client's computer and diagnose/fix if possible. The 486 based PC had a mix of grease and lint/dust on every possible surface including the power supply fan, all cabling and the entirety of the motherboard. It had been placed on a shelf near one of the deep fryers and had run without problems for years. Certainly the other end of the 'long tail' of computing!
2d8a875f-39a2-4 4y ago
HA design for stores is interesting. You want the store to stay up (dispensing goods and accepting payments generally, additionally preparing food in this case) as much as possible. At the same time you want a solution that is cheap to deploy and maintain. Making everything in the store a dumb client is cheap and easy, but also fragile. Doing as much computing in each store (and even on each POS) as possible is great for HA but now you have more complicated hardware and software deployment problems. Different merchants trade these off in different ways. CFA seem to have gone for a lot of computing in the store, and the rest of the design is about mitigating those deployment and maintenance problems and costs. I like the NUC cluster, Gitops, API and support team stories. Am less keen on the K3S deployment per store, seems like a questionable choice of orchestration engine for this scenario but maybe there are details of rest of their store architecture that I'm missing.
koz1000 4y ago
Why do all the tech? The goal is to have the ultimate model of just-in-time manufacturing: to never run out of supplies and, without sounding creepy, to know precisely what the customer plans to order and have it in a bag in and in their hands the moment they arrive at the store. There are upstream benefits for marketing to see the feedback of their campaigns in real-time, but this is mostly about keeping inventories low and service times lower. That equates directly to profit. CFA isn't the only chain working on this, but they're the most open about how they're implementing the infrastructure.

silisili 4y ago

[flagged]

quanticle 4y ago
It's easy to have a low-effort cynical take about Chick-Fil-A's use of Kubernetes in its restaurants, but the actual post outlining why they chose to provision each restaurant with a 3-node Kubernetes cluster [1] makes a fairly compelling argument. Simply put, they needed to ensure that their point-of-sale terminals had bulletproof reliability, and they wanted to do it in a way that leveraged commodity hardware. The solution they came up with, Kubernetes clusters running on consumer-grade Intel NUCs is a fairly elegant solution to that problem. It reminds me of how Google chose to run on "commodity" Intel servers, rather than "enterprise-grade" hardware from IBM or Sun, realizing that, at scale, failure is guaranteed, no matter how overdesigned your hardware is, and it's more important to design for redundancy, easy provisioning and replacement than it is to design for absolute uptime. Chick-Fil-A has come to a similar realization for their point of sale systems. [1]: https://medium.com/chick-fil-atech/edge-computing-at-chick-fil-a-2621f4b5a969 https://medium.com/chick-fil-atech/edge-computing-at-chick-f...
- silisili 4y ago
  Low effort is pointing out just one thing that may or may not even help. This doesn't address the machine learning pieces, 100k sensors per restaurant, fryer sensors, everything else that is listed. It's a chicken sandwich restaurant. How much will all of this really buy over typical weekly/monthly/yearly sales trends? It's not just CFA. I for some reason watched a Taco Bell video on LinkedIn that layed out one small problem regarding the ability to accept orders, and how a -completely- new k8s microservice architecture addressed it. This smells a lot of the same, mixed in with more tech overoptimism. GE had the same spiels 10 years ago, with better use cases, and realized most of it was a dead end.
  - llanowarelves 4y ago
    I don't know how Chikfilas are near you, but with the ones I've been to there are often lines of cars wrapping around the building a couple times. And when that happens, there are several Chikfila employees walking car-to-car with tablets, pre-emptively taking and charging orders from customers. Allowing them to batch-make orders from cars that would otherwise have to wait for someone else to get their food to place the order. Not having to have the stores talk to a remote server to record an order during peak load could be a win. Not sure if it justifies "edge computing" but if a fast food restaurant needed it, it'd be them. I've never seen anything like that level of congestion at McDonalds or Taco Bell.

almost_usual 4y ago

Year 2023, FAA systems fail nationwide due to a “corrupt file” and Chick Fil A is highly redundant with a Kubernetes cluster running every store.

decremental 4y ago
The private sector will almost always be more efficient than the public sector, but that said, the last time that system in question was in the news it was because it had been given a more "inclusive" name. It speaks to the priorities of the regime.
deleted 4y ago
[deleted]
MBCook 4y ago
Isn’t NOTAM run by a private company already? I don’t think the FAA runs the actual hardware.

billllll 4y ago

> The goal of the Restaurant Edge Compute platform was to create a robust platform in each restaurant where our DevOps Product teams could deploy and manage applications to help Operators and Team Members keep pace with ever-growing business, whether in the kitchen, the supply chain, or in directly serving customers. > (Previous article) Our hypothesis: By making smarter kitchen equipment we can collect more data. By applying data to our restaurant, we can build more intelligent systems. By building more intelligent systems, we can better scale our business. I must admit, from an outsiders perspective, it really sounds like a bunch of buzzwords justifying a solution in search of a problem. Their examples of forecasting waffle fries reminds me of a failed startup that forecasted how many checkout lines to open via computer vision (which I can't find on Google). In the end, it turned out it was a lot easier for a human manager to simply open a new line when required, and the computer vision provided the wrong forecasting to be accurate. I wonder what CFAs success criteria and metrics are for this project. Tech-wise, wouldn't it be a lot simpler to do a single node, single application that gets updated via something like RAUC? Especially if you have a small team (which they emphasized), it seems to me like adding a Kubernetes cluster at the edge adds complication without much benefit, other than "redundancy" (how redundant is a single rack with the same power source anyways?). Also, how would they get an important security update to the host, if it becomes necessary? It's a lot of nitpicks, but the project overall is very cool. Sounds like they solved a lot of hard tech problems and executed well on the ops.

FooHentai 4y ago
Fundamentally what they've deployed here is something a lot of organisations struggle with or don't even recognize as worth having - A reliable edge which you can trust sufficiently to form the functional core of a site. For a restaurant chain this is something worth putting the development effort into because once you've figured it out and ran it for a few years to demonstrate it's reliability you can pitch the shift from a network-optional edge at each site to a network-dependent site with intelligent components hanging off it and depending on it. That's a pathway to having a major competitive advantage in the medium term that your competitors won't be able to put into place overnight once they realize you've left them behind. You can't get there with the amount of effort often put into untrusted edge sites like this - aka a pc in a cupboard. You also can't get there with cloud when the weakest element in the chain is unrealiable site connectivity. They could have done it in a lot of different ways, but going with cheap commodity hardware and avoiding expensive cluster license nonsense (vSphere etc) were smart choices. Spend that money on a compenent centralized tech team rather than vendor shinyness, and you can do a hell of a lot more (and often move faster, to boot).

greatpostman 4y ago

People laughing at chick fil a should look at dominoes stock price. Which has absolutely exploded due to technical innovations (and a new recipe)

pm90 4y ago

They talked about this at a Kubecon iirc. I wasn’t sure if it was an elaborate prank but they were seriously smart folks who patiently explained why they needed to do this and I remember being very impressed.

wcerfgba 4y ago

[flagged]

deleted 4y ago
[deleted]
exogeny 4y ago
[flagged]

recuter 4y ago

Our hypothesis: By making smarter kitchen equipment we can collect more data. By applying data to our restaurant, we can build more intelligent systems. By building more intelligent systems, we can better scale our business. As a simple example, imagine a forecasting model that attempts to predict how many Waffle Fries (or replace with your favorite Chick-fil-A product) should be cooked over every minute of the day. The forecast is created by an analytics process running in the cloud that uses transaction-level sales data from many restaurants. This forecast can most certainly be produced with a little work. Unfortunately, it is not accurate enough to actually drive food production. Sales in Chick-fil-A restaurants are prone to many traffic spikes and are significantly affected by local events (traffic, sports, weather, etc.). However, if we were to collect data from our point-of-sale system’s keystrokes in real-time to understand current demand, add data from the fryers about work in progress inventory, and then micro-adjust the initial forecast in-restaurant, we would be able to get a much more accurate picture of what we should cook at any given moment. This data can then be used to give a much more intelligent display to a restaurant team member that is responsible for cooking fries (for example), or perhaps to drive cooking automation in the future. Goals like this led us to develop an Internet of Things (IOT) platform for our restaurants. To successfully scale our business we need the ability to 1) collect data and 2) use it to drive automation in the restaurant. The football game next door is over and the home team won? Start extra burgers in anticipation of hungry fans - great. I buy that. The whole thing can be one app running on an iPad with multiple redundant data plans enabled, esims from AT&T and Verizon or whatever. You're going to need a touchscreen tablet for the POS anyway, no need for additional hardware or Kubernetes.

hgsgm 4y ago
Do people go out to eat more after wins than losses?
- recuter 4y ago
  No idea, but I'm sure a super fancy machine learning big data model can run on the iPad/POS itself instead of 2000 Kubernetes clusters.
4y ago

jwsteigerwalt 4y ago

Refreshing to see dynamic and nimble solutions from a large organization.

psychphysic 4y ago
It's how they dominate. It's all the more important as walk ins and drive thrus reduce while delivers continue to rise.

lovetocode 4y ago

Could we have replaced the EdgeCommander with infrastructure tools like Puppet or Ansible?

rajnathani 4y ago

HN discussion on their initial 2018 blog post: https://news.ycombinator.com/item?id=17820626 https://news.ycombinator.com/item?id=17820626 (570 points, 392 comments)