6 ms·

Can anyone explain what the actual problem is they're solving using 'edge computing'? I get the feeling this is a completely unnecessary use of technology.

ereyes01 8y ago
They said in the article without getting super-specific that they have low-latency / high-throughput / time sensitive data processing needs in each restaurant. It sounds like they have a whole bunch of sensors tracking their inventory and food preparation pipeline. Other comments here link to other blog posts by them with more details about french fry done-ness deep learning models, for example (wacky but cool). If you had to go across the internet to the cloud at such a high rate from each restaurant, this would probably come with pretty high failure rates and levels of flakiness. It makes sense to me to process the local data firehose in-house and keep the rest of the common high-level stuff in the cloud.
- TickleSteve 8y ago
  I suppose I'm struggling to see how much of a requirement for low-latency data processing exists in a restaurant. There just isn't that much data there unless they're attempting to do real-time control of equipment (in which case, just use a regular control system). I can imagine a requirement for logging potentially, but thats not a low-latency requirement and the use-case for edge-computing is exactly that, low-latency. Still not seeing anything here but a dramatic overuse of technology I'm afraid.
  - coffeesn0b 8y ago
    It’s more about network outages than latency specifically... although in several remote locations latency is permanently low due to slow providers.
    - TickleSteve 8y ago
      But again, latency of what? Requirement for low-latency implies a use of data that is time-sensitive. What time-sensitive data is there in a single chicken restaurant??
      - yazmeya 8y ago
        In a fast food restaurant the latency of food orders being fulfilled (i.e. how fast your food really is) would be a reasonable metric to track.
    - coffeesn0b 8y ago
      Credit card transactions, mobile orders, timer synchronization, order receiving (tablets etc), iot devices (cameras, cooking devices) and other things planned for the future.
      - TickleSteve 8y ago
        Ok, lets go thru that list: - Credit-card transactions don't have low-latency requirements, its 'nice' if they're quick, same as everything else. The bottleneck here is entirely dependent on your ISP tho. No 'edge-computing' here. - Mobile orders. This will necessarily go thru to a central server. This is traditional client-server stuff. Again, no 'edge-compute'. - Order receiving. Simple local data entry. Again, no 'edge-compute'. - IoT devices. Hopefully these have local control systems without the server being in the loop. Control systems are not 'edge-compute' either. 'Edge-compute' is generation of knowledge at the edge rather than shipping raw data to a central server. This reduces required bandwidth. What in your system takes a high rate of data and generates a low-rate of data for transfer to a server for further use. I see no analysis of raw data into a more processed form, this is simply traditional data entry and CRUD activites.
  - ereyes01 8y ago
    I think you are right that this blog post doesn't lay out a specific use case to justify this setup in a restaurant- that does not appear to be the goal of the post. An exploration of problems and requirements is probably what you're looking for. They did drop a few hints, though. They are just hints, and going by what's in the blog post, a bit of imagination is required to fill in some of the gaps. However, if you do that, I can envision a few possible needs for low-latency edge computing that are interesting and forward-looking: - Kitchen automation. Sensors that monitor food being cooked, and help implement a pipeline to ensure quality is consistent. (example here is that neural net monitoring the staleness of fries). - Inventory automation. Monitor and automatically restock items. Track purchases and ingredient usage in real time. - Data analytics. Collect detailed time-series data about food quality, facilities maintenance, foot traffic, noise levels, security, etc. These aren't standard restaurant problems. However, it's an interesting approach to scaling data analysis and automation in the physical world. All of these use cases assume the need for "real-time" data, and they assume that such a thing has business value (which is where I think your critique is coming from). I'm not affiliated with Chick Fil-A, so I don't know how correct any of my speculation is, but that's my takeaway from reading the article.
coffeesn0b 8y ago
Here's the abbreviated reason; Internet connectivity for restaurants stink. So we put our compute at the edge. Trust me... as an SRE I would much prefer we shove it in the cloud and call it a day, but alas... no such luck.
- TickleSteve 8y ago
  Yeah, but what compute? what are they computing? The driving need for edge computing is low-latency. There just isn't the need for it in a restaurant. I do hope they're not doing real-time control of kitchen equipment via something in a container, containers in Linux are nowhere near deterministic enough. This strikes me as someone who is familiar with only web technologies but unfamiliar with control systems attempting to create a control system.... completely inappropriate tools for the job.
  - coffeesn0b 8y ago
    Could you define what is not deterministic about containerized computing? Examples of things we are controlling are timers for food, cameras that can recognize and track food items, drying machines that will automatically trigger and fry fries.
    - TickleSteve 8y ago
      Linux is general is not a real-time system, in that you cannot get deterministic response time from it. For example I'm guessing it doesn't matter if one of your timers are a second late due to Linux deciding to swap a process out or prioritise some upgrade check but imagine if your time constraints were a lot tighter. Similarly if you're attempting to track food on a conveyor belt and Linux decides to prioritise the file indexer on your filesystem... oops, a lettuce has been thrown on the floor. Real-time operating systems exist for a reason.
      - coffeesn0b 8y ago
        Our tolerance requirements for synchronicity are broad enough that we can tolerate blips like this... at the end of the day we are automating away some simple human interactions (ie; fry the fries or track the food), we aren’t performing surgery with these systems. So far we’ve been satisfied with Linux (Ubuntu 18.04 to be exact) and it’s overall capabilities. Also, don’t forget that cost at scale is a big factor... doing things perfectly but expensively is not profitable at 2k locations.
imh 8y ago
>In a recent post, we shared about how we do bare metal clustering for Kubernetes on-the-fly at the Edge in our restaurants. One of the most common (and best) questions we got was “but why?”. In the post we will answer that question. [emphasis mine] Luckily there's a whole article on that topic. I try to avoid "Did you read the article?" comments, but the first three sentences in the article tell you that this is the question they're answering.
- TickleSteve 8y ago
  No, there is some hand-waving about "kitchen equipment" and "low latency". No business case for avoiding more traditional architecture.
  - coffeesn0b 8y ago
    We have frequent internet outages and still need to run compute loads at the edge.
    - symlinkk 8y ago
      > compute loads What are you computing? This is what everyone is wondering.
      - TickleSteve 8y ago
        There is no real edge-compute loads here, just traditional CRUD data entry type systems. Riding the hype-wave methinks....
        coffeesn0b 8y ago
        Iot device control, such as timers for food, and cameras tracking food (to name a few). These also output streams of data that is interesting to us, such that we want to exfiltrate them up to the cloud. I’m not a fan of hype... k8s is one of the few hyped technologies that has delivered on its promises.
arduinomancer 8y ago
I've worked in a fast food restaurant and one of the key pieces of information we used was something called a "drop chart". This was basically a printed out excel sheet with predicted amounts of food to keep ready to serve. i.e. between 5pm-6pm have 12 chicken breasts already cooked. People follow these charts to know how much stock to pre-cook before the hour basically. These predictions are usually based on previous years sales for that same day/time + growth + whatever other inputs to the sheet. It sounds like what they're trying to do is run ML models and make really fast predictions in real time to react to surges of customers. To me this sounds a lot smarter than the approach I worked with. From my experience when the numbers were off you either generate a lot of waste product or you make customers wait really long. I imagine the inputs to the ML model are probably something like real time sales data + whatever else. I think you'd want edge compute here not just for latency but for availability. You'd want to both run these models fast and make sure even if the crappy internet connection of the restaurant dies you can keep providing these numbers.
- TickleSteve 8y ago
  ML implies lots of training data. That gets done centrally as it would need to be done in aggregate. You don't have low-latency training of a model and use that same model in real-time. And in your example (admittedly small scale) 12 items of data an hour is not exactly high-data-rate, not enough to justify racks of machines.
  - coffeesn0b 8y ago
    You’re correct, training is centralized but the devices that use it to function are operated at the edge.
    - TickleSteve 8y ago
      Of course, thats how ML works, central training of a model with use of that model at the edges. So how often do you distribute a new model for controlling fries?
coupdejarnac 8y ago
"Edge" computing seems like a term invented by people who grew up with ubiquitous cloud computing. Go back in time 10 years or so, and all this automation would have been onsite by default. You cannot seriously tell me that creating a predictive model for cooking french fries requires a lot of computing power.
- TickleSteve 8y ago
  Exactly. And the results would have been no different. This is just hype-driven development (HDD).

ereyes01 8y ago

They said in the article without getting super-specific that they have low-latency / high-throughput / time sensitive data processing needs in each restaurant. It sounds like they have a whole bunch of sensors tracking their inventory and food preparation pipeline. Other comments here link to other blog posts by them with more details about french fry done-ness deep learning models, for example (wacky but cool). If you had to go across the internet to the cloud at such a high rate from each restaurant, this would probably come with pretty high failure rates and levels of flakiness. It makes sense to me to process the local data firehose in-house and keep the rest of the common high-level stuff in the cloud.

TickleSteve 8y ago
I suppose I'm struggling to see how much of a requirement for low-latency data processing exists in a restaurant. There just isn't that much data there unless they're attempting to do real-time control of equipment (in which case, just use a regular control system). I can imagine a requirement for logging potentially, but thats not a low-latency requirement and the use-case for edge-computing is exactly that, low-latency. Still not seeing anything here but a dramatic overuse of technology I'm afraid.
- coffeesn0b 8y ago
  It’s more about network outages than latency specifically... although in several remote locations latency is permanently low due to slow providers.
  - TickleSteve 8y ago
    But again, latency of what? Requirement for low-latency implies a use of data that is time-sensitive. What time-sensitive data is there in a single chicken restaurant??
    - yazmeya 8y ago
      In a fast food restaurant the latency of food orders being fulfilled (i.e. how fast your food really is) would be a reasonable metric to track.
  - coffeesn0b 8y ago
    Credit card transactions, mobile orders, timer synchronization, order receiving (tablets etc), iot devices (cameras, cooking devices) and other things planned for the future.
    - TickleSteve 8y ago
      Ok, lets go thru that list: - Credit-card transactions don't have low-latency requirements, its 'nice' if they're quick, same as everything else. The bottleneck here is entirely dependent on your ISP tho. No 'edge-computing' here. - Mobile orders. This will necessarily go thru to a central server. This is traditional client-server stuff. Again, no 'edge-compute'. - Order receiving. Simple local data entry. Again, no 'edge-compute'. - IoT devices. Hopefully these have local control systems without the server being in the loop. Control systems are not 'edge-compute' either. 'Edge-compute' is generation of knowledge at the edge rather than shipping raw data to a central server. This reduces required bandwidth. What in your system takes a high rate of data and generates a low-rate of data for transfer to a server for further use. I see no analysis of raw data into a more processed form, this is simply traditional data entry and CRUD activites.
- ereyes01 8y ago
  I think you are right that this blog post doesn't lay out a specific use case to justify this setup in a restaurant- that does not appear to be the goal of the post. An exploration of problems and requirements is probably what you're looking for. They did drop a few hints, though. They are just hints, and going by what's in the blog post, a bit of imagination is required to fill in some of the gaps. However, if you do that, I can envision a few possible needs for low-latency edge computing that are interesting and forward-looking: - Kitchen automation. Sensors that monitor food being cooked, and help implement a pipeline to ensure quality is consistent. (example here is that neural net monitoring the staleness of fries). - Inventory automation. Monitor and automatically restock items. Track purchases and ingredient usage in real time. - Data analytics. Collect detailed time-series data about food quality, facilities maintenance, foot traffic, noise levels, security, etc. These aren't standard restaurant problems. However, it's an interesting approach to scaling data analysis and automation in the physical world. All of these use cases assume the need for "real-time" data, and they assume that such a thing has business value (which is where I think your critique is coming from). I'm not affiliated with Chick Fil-A, so I don't know how correct any of my speculation is, but that's my takeaway from reading the article.

coffeesn0b 8y ago

Here's the abbreviated reason; Internet connectivity for restaurants stink. So we put our compute at the edge. Trust me... as an SRE I would much prefer we shove it in the cloud and call it a day, but alas... no such luck.

TickleSteve 8y ago
Yeah, but what compute? what are they computing? The driving need for edge computing is low-latency. There just isn't the need for it in a restaurant. I do hope they're not doing real-time control of kitchen equipment via something in a container, containers in Linux are nowhere near deterministic enough. This strikes me as someone who is familiar with only web technologies but unfamiliar with control systems attempting to create a control system.... completely inappropriate tools for the job.
- coffeesn0b 8y ago
  Could you define what is not deterministic about containerized computing? Examples of things we are controlling are timers for food, cameras that can recognize and track food items, drying machines that will automatically trigger and fry fries.
  - TickleSteve 8y ago
    Linux is general is not a real-time system, in that you cannot get deterministic response time from it. For example I'm guessing it doesn't matter if one of your timers are a second late due to Linux deciding to swap a process out or prioritise some upgrade check but imagine if your time constraints were a lot tighter. Similarly if you're attempting to track food on a conveyor belt and Linux decides to prioritise the file indexer on your filesystem... oops, a lettuce has been thrown on the floor. Real-time operating systems exist for a reason.
    - coffeesn0b 8y ago
      Our tolerance requirements for synchronicity are broad enough that we can tolerate blips like this... at the end of the day we are automating away some simple human interactions (ie; fry the fries or track the food), we aren’t performing surgery with these systems. So far we’ve been satisfied with Linux (Ubuntu 18.04 to be exact) and it’s overall capabilities. Also, don’t forget that cost at scale is a big factor... doing things perfectly but expensively is not profitable at 2k locations.

imh 8y ago

>In a recent post, we shared about how we do bare metal clustering for Kubernetes on-the-fly at the Edge in our restaurants. One of the most common (and best) questions we got was “but why?”. In the post we will answer that question. [emphasis mine] Luckily there's a whole article on that topic. I try to avoid "Did you read the article?" comments, but the first three sentences in the article tell you that this is the question they're answering.

TickleSteve 8y ago
No, there is some hand-waving about "kitchen equipment" and "low latency". No business case for avoiding more traditional architecture.
- coffeesn0b 8y ago
  We have frequent internet outages and still need to run compute loads at the edge.
  - symlinkk 8y ago
    > compute loads What are you computing? This is what everyone is wondering.
    - TickleSteve 8y ago
      There is no real edge-compute loads here, just traditional CRUD data entry type systems. Riding the hype-wave methinks....
      - coffeesn0b 8y ago
        Iot device control, such as timers for food, and cameras tracking food (to name a few). These also output streams of data that is interesting to us, such that we want to exfiltrate them up to the cloud. I’m not a fan of hype... k8s is one of the few hyped technologies that has delivered on its promises.

arduinomancer 8y ago

I've worked in a fast food restaurant and one of the key pieces of information we used was something called a "drop chart". This was basically a printed out excel sheet with predicted amounts of food to keep ready to serve. i.e. between 5pm-6pm have 12 chicken breasts already cooked. People follow these charts to know how much stock to pre-cook before the hour basically. These predictions are usually based on previous years sales for that same day/time + growth + whatever other inputs to the sheet. It sounds like what they're trying to do is run ML models and make really fast predictions in real time to react to surges of customers. To me this sounds a lot smarter than the approach I worked with. From my experience when the numbers were off you either generate a lot of waste product or you make customers wait really long. I imagine the inputs to the ML model are probably something like real time sales data + whatever else. I think you'd want edge compute here not just for latency but for availability. You'd want to both run these models fast and make sure even if the crappy internet connection of the restaurant dies you can keep providing these numbers.

TickleSteve 8y ago
ML implies lots of training data. That gets done centrally as it would need to be done in aggregate. You don't have low-latency training of a model and use that same model in real-time. And in your example (admittedly small scale) 12 items of data an hour is not exactly high-data-rate, not enough to justify racks of machines.
- coffeesn0b 8y ago
  You’re correct, training is centralized but the devices that use it to function are operated at the edge.
  - TickleSteve 8y ago
    Of course, thats how ML works, central training of a model with use of that model at the edges. So how often do you distribute a new model for controlling fries?

coupdejarnac 8y ago

"Edge" computing seems like a term invented by people who grew up with ubiquitous cloud computing. Go back in time 10 years or so, and all this automation would have been onsite by default. You cannot seriously tell me that creating a predictive model for cooking french fries requires a lot of computing power.

TickleSteve 8y ago
Exactly. And the results would have been no different. This is just hype-driven development (HDD).