9 ms·

If you want to see the actual hardware used at the restaurant, check out the previous blog post: https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-ch

If you want to see the actual hardware used at the restaurant, check out the previous blog post: https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at... Spoiler: It's a stack of Intel NUCs.

mmt 8y ago
Previously discussed at https://news.ycombinator.com/item?id=17448089 https://news.ycombinator.com/item?id=17448089
ocdtrekkie 8y ago
NUCs are surprisingly well-built little machines. I have one in my car, and I give them my absolute seal of approval. I've heard of people running little VMware clusters on them too.
- gbear605 8y ago
  What is it running in your car?
  - ocdtrekkie 8y ago
    I have a custom piece of software that powers a display in the car, syncs and plays my music, and logs GPS. It also can manage audio and video recording and is intended to handle some future features[1] I have planned, so I wanted to over-spec rather than under-spec. [1] It's the same software that runs my home automation system, so an inevitable feature is my car and home units interacting, I wrote half of a piece of navigation software before I stopped, and the entire interface is intended for voice control so I need to throw that in there at some point too. I intend to work on a CAN interface to connect to the car, but I've had a few roadblocks to getting started on that project.
    - dvtrn 8y ago
      Have you documented any of this or written on your home automation endeavors? I'm looking to "Tony Stark" my house up a bit (i.e. voice control, some touch enabled panes in the kitchen/office) but want a bit more control than something purpose built, off-the-shelf while also being a bit more insulated than Alexa or Echo. The NUC has been on my radar as a device that I may go with as the brains of the operation, and I'm just curious to read what design and implementation patterns others have taken.
      - ocdtrekkie 8y ago
        I get asked to do more of a "write-up" a lot, but it's hard to figure out how to format or frame it. I have a fairly large pile of opinions about how one should go about doing this, but it's hard to concisely organize my thoughts on it into a coherent write-up. The code is here: https://github.com/ocdtrekkie/HAController https://github.com/ocdtrekkie/HAController but I don't know if I'd really recommend others use it. The main perk to me is that it's designed around what I want and use (I tried a lot of alternative options before rolling my own), so unless you also really like Visual Basic code and have a brain ordered bizarrely similarly to mine, you may want to start somewhere else. I'd love to chat about what I've learned along the way or what ideas you have (because I might borrow them!), if you want to chat elsewhere hit me up at inbox (at) jacobweisz (dot) com
    - wenc 8y ago
      What's the power draw like? I once left the reading lights on; car was supposed to have turned it off when the doors locked, but somehow didn't. I found myself calling AAA in the morning to get a jumpstart.
      - ochoseis 8y ago
        Also curious, does GP have some way to shutdown gracefully when the car turns off, and reboot fast when it turns on? You could potentially handle this with a laptop and some battery-settings but not aware of an external battery solution for the NUC.
- SheinhardtWigCo 8y ago
  Why do you have one in your car?
  - ocdtrekkie 8y ago
    Answered here: https://news.ycombinator.com/item?id=17822015 https://news.ycombinator.com/item?id=17822015
- 086421357909764 8y ago
  It's actually suprising that people think they're not "normal" computers. I provisioned a few of them and they're little beasts. Most in use are I7, 16-32gb ram, Dual NVME. The Skulltrail nucs are extremely powerfull and on the lower end the atom based units are solid too. I also ran a nuc in my car for a while, mostly stumbling radio / wifi spectrum as I drove around, but it was perfect with its relatively low power requirements.
  - uncreativebrian 8y ago
    This is Brian -- I wrote the article. Super cool use for a NUC! I'm going to follow the links and check out what you did some more.
  - tormeh 8y ago
    But is it real, 6-core 3GHz+ i7? Or is it this mobile crap Intel has been pushing lately?
- starkruzr 8y ago
  If they had OOBM of some kind I would ditch all my hardware at home and replace it with them.
  - deleted 8y ago
    [deleted]
- dnautics 8y ago
  they've kind of a pain in the butt for us... For example, it would be nice for them to have remote admin, more than one NIC, not freak out when the HDMI is not plugged in, etc.
  - ocdtrekkie 8y ago
    I use my NUC without HDMI plugged in, and I assume Chick-fil-A does as well. ;) They do actually make a dual NIC NUC, but it's a little overkill for most uses: https://www.amazon.com/Intel-Machine-NUC8i7HVK-Radeon-Graphics/dp/B07BR5GK1V https://www.amazon.com/Intel-Machine-NUC8i7HVK-Radeon-Graphi... On the other hand, my NUC has plenty of USB 3 ports, and an Ethernet dongle will run you about $7.
    - dnautics 8y ago
      Hmm we did buy NUCs about two years ago so I suppose it's possible things have improved
- protomyth 8y ago
  Nice machines, fast enough, VESA mount to the back of a monitor, and easy to upgrade. They've basically replaced Mac minis and iMacs in our labs. Have not had one bit of trouble with any of them.
trhway 8y ago
Interesting as it seemed that smarter edge/iot nodes would be like ARM/RaspPi, yet it seems it may get even as high as (or just straight to) Intel nuc.
- walrus01 8y ago
  Raspi cannot be used for a semi serious server application due to microsd card write wear issues. All other methods of attaching external storage by USB bus to an rpi are not reliable or suitable for this sort of purpose. Nowhere near the longevity of something with a native sata or m.2 SSD interface and a real SSD.
  - api 8y ago
    Does that remain true if you grossly over-provision the SD card? Use a 64GB card for 1GB of data and you have a lot of wear leveling space.
    - walrus01 8y ago
      The write lifespan in DWPD on any sd card is a tiny fraction of the expected lifespan of even a cheap $55 consumer SSD. Part of the problem is that even if you only write 1GB per day the card is highly likely to reuse the same sectors over and over.
      - jessaustin 8y ago
        ...the card is highly likely to reuse the same sectors over and over. One would have expected this to be an area of focus for RasPi developers? Surely some driver could be updated to avoid this?
        walrus01 8y ago
        The Linux kernel in raspbian treats the SD card like a normal block device. I'm not aware of any special optimization for write wear leveling at the OS level, or on the board's sd card controller.
reaperducer 8y ago
Spoiler: It's a stack of Intel NUCs. Funny how the world has changed. I got my first taste of real computing from a guy in the next town who would buy dead PDP's from local McDonald's restaurants, fix them up, and sell them on. He always said the hardest part wasn't replacing broken parts, it was that invariably the machines would always have soda spilled in them, no matter where they were stored. Cleaning the insides was his most time-consuming task. One of the advantages of an Intel NUC is that it probably has a lot more horsepower than an old PDP/11. It's probably also a smaller target for wayward beverages.
- colonelpopcorn 8y ago
  Used to work for a very large POS system provider, and I can confirm that soda spills make repairs take 10x longer.
- mynameishere 8y ago
  I'd like to hear the story of why a McDonalds would need a PDP. Weren't they...expensive? They would be at the heart of university's IT department, or running a steel mill. Stuff like that. But until PCs came out, at a restaurant I can only picture registers and a dumb terminal hooked up to a distant mainframe.
  - lovich 8y ago
    The logistics improvements from early computers were massive. If you compare the equivalent price nowadays it's not worth it compared to other options, but going from paper to computing wasn't just an incremental improvement but allowed for entirely new abilities. It wasn't like having an account to crunch all the numbers on staff full time was cheap. Especially when you'd have to have 3-4 accountants just to have 24 hour service, regardless of the work load
  - patrickg_zill 8y ago
    McDonald's corporate has or used to have a royalty fee for each item sold. That's a lot of tracking of sales info in the early days...
- 8y ago
zzzeek 8y ago
I'm just trying to understand, I see three of those machines, inside a pretty nice rack, with the switches, routers, a keyboard, I would assume some fairly high quality hard drive / RAID type of thing etc. then the blog post claims "We run our Edge infrastructure on commodity hardware that costs us, ballpark, $1000/restaurant. ". Just the intel nuc I can find on newegg alone costs $349 each, with no memory or harddrive installed, retail. I know they are likely getting amazing volume discounts but it still seems amazing that whole stack could possibly be only $1000 each.
- coffeesn0b 8y ago
  This is Caleb (SRE on this project). Here’s a similar NUC we use ... this one is like $300... just an example of what we use; https://www.bhphotovideo.com/c/product/1316113-REG/intel_boxnuc7i3bnh_7th_generation_intel_core.html?ap=y&gclid=Cj0KCQjwk_TbBRDsARIsAALJSObWJD1G2qYgSmCUF5wgGwwwo5ScdixUiY_M0l0FOcdVb1q4Egv6IMEaAilxEALw_wcB&smp=y https://www.bhphotovideo.com/c/product/1316113-REG/intel_box...
  - dnautics 8y ago
    Just curious (since I work in edge computing products) the deploy here seems rather complicated... What are your feelings on the matter and is there a market for simpler deployments?
  - therealdrag0 8y ago
    Caleb, I'm working on a project with a similar architecture (on-prem services). I'm curious what your auth-pattern looks like as it's something we've struggled with. Mostly the balance between being convenient and being secure. It seems like each site needs an api key to access your cloud. Do you have an auth-pattern that would prevent the key from being readily available to an attacker who got access to the machine? Or do you kinda just say if they get access to the machine, then it's game over? Also with shoddy networks it is difficult to be confident that key-rotation will happen successfully. Happy to get your thoughts. Cheers.
- mi100hael 8y ago
  I'd assume it's 3x$349 that they're referring to in the article, since most places these days already have some networking equipment (probably including a racked switch someplace) in order to provide guest wifi, connect POS terminals, etc. So that portion isn't directly tied to this effort.

mmt 8y ago

Previously discussed at https://news.ycombinator.com/item?id=17448089 https://news.ycombinator.com/item?id=17448089

ocdtrekkie 8y ago

NUCs are surprisingly well-built little machines. I have one in my car, and I give them my absolute seal of approval. I've heard of people running little VMware clusters on them too.

gbear605 8y ago
What is it running in your car?
- ocdtrekkie 8y ago
  I have a custom piece of software that powers a display in the car, syncs and plays my music, and logs GPS. It also can manage audio and video recording and is intended to handle some future features[1] I have planned, so I wanted to over-spec rather than under-spec. [1] It's the same software that runs my home automation system, so an inevitable feature is my car and home units interacting, I wrote half of a piece of navigation software before I stopped, and the entire interface is intended for voice control so I need to throw that in there at some point too. I intend to work on a CAN interface to connect to the car, but I've had a few roadblocks to getting started on that project.
  - dvtrn 8y ago
    Have you documented any of this or written on your home automation endeavors? I'm looking to "Tony Stark" my house up a bit (i.e. voice control, some touch enabled panes in the kitchen/office) but want a bit more control than something purpose built, off-the-shelf while also being a bit more insulated than Alexa or Echo. The NUC has been on my radar as a device that I may go with as the brains of the operation, and I'm just curious to read what design and implementation patterns others have taken.
    - ocdtrekkie 8y ago
      I get asked to do more of a "write-up" a lot, but it's hard to figure out how to format or frame it. I have a fairly large pile of opinions about how one should go about doing this, but it's hard to concisely organize my thoughts on it into a coherent write-up. The code is here: https://github.com/ocdtrekkie/HAController https://github.com/ocdtrekkie/HAController but I don't know if I'd really recommend others use it. The main perk to me is that it's designed around what I want and use (I tried a lot of alternative options before rolling my own), so unless you also really like Visual Basic code and have a brain ordered bizarrely similarly to mine, you may want to start somewhere else. I'd love to chat about what I've learned along the way or what ideas you have (because I might borrow them!), if you want to chat elsewhere hit me up at inbox (at) jacobweisz (dot) com
  - wenc 8y ago
    What's the power draw like? I once left the reading lights on; car was supposed to have turned it off when the doors locked, but somehow didn't. I found myself calling AAA in the morning to get a jumpstart.
    - ochoseis 8y ago
      Also curious, does GP have some way to shutdown gracefully when the car turns off, and reboot fast when it turns on? You could potentially handle this with a laptop and some battery-settings but not aware of an external battery solution for the NUC.
SheinhardtWigCo 8y ago
Why do you have one in your car?
- ocdtrekkie 8y ago
  Answered here: https://news.ycombinator.com/item?id=17822015 https://news.ycombinator.com/item?id=17822015
086421357909764 8y ago
It's actually suprising that people think they're not "normal" computers. I provisioned a few of them and they're little beasts. Most in use are I7, 16-32gb ram, Dual NVME. The Skulltrail nucs are extremely powerfull and on the lower end the atom based units are solid too. I also ran a nuc in my car for a while, mostly stumbling radio / wifi spectrum as I drove around, but it was perfect with its relatively low power requirements.
- uncreativebrian 8y ago
  This is Brian -- I wrote the article. Super cool use for a NUC! I'm going to follow the links and check out what you did some more.
- tormeh 8y ago
  But is it real, 6-core 3GHz+ i7? Or is it this mobile crap Intel has been pushing lately?
starkruzr 8y ago
If they had OOBM of some kind I would ditch all my hardware at home and replace it with them.
- deleted 8y ago
  [deleted]
dnautics 8y ago
they've kind of a pain in the butt for us... For example, it would be nice for them to have remote admin, more than one NIC, not freak out when the HDMI is not plugged in, etc.
- ocdtrekkie 8y ago
  I use my NUC without HDMI plugged in, and I assume Chick-fil-A does as well. ;) They do actually make a dual NIC NUC, but it's a little overkill for most uses: https://www.amazon.com/Intel-Machine-NUC8i7HVK-Radeon-Graphics/dp/B07BR5GK1V https://www.amazon.com/Intel-Machine-NUC8i7HVK-Radeon-Graphi... On the other hand, my NUC has plenty of USB 3 ports, and an Ethernet dongle will run you about $7.
  - dnautics 8y ago
    Hmm we did buy NUCs about two years ago so I suppose it's possible things have improved
protomyth 8y ago
Nice machines, fast enough, VESA mount to the back of a monitor, and easy to upgrade. They've basically replaced Mac minis and iMacs in our labs. Have not had one bit of trouble with any of them.

trhway 8y ago

Interesting as it seemed that smarter edge/iot nodes would be like ARM/RaspPi, yet it seems it may get even as high as (or just straight to) Intel nuc.

walrus01 8y ago
Raspi cannot be used for a semi serious server application due to microsd card write wear issues. All other methods of attaching external storage by USB bus to an rpi are not reliable or suitable for this sort of purpose. Nowhere near the longevity of something with a native sata or m.2 SSD interface and a real SSD.
- api 8y ago
  Does that remain true if you grossly over-provision the SD card? Use a 64GB card for 1GB of data and you have a lot of wear leveling space.
  - walrus01 8y ago
    The write lifespan in DWPD on any sd card is a tiny fraction of the expected lifespan of even a cheap $55 consumer SSD. Part of the problem is that even if you only write 1GB per day the card is highly likely to reuse the same sectors over and over.
    - jessaustin 8y ago
      ...the card is highly likely to reuse the same sectors over and over. One would have expected this to be an area of focus for RasPi developers? Surely some driver could be updated to avoid this?
      - walrus01 8y ago
        The Linux kernel in raspbian treats the SD card like a normal block device. I'm not aware of any special optimization for write wear leveling at the OS level, or on the board's sd card controller.

reaperducer 8y ago

Spoiler: It's a stack of Intel NUCs. Funny how the world has changed. I got my first taste of real computing from a guy in the next town who would buy dead PDP's from local McDonald's restaurants, fix them up, and sell them on. He always said the hardest part wasn't replacing broken parts, it was that invariably the machines would always have soda spilled in them, no matter where they were stored. Cleaning the insides was his most time-consuming task. One of the advantages of an Intel NUC is that it probably has a lot more horsepower than an old PDP/11. It's probably also a smaller target for wayward beverages.

colonelpopcorn 8y ago
Used to work for a very large POS system provider, and I can confirm that soda spills make repairs take 10x longer.
mynameishere 8y ago
I'd like to hear the story of why a McDonalds would need a PDP. Weren't they...expensive? They would be at the heart of university's IT department, or running a steel mill. Stuff like that. But until PCs came out, at a restaurant I can only picture registers and a dumb terminal hooked up to a distant mainframe.
- lovich 8y ago
  The logistics improvements from early computers were massive. If you compare the equivalent price nowadays it's not worth it compared to other options, but going from paper to computing wasn't just an incremental improvement but allowed for entirely new abilities. It wasn't like having an account to crunch all the numbers on staff full time was cheap. Especially when you'd have to have 3-4 accountants just to have 24 hour service, regardless of the work load
- patrickg_zill 8y ago
  McDonald's corporate has or used to have a royalty fee for each item sold. That's a lot of tracking of sales info in the early days...
8y ago

zzzeek 8y ago

I'm just trying to understand, I see three of those machines, inside a pretty nice rack, with the switches, routers, a keyboard, I would assume some fairly high quality hard drive / RAID type of thing etc. then the blog post claims "We run our Edge infrastructure on commodity hardware that costs us, ballpark, $1000/restaurant. ". Just the intel nuc I can find on newegg alone costs $349 each, with no memory or harddrive installed, retail. I know they are likely getting amazing volume discounts but it still seems amazing that whole stack could possibly be only $1000 each.

coffeesn0b 8y ago
This is Caleb (SRE on this project). Here’s a similar NUC we use ... this one is like $300... just an example of what we use; https://www.bhphotovideo.com/c/product/1316113-REG/intel_boxnuc7i3bnh_7th_generation_intel_core.html?ap=y&gclid=Cj0KCQjwk_TbBRDsARIsAALJSObWJD1G2qYgSmCUF5wgGwwwo5ScdixUiY_M0l0FOcdVb1q4Egv6IMEaAilxEALw_wcB&smp=y https://www.bhphotovideo.com/c/product/1316113-REG/intel_box...
- dnautics 8y ago
  Just curious (since I work in edge computing products) the deploy here seems rather complicated... What are your feelings on the matter and is there a market for simpler deployments?
- therealdrag0 8y ago
  Caleb, I'm working on a project with a similar architecture (on-prem services). I'm curious what your auth-pattern looks like as it's something we've struggled with. Mostly the balance between being convenient and being secure. It seems like each site needs an api key to access your cloud. Do you have an auth-pattern that would prevent the key from being readily available to an attacker who got access to the machine? Or do you kinda just say if they get access to the machine, then it's game over? Also with shoddy networks it is difficult to be confident that key-rotation will happen successfully. Happy to get your thoughts. Cheers.
mi100hael 8y ago
I'd assume it's 3x$349 that they're referring to in the article, since most places these days already have some networking equipment (probably including a racked switch someplace) in order to provide guest wifi, connect POS terminals, etc. So that portion isn't directly tied to this effort.