coffeesn0b

6 ms·

1.
by coffeesn0b 7y ago
Is this the apocalypse?
2.
Compassion in Technology: My First Year as an Engineer at Chick-Fil-A(medium.com)
1 points by coffeesn0b 8y ago|1 comments
3.
by coffeesn0b 8y ago
Woot woot! (Biased author here, sorry... can’t help it)
4.
by coffeesn0b 8y ago
We will open source highlander eventually, but it’s not quite there yet. We use a VIP that the NUCs share... ie; one of the three will always have a VIP, and if it dies another NUC grabs it. This is a poor man’s load balancer in that sense,
5.
by coffeesn0b 8y ago
Have you worked with k8s yet? A lot of the things you bring up are addressed if k8s is used properly.
6.
by coffeesn0b 8y ago
Iot device control, such as timers for food, and cameras tracking food (to name a few). These also output streams of data that is interesting to us, such that we want to exfiltrate them up to the cloud. I’m not a fan of hype... k8s is one o
7.
by coffeesn0b 8y ago
You’re correct, training is centralized but the devices that use it to function are operated at the edge.
8.
by coffeesn0b 8y ago
They’re really reasonable about it... you do what you need to do. That being said, it’s really REALLY nice to mute pagerduty for a whole day every week.
9.
by coffeesn0b 8y ago
Credit card transactions, mobile orders, timer synchronization, order receiving (tablets etc), iot devices (cameras, cooking devices) and other things planned for the future.
10.
by coffeesn0b 8y ago
Our tolerance requirements for synchronicity are broad enough that we can tolerate blips like this... at the end of the day we are automating away some simple human interactions (ie; fry the fries or track the food), we aren’t performing su
11.
by coffeesn0b 8y ago
We have frequent internet outages and still need to run compute loads at the edge.
12.
by coffeesn0b 8y ago
It’s more about network outages than latency specifically... although in several remote locations latency is permanently low due to slow providers.
13.
by coffeesn0b 8y ago
Could you define what is not deterministic about containerized computing? Examples of things we are controlling are timers for food, cameras that can recognize and track food items, drying machines that will automatically trigger and fry fr
14.
by coffeesn0b 8y ago
If you decide to play around with it, we use RKE to build and manage the k8s environments locally.
15.
by coffeesn0b 8y ago
That is epic, and i am stealing that name.
16.
by coffeesn0b 8y ago
My coworkers found this one quite amusing.
17.
by coffeesn0b 8y ago
Well, it’s a 6 man team so most of our trade offs were for expedience, not the worlds greatest architecture. We cheated and sync’d data using a HA MongoDB setup amongst the cluster. RKE for K8s clustering was a life saver (nothing easier at
18.
by coffeesn0b 8y ago
The primary challenge was just reasoning with the template and using Helm at scale... ie; what exactly did we deploy on those hundreds of varying clusters? Other issues included; tiller would sometimes become unstable... version mismatch is
19.
by coffeesn0b 8y ago
Ultimately “the edge” will just do it for him :-P
20.
by coffeesn0b 8y ago
This is Caleb (SRE on this project). Here’s a similar NUC we use ... this one is like $300... just an example of what we use; https://www.bhphotovideo.com/c/product/1316113-REG/intel_box...
21.
by coffeesn0b 8y ago
Here's the abbreviated reason; Internet connectivity for restaurants stink. So we put our compute at the edge. Trust me... as an SRE I would much prefer we shove it in the cloud and call it a day, but alas... no such luck.
22.
by coffeesn0b 8y ago
I'd send that feedback through our feedback form... our product teams definitely need to hear that; https://www.chick-fil-a.com/Customer-Service/Contact
23.
by coffeesn0b 8y ago
Thanks! We appreciate the compliment! :)
24.
by coffeesn0b 8y ago
Interesting - why? (this is Caleb, one of the authors and lead SRE for the project)
25.
by coffeesn0b 8y ago
We will have an order of magnitude more cloud datacenters than AWS! :)
26.
by coffeesn0b 8y ago
Hey! Caleb here... I'm the lead SRE building the clusters on the restaurants. Each individual restaurant gets it's own cluster (we aren't fully deployed yet). There are too many network latency challenges and too much immatur
27.
by coffeesn0b 8y ago
We're still in the process of rolling things out, although I'm not sure I can give a percentage... I think it's safe to say that in the next 6-12 months most (if not all) of them should have an active cluster. And no, there i
28.
by coffeesn0b 8y ago
We deal with a challenge of frequent network outages or latency issues... keep in mind that we have locations out in the middle of nowhere with no QoS. There are a variety of loads at the restaurant that require low latency and high up time
29.
by coffeesn0b 8y ago
We haven't open sourced how we do this yet... we have an MVP way of doing it by using Ansible to provision the NUCs, and nmap (please don't laugh!) so that they can find each other on a specific virtual network at the restaurants.
30.
by coffeesn0b 8y ago
Caleb with CFA here - random sleeps aren't exactly the most elegant solution... this was our MVP, and we'll definitely be improving on it.

More ›