12 ms·
Here's some detail, as someone that has worked with those machines before (internally and externally). In the good old days the soft serve and shake machines w
by joezydeco 6y ago
Here's some detail, as someone that has worked with those machines before (internally and externally).
In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. Then it was reassembled in the morning. This would leave a lot of room for error and possible contamination in the process. And the sight of a shake machine hitting full pressure with a misaligned O-ring in the barrel is a lovely one indeed.
So the newer machines take a different route: they self-pasturize. The machine goes out of service and slowly heats everything up to a bacteria-killing temperature (including the dairy mix inside), then cools everything back down to freezing temperature to serve.
This process takes about four to five hours. But if you set the clock wrong, the machine will go into its cycle at some super-inconvenient time. There are other triggers that can force a self-clean but in general it's not "broken", it's just doing this process at a bad time for you.
I'm curious how this data is being sourced. The last I checked there was an effort to put cloud-based telemetry into every unit and have them report home constantly (both to the equipment maker and to McD), but that was never intended to be public data.
EDIT: Ah, I didn't RTFT. He's using the public ordering site to scrape it (and now the title has changed). But that does mean there's some working connection from the equipment to the McCloud now. That's cool. Or maybe a manager is doing it by hand.
- deleted 6y ago[deleted]
- ehsankia 6y agoAt first I was gonna ask why does this not run on off hours, but then you mention bad clock, so the next question is, why is the % of bad clock so high, and why isn't the clock also cloud based or set in a way that can go wrong. Next up, what about redundancy, do stores only have one machine?
- thisisnico 6y agoCloud based clock. Don't you mean NTP? We've had this technology for years! If it's connected to the internet, it's a pretty fundamental piece of pretty much any networked OS. You can ensure correct time this way.
- Dylan16807 6y agoNTP is fine but so is pulling time off a server header if it's phoning home to report status.
- nitrogen 6y agoNot that it matters for ice cream, but NTP can compensate for latency better than HTTPx can.
- thisisnico 6y agoIt's also easy to implement since it will be integrated into the OS of the ice cream machine already.
- ehsankia 6y agoNTP is what I meant, yes.
- ezconnect 6y agoGPS based clock is better no need for internet connection. Seiko and other company has been doing this for years.
- marshray 6y agoNTP is typically backed by GPS. If you have a datacenter full of computers, obviously you don't want to run a separate GPS antenna for every system. NTP allows systems to synchronize to GPS over a network.
- Scoundreller 6y agoNeed a clear sky connection though.
- arkitaip 6y agoThe clock probably resets when the power goes out or the fuse blows. They would need a battery as a backup to keep the time current.
- joezydeco 6y agoMcD is conservative when it comes to kitchen technology. With certain exceptions, franchisees determine when they buy new equipment and not the corporation. Most of it is designed for a 7-year capex cycle. But to answer the question: no, the machines aren't net enabled and do not NNTP their clocks. But that's changing. But I still wouldn't design my architecture to let a central server decide when a machine should go out of service. Think about what else you can fuck up that way. Why are they off? Good question. Sometimes it's as simple as a store owner setting the heat cycle for 2pm instead of 2am. Why are they not redundant? They're expensive, take up a lot of valuable space and a lot more valuable energy, they need to be filled with 2x the product, and the customer demand isn't that high at any given moment to warrant the expense in most stores.
- nrmitchi 6y ago> Next up, what about redundancy, do stores only have one machine? From my (old) experience working at one, yes. The annoyance of having to maintain a second machine (especially when cleaning and maintenance involved a physical person cleaning things) was not worth keeping extra 9's of uptime.
- ehsankia 6y agoWell again, looking at the graph, I'm seeing 10-20% down time, so it's not "just an extra 9"
- msandford 6y agoI think you might find that in a certain way it is. 90% to 99% is an extra nine. /ducks
- xboxnolifes 6y agoOr even zero 9s (80%) to one 9 (90%).
- reaperducer 6y agowhy is the % of bad clock so high For the same reason that a certain % of orders are wrong. When you pay peanuts, you get monkeys.
- antris 6y agoCalling fast food workers monkeys won't make you look any smarter.
- reaperducer 6y agoI'm not. It's a very common business phrase about cheap labor, even here on HN. The fact that you equate the two says more about you than me.
- javajosh 6y agoI've not heard that term used, and its bad. You shouldn't dehumanize people for being "cheap labor". Honestly, that guy flipping burgers is doing more real good for people than any number of highly paid software devs working on VC funded projects that will mostly end up in the trash.
- ahartmetz 6y agoI think it's more acceptable to make fun of people of your level of privilege and up (like the management who underpays) and less of those lower. If a fast food worker at a better paying, better run chain or restaurant said it, it would be still a little mean (as intended), but not kicking down.
- xeromal 6y agoyup. They call it punching up. No one likes someone who punches down. lol
- Dylan16807 6y agoFour to five hours, so let's say that happens daily and machines are out of commission 20% of the time. Currently it's early afternoon and the site says that over 8% are out. That's about halfway between "all overnight" and "all at completely random times". Are almost half of the clocks set wrong? Assuming that the actual number of broken machines is relatively small.
- shadowgovt 6y agoI'd be willing to wager that half the clocks might be wrong. It's something manager-on-duty can control, so Murphy's Law kicks in.
- dcolkitt 6y agoA standalone GPS NTP server is $750 retail. Commercial soft serve ice cream machines are at least $4000. Just building the machines with an autonomous GPS clock that doesn't need to be set, would double the revenue but only increase the upfront cost by 18%.
- K0balt 6y agoIf it was being integrated, the cost would be more like 5 dollars, since the ntp unit could just be an esp8266 chip that connects to the store wifi to sync the time.
- rkagerer 6y agoIf they can't manage to set the time what makes you think they'll connect the Wifi? (...although in fairness some of the other comments here suggest the things might already have some kind of int[er/ra]net connectivity in which case no extra hardware required)
- Nexxxeh 6y agoHave them log onto the free public McDonalds WiFi on first boot, and auto-provision themselves to the correct private WiFi?
- d23 6y agoSince you and the OP are here, I have a random question: were the blizzards ever made with liquid plastic ingredients? It probably sounds insane, but someone from a plastics company spoke at my high school and said this was the case since it was easier to ship and needed no refrigeration. Now, decades later, I'm wondering if this was just complete nonsense or had an element of truth to it. I've been able to find nothing online.
- fossuser 6y agoMaybe you mean McFlurry (Blizzards were Dairy Queen). Basically the same thing, but not sure if whoever said that was specifically talking about Blizzards.
- d23 6y agoAh yes, McFlurries. Got my wires crossed.
- jonny_eh 6y agohttps://www.snopes.com/fact-check/a-fair-shake/ https://www.snopes.com/fact-check/a-fair-shake/ Mcflurry ingredients: https://www.mcdonalds.com/us/en-us/product/mcflurry-with-oreo-cookies.html https://www.mcdonalds.com/us/en-us/product/mcflurry-with-ore... (no mention of plastic)
- shakna 6y ago> Mcflurry ingredients: https://www.mcdonalds.com/us/en-us/product/mcflurry-with-ore https://www.mcdonalds.com/us/en-us/product/mcflurry-with-ore... (no mention of plastic) That's actually incorrect, they list two plastics there. It's just that "plastic" isn't something that is actually necessarily toxic, and the media misuse the terminology all the time. The two are: > Cellulose Gum, Guar Gum
- Judgmentality 6y agoNot the same, but plastic-y chemicals have been used in fast food before. https://www.eater.com/2016/8/8/12403338/subway-yoga-mat-chemical-mcdonalds-burger-king-wendys https://www.eater.com/2016/8/8/12403338/subway-yoga-mat-chem... https://www.npr.org/sections/thesalt/2014/03/06/286886095/almost-500-foods-contain-the-yoga-mat-compound-should-we-care-keep https://www.npr.org/sections/thesalt/2014/03/06/286886095/al...
- punnerud 6y agoSomeone should save the data over time and figure what is wrong with every machine. If this is made public and enough tech people tell the store every day to change example from 2pm to 2am, this will likely result in a better product for everyone.
- codetrotter 6y ago> McCloud That’s an awesome way to name their cloud computing stuff that fits the brand really well. Is this an official/semiofficial name?
- realityking 6y agoImagine McDonalds had invented AWS and called it McCloud.
- javajosh 6y agoWith the tagline "There can be only one."
- codetrotter 6y agoWhen you order a VPS they would’ve asked you then: “Do you want to supersize your order? For $1 extra you can have a whole bucket of RAM instead of a small cup of RAM”
- nullsense 6y agoImagine KFC had invented it and called it Kentucky Fried Cloud
- bonestamp2 6y agoThen later rebranded it as "KFC"... the C is for Chaos.
- nsxwolf 6y agoCleaning these things was my second most hated job when I worked in the industry. But deep fryers are a far more dreadful task.
- balls187 6y agoWorked at BK. #1 hated task was cleaning the broiler. #2 cleaning the shake machine
- trelliscoded 6y agoA manager is doing it by hand. Their POS system has the ability to lock out individual items, and this information is available in real time to the server backing the mobile app (GMA). The machines that I've seen don't have real-time telemetry, and are not networked.
- joezydeco 6y agoGood info, thank you! There was an old effort a long time ago where the machines were on a powerline-networking system (LONWorks) but that is dead and buried. The next-gen effort was ramping up a few years ago and I don't really know where it is now. The predictive analytics company they bought was a big part of it.
- chapium 6y agoNot sure if this is the same thing, but I recall reading a case study in 2006 on McD's cloudifying their kitchen equipment. The project was apparently a huge failure and was eventually stopped due to runaway costs.
- imglorp 6y agoMeanwhile, Chick-fil-a is running Kubernetes at the edge for equipment, POS telemetry, and ML. https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at...
- homero 6y agoChick-fil-a is ahead of the game here, they recently put edge compute into every store
- oh-4-fucks-sake 6y agoSaid another way: - You have a system whereby every instance has a 75%-79% uptime. - Each instance's 21%-25% daily scheduled maintenance window can (generally) be arbitrarily-scheduled by the ops team. How do you near guarantee that there's always one instance available? If you said: "manage a redundant cluster of at least 2 instances with non-overlapping maintenance windows" Then you were correct! BONUS QUESTIONS: Q: What's the max number of ice cream machines found in a McDonald's location? Q: What's the cluster size of the McDonald's ice cream monitoring system? Q: How much money does McDonald's have? Q: Why the fuck? If you said: "1", "at least 3", "metric fuck-tons", and "I don't know", respectively, then you were correct!
- throwaway373438 6y agoThe answer to #4 is probably that the goal isn't to always have ice cream. The goal is to run a profitable business. Doubling the cost of the ice cream infrastructure to avoid a 25% downtime may not be a profitable venture -- especially if the downtime is scheduled to occur while the store is closed.
- dmix 6y agoShelf-space is in high-demand and those machines aren't cheap.
- newforms 6y agoMcDonalds is in the real estate business (letting buildings), fast food is just a side order.
- justinclift 6y agoAren't the stores franchise's, with the individual stores owned by the relevant franchisee?
- maxerickson 6y agoThe operations are mostly franchised, but the corp owns many of the buildings and plots of lands that they are located on. https://www.mcdonalds.com/us/en-us/about-us/franchising/real-estate.html https://www.mcdonalds.com/us/en-us/about-us/franchising/real...
- imwillofficial 6y agoI actually froze my hand inside an ice cream machine like this when I was a teenager working fast food. I hit a standby button instead of the off switch. (I was too tall to see the off switch and was never trained.) I spent two weeks in a burn unit while they healed my hand with magic synthetic sin grafts. Today you can’t even tell which hand got frozen in there.
- pathseeker 6y agoWoah, how cold are these things that you would need skin grafts like that?
- HenryBemis 6y agoBack in the day I worked in a dairy company as an IT guy. When the RFID antennas would need a reboot I had to go to the deep freeze storage (-40 Celsius), walk in with a ladder, climb the ladder, switch off and then on (IT crowd style), wait to see the green light bleeping, and then climb down and leave as fast as I could. The whole thing would take 40-50 seconds (I was counting EVERY time). I couldn't wear gloves, because ladder/switch. I was wearing all other gear though. Touching anything (aluminum ladders get cold very fast at -40)(so are walls, switches, everything). I did this process a couple of times wearing just jeans and tshirt. In that "less than a minute" my clothes would freeze solid, the wool was hardning and it felt like wearing a plank. Same with jeans. In other cold related story, I was in a "cold country", out in the open, -25 Celsius, my phone rang, my gloves didn't have the electro-thingie so I couldn't "slide to answer" so I too my glove off, answered, talked for 60 seconds. Then I went to a warm pub, and after 10mins I could bend my fingers again. Yes cold can .... you up badly in seconds.
- maaaaattttt 6y agoI used to walk to school in sometimes very cold weather (-35 Celsius). Once, in order to try preventing the inside of my nose to freeze, I put my scarf over it. The scarf froze very quickly and I ended up with ice around my neck... Now that I’m thinking about it, I wonder how people will manage masks this year with COVID in cold countries. TL;DR: don’t put a scarf over your nose in very cold temperatures.
- EdgarVerona 6y agoI worked there in the old days when I was in high school (late 90's), and one of the most disgusting stories I have about a job is when I went to refill the orange juice dispenser machine, and a giant swarm of flies were all over and in the top of the machine. In the orange juice, all along the sides of the inside, and flying around like a small swarm when I opened it. I yelped and stepped back. A manager rushed over to me, slammed the lid shut, and told me to not tell anyone. I quit a couple weeks later.
- fingerlocks 6y agoFruit flies or house flies? The difference matters.
- xeromal 6y agoDid you work in Oakbrook? I was on a team that was trying to make a few smart restaurants. We got to about 50 before the project was canned but we were trying to put everything from fryer oil life to what the platens were cooking and HVAC all on a touch screen. It was a lot of fun.
- joezydeco 6y agoI’ve worked for a few commercial equipment makers. Talked to Oakbrook and Romeoville but never been inside. Every time McDs changes CEOs I feel bad because that means almost every current project is going to be shitcanned as the new person tries to make their mark. Are you still there now?
- xeromal 6y agoHaha, nah, the project was canned and I moved on to greener pastures.
- DrScump 6y agoIn the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. I was a McD manager in my youth. Not only were the internals fully disassembled and sanitized, they were left to soak in the solution overnight. All that was left behind was the cylindrical plastic chamber, sanitized and left to air dry overnight. The messiest part was lubricating the O-rings with a petroleum jelly. Failure to remove all that when cleaning seemed like a good vector for microbes. The worst job of all was cleaning the fryer vats.
- joezydeco 6y agoHeh! I did both of those jobs as well. I still remember the large plastic tray you had to use to organize all the little parts and O-rings. Did you ever have a clever employee that thought they could use the 5-gallon plastic buckets as a quicker way to dispose of the old fryer shortening? I watched that happen one Sunday morning. That was amazing.