9 ms·

McDonald's Closes Amid Global IT Outage

84 points by ytch 3y ago

usr1106 3y ago
That they have nationwide systems does not surprise me. But that it's a global system does surprise me. Their main market should be the US, but EU data protection is legally mostly incompatible. I am pretty sure they have major violations. For FAANGs that's not news and they have their cases with the commission regularly. But for pretty old brick and mortar restaurant business I am suprised. Edit: Of course the central recipes are globally standard. But otherwise I'd expect the logistics more regional. Especially for Australia mentioned in the article.
- filleokus 3y ago
  I think one of the value propositions / moats / benefit of scale (nowadays) for a mega corp like McDonald's is that the cost of e.g the kiosk software is spread over so tremendously many restaurants. Even if they do follow local regulations and have mostly "isolated deployments" of things touching customer data, they might still have things like a centralised DNS server or update server for kiosk/POS software etc. I would be pretty surprised if anno 2024 an organisation like McDonald's had regional independent SWE/Ops teams building essentially the same thing but for their region.
  - usr1106 3y ago
    Megacorp and megacorp. Here they are small franchises, not owned by McDonald's. Of course using basically the same kiosk software makes sense despite being different localisations and different business owners. Having single points of failure globally not so much. I wonder how business owners take functioning sea cables for granted. We have wars in several places on the planet.
    - greedo 3y ago
      A significant number of McDonalds locations in the US are owned by the McDonalds corporation, not franchisees. And McDonalds wields a heavy hand in "encouraging" franchisees to participate in whatever program they're currently running or implementing. This goes from food introductions, to hardware, to back of the office programs.
      - usr1106 3y ago
        Of course that's a core of franchising: Limiting the choice of business owners. They have to source your stuff even if choices more suitable for them might exist.
- trollied 3y ago
  Why is data protection an issue & what would be a violation? If I walk into one of their establishments & order a burger, the store has zero idea who I am. I just pay, get an order ID & stand there and wait. I don't have to login. There's no PII.
  - bigmattystyles 3y ago
    Did you pay cash?
    - trollied 3y ago
      What's that got to do with anything? Card details aren't stored.
      - reportgunner 3y ago
        Is there no trace of you sending money from your card to McDonalds ? How do they audit for fraud ?
        Stranger43 3y ago
        It's stored by your bank but your credit card or account id is never transferred to the retailer by the payment processors instead what happens is that McDonalds get an unique transaction ID that only your card provider knows was generated by your card.
        reportgunner 3y ago
        So how do they know it was me who sent the money and not some McDonalds money launderer ?
        Stranger43 3y ago
        McDonald's don't your bank does know you authorized an transaction using pin/card/app. The payment terminal is essentially not actually talking directly to the POS except to tell it that transaction id "xyz"(which only your bank/"card issuer" know is you) was approved, everything else is encrypted between terminal and payment processor and not supposed to be visible to the retailer.
- karmakaze 3y ago
  This reminds me of the danger of centralizing without safeguards. Like the time Google was able to fat-finger a network change globally. Similar to writing testable code, config changes should be able to be rolled out incrementally, especially if it's not easily revertable. Otherwise don't over-centralize--you have to be this tall to ride this ride.
usr1106 3y ago
Story time: In Germany in the 1980s an American asked me on the train for a restaurant recommendation in a town of 60,000. I thought hard for a place with good food and made big efforts to describe how to get there. In the end he asked: Is there no McDonalds? We had had one for about 5 years, but I had not considered it a restaurant. It's a different thing :) Right, it's just around the the corner from the train station, 2 minute walk.
- subpixel 3y ago
  On the other hand, I lived in NYC for a long time and was always surprised at how European travelers wound up eating at McDconalds in Chinatown, surrounded by amazing food. I concluded that humans are very conservative when factoring “lunch risk”. The perceived cost of time, money, and disappointment if they don’t find food they like drives them to McDonalds where they know exactly what they are getting and the risk is zero. A Swiss acquaintance of mine was the first to show me how this works first-hand. Lemongrass chicken? Congee? Pho? Let’s just get “a Mac”.
  - usr1106 3y ago
    Yeah, McDonald's and friends have had no general need to close their places because of lack of customers in Europe. Counting locations I think compared to other kinds of restaurants their market share is small. Measured in turnover it might be a different story. Few people might enjoy their meal at a McDonald's for 30-60 minutes.
  - ChicagoBoy11 3y ago
    I'll have a slightly different take on this -- I suspect quite a bit of this might be a form of a "when in Rome" effect. I'm an immigrant myself, and the number of family members for whom going to a "original" McDonald's (and by original all that is meant is a mcdonalds in the land of where mcdonalds is from) is actually a thing. I suspect a significant portion of the folks who do go in as tourists are either in the "well this is just a quick bite that i know what I'll get cause we need to get to X place on time" or the "OMG A REAL* MCDONALDS" types for whom going to one in the US is an experience itself. Whereas we locals might recognize that perhaps the wealth of coming to somewhere like NYC is the incredible offerings available, I know for quite a few folks that the blander fast food experience actually may be the "novelty" they are seeking out!
dusted 3y ago
Burgers.. They are not transported via HTTP (yet), and still, we can't buy them due to IT trouble? Seems easy enough.. You tell what you want, they make it, you hand over your cash money and they hand over the burger.
- ChicagoBoy11 3y ago
  Until it isn't, though. When the thing you are selling is essentially a commodity, when the business model only works because you are balancing some crazy thing margins that rely on all the other gizmos to make work, when the tech supplants the most minimal staff (which is largely seasonal, unskilled, and poorly paid) possible to even keep the lights out, suddenly the fact that this went a bit sideways becomes a big, big problem. It's not like a mom and pop diner who all of a suddenly can't use the handheld CC system.
- turnsout 3y ago
  In 2024, if you can only take cash, you might as well close down until you sort out CC processing.
  - mmcgaha 3y ago
    Where I live we still have restaurants that only take cash. There are even two places nearby where you can walk in 24/7, grab what you want out of their fridge, and put your cash in a box on the way out. Cash only can still work.
    - phone8675309 3y ago
      Where I live those fridges and the cash box would both be empty in a matter of hours.
    - turnsout 3y ago
      Now imagine that instead of a local restaurant, it was a global conglomerate with restaurants in nearly every country
apimade 3y ago
This is all second-hand and probably wrong, but I worked for MuleSoft shortly after McDonalds began to pull the plug on them after they experienced two regional outages. The outage was a matter of minutes, not hours, but payments weren’t affected. It only affected, as far as I know, kiosks - this was before they were everywhere. I can imagine this will cost a few heads at McDonalds who didn’t have a DRP or BCP which included secondary payment infrastructure. Even if you were paying double retail rates and ate the cost for your franchisees, it would have been worth it. For a company so dependent on external vendors, I’m surprised we don’t see more outages. I was always surprised to hear they outsourced their integration infrastructure for the core business. I’m sure it’s like that elsewhere given the number of customer stories floating around for McDonalds. They have the capital, brand, network and infrastructure to be a tech powerhouse. But they’re not. https://www.mastercardservices.com/en/dynamic-yield/case-studies/mcdonald-s-enhances-customer-experience-new-era-digital-menu-decisioning https://www.mastercardservices.com/en/dynamic-yield/case-stu... https://blogs.mulesoft.com/digital-transformation/business/mcdonalds-digital-innovation/ https://blogs.mulesoft.com/digital-transformation/business/m... https://aws.amazon.com/solutions/case-studies/mcdonalds/ https://aws.amazon.com/solutions/case-studies/mcdonalds/ https://www.evoke-creative.com/projects/mcdonalds https://www.evoke-creative.com/projects/mcdonalds https://www.abbyy.com/customer-stories/mcdonalds-relies-on-abbyy-ocr-technology-to-power-mobile-app/ https://www.abbyy.com/customer-stories/mcdonalds-relies-on-a... https://newsroom.accenture.com/news/2023/mcdonald-s-corporation-and-accenture-expand-partnership-to-accelerate-technology-adoption-and-reinvent-customer-and-crew-experiences https://newsroom.accenture.com/news/2023/mcdonald-s-corporat...
- mort96 3y ago
  You call it the "core business", but is it? Isn't McDonald's core business real estate? https://www.wallstreetsurvivor.com/mcdonalds-beyond-the-burger/ https://www.wallstreetsurvivor.com/mcdonalds-beyond-the-burg...
  - joezydeco 3y ago
    They still need to make sure the restaurants on top of them are generating money. Otherwise they'd just build self-storage units on each one.
ChrisArchitect 3y ago
Some more discussion: https://news.ycombinator.com/item?id=39713489 https://news.ycombinator.com/item?id=39713489
FredPret 3y ago
When I was a kid in the 90s, I would excitedly rant to my dad about what a life-and-death issue cybersecurity was. He said "but the computer only does the accounting, nothing is connected to it - business can continue for a while without it." Interesting how you can't even get a hamburger without being connected now. I'm not sure if I'm happy about having been right.
- RGamma 3y ago
  Even hospitals outsource some core infrastructure to the cloud. Cut the internet connection and suddenly they're in the 80s.
  - baud147258 3y ago
    worse, in the 80s, hospitals would know how to work without internet access.
hagbard_c 3y ago
My personal McDonald's data-security history: having been part of Hacking in Progress '97 in the Netherlands where I got a coupon for a free burger at McDonald's. Not being a regular - as in never going to - McDonald's I thought I'd give it a try as there happened to be a McDonald's drive-through in Lelystad which was on my route home. (Maybe this is where I should explain that I was cycling home with a cycle cart behind my bike filled with computer and camping hardware, wearing wooden shoes) I entered the drive-through lane behind a car but when it was my turn the woman behind the counter said 'only cars allowed'. That was close to the first and thus far the last time I went to a McDonald's. It is also a good example of correlation without causation...
password54321 3y ago
And another outage a day later: https://www.bbc.co.uk/news/uk-68584235 https://www.bbc.co.uk/news/uk-68584235 I can't even use an app from my Gym without it lagging / crashing. Many of us predicted this was going to start happening at an increasing rate because of AI and the over-saturation of under-qualified students with CS degrees. It is going to be a bad decade for software.

84 points by ytch 3y ago

That they have nationwide systems does not surprise me. But that it's a global system does surprise me. Their main market should be the US, but EU data protection is legally mostly incompatible. I am pretty sure they have major violations. For FAANGs that's not news and they have their cases with the commission regularly. But for pretty old brick and mortar restaurant business I am suprised. Edit: Of course the central recipes are globally standard. But otherwise I'd expect the logistics more regional. Especially for Australia mentioned in the article.

filleokus 3y ago
I think one of the value propositions / moats / benefit of scale (nowadays) for a mega corp like McDonald's is that the cost of e.g the kiosk software is spread over so tremendously many restaurants. Even if they do follow local regulations and have mostly "isolated deployments" of things touching customer data, they might still have things like a centralised DNS server or update server for kiosk/POS software etc. I would be pretty surprised if anno 2024 an organisation like McDonald's had regional independent SWE/Ops teams building essentially the same thing but for their region.
- usr1106 3y ago
  Megacorp and megacorp. Here they are small franchises, not owned by McDonald's. Of course using basically the same kiosk software makes sense despite being different localisations and different business owners. Having single points of failure globally not so much. I wonder how business owners take functioning sea cables for granted. We have wars in several places on the planet.
  - greedo 3y ago
    A significant number of McDonalds locations in the US are owned by the McDonalds corporation, not franchisees. And McDonalds wields a heavy hand in "encouraging" franchisees to participate in whatever program they're currently running or implementing. This goes from food introductions, to hardware, to back of the office programs.
    - usr1106 3y ago
      Of course that's a core of franchising: Limiting the choice of business owners. They have to source your stuff even if choices more suitable for them might exist.
trollied 3y ago
Why is data protection an issue & what would be a violation? If I walk into one of their establishments & order a burger, the store has zero idea who I am. I just pay, get an order ID & stand there and wait. I don't have to login. There's no PII.
- bigmattystyles 3y ago
  Did you pay cash?
  - trollied 3y ago
    What's that got to do with anything? Card details aren't stored.
    - reportgunner 3y ago
      Is there no trace of you sending money from your card to McDonalds ? How do they audit for fraud ?
      - Stranger43 3y ago
        It's stored by your bank but your credit card or account id is never transferred to the retailer by the payment processors instead what happens is that McDonalds get an unique transaction ID that only your card provider knows was generated by your card.
        reportgunner 3y ago
        So how do they know it was me who sent the money and not some McDonalds money launderer ?
        Stranger43 3y ago
        McDonald's don't your bank does know you authorized an transaction using pin/card/app. The payment terminal is essentially not actually talking directly to the POS except to tell it that transaction id "xyz"(which only your bank/"card issuer" know is you) was approved, everything else is encrypted between terminal and payment processor and not supposed to be visible to the retailer.
karmakaze 3y ago
This reminds me of the danger of centralizing without safeguards. Like the time Google was able to fat-finger a network change globally. Similar to writing testable code, config changes should be able to be rolled out incrementally, especially if it's not easily revertable. Otherwise don't over-centralize--you have to be this tall to ride this ride.

usr1106 3y ago

Story time: In Germany in the 1980s an American asked me on the train for a restaurant recommendation in a town of 60,000. I thought hard for a place with good food and made big efforts to describe how to get there. In the end he asked: Is there no McDonalds? We had had one for about 5 years, but I had not considered it a restaurant. It's a different thing :) Right, it's just around the the corner from the train station, 2 minute walk.

subpixel 3y ago
On the other hand, I lived in NYC for a long time and was always surprised at how European travelers wound up eating at McDconalds in Chinatown, surrounded by amazing food. I concluded that humans are very conservative when factoring “lunch risk”. The perceived cost of time, money, and disappointment if they don’t find food they like drives them to McDonalds where they know exactly what they are getting and the risk is zero. A Swiss acquaintance of mine was the first to show me how this works first-hand. Lemongrass chicken? Congee? Pho? Let’s just get “a Mac”.
- usr1106 3y ago
  Yeah, McDonald's and friends have had no general need to close their places because of lack of customers in Europe. Counting locations I think compared to other kinds of restaurants their market share is small. Measured in turnover it might be a different story. Few people might enjoy their meal at a McDonald's for 30-60 minutes.
- ChicagoBoy11 3y ago
  I'll have a slightly different take on this -- I suspect quite a bit of this might be a form of a "when in Rome" effect. I'm an immigrant myself, and the number of family members for whom going to a "original" McDonald's (and by original all that is meant is a mcdonalds in the land of where mcdonalds is from) is actually a thing. I suspect a significant portion of the folks who do go in as tourists are either in the "well this is just a quick bite that i know what I'll get cause we need to get to X place on time" or the "OMG A REAL* MCDONALDS" types for whom going to one in the US is an experience itself. Whereas we locals might recognize that perhaps the wealth of coming to somewhere like NYC is the incredible offerings available, I know for quite a few folks that the blander fast food experience actually may be the "novelty" they are seeking out!

dusted 3y ago

Burgers.. They are not transported via HTTP (yet), and still, we can't buy them due to IT trouble? Seems easy enough.. You tell what you want, they make it, you hand over your cash money and they hand over the burger.

ChicagoBoy11 3y ago
Until it isn't, though. When the thing you are selling is essentially a commodity, when the business model only works because you are balancing some crazy thing margins that rely on all the other gizmos to make work, when the tech supplants the most minimal staff (which is largely seasonal, unskilled, and poorly paid) possible to even keep the lights out, suddenly the fact that this went a bit sideways becomes a big, big problem. It's not like a mom and pop diner who all of a suddenly can't use the handheld CC system.
turnsout 3y ago
In 2024, if you can only take cash, you might as well close down until you sort out CC processing.
- mmcgaha 3y ago
  Where I live we still have restaurants that only take cash. There are even two places nearby where you can walk in 24/7, grab what you want out of their fridge, and put your cash in a box on the way out. Cash only can still work.
  - phone8675309 3y ago
    Where I live those fridges and the cash box would both be empty in a matter of hours.
  - turnsout 3y ago
    Now imagine that instead of a local restaurant, it was a global conglomerate with restaurants in nearly every country

apimade 3y ago

This is all second-hand and probably wrong, but I worked for MuleSoft shortly after McDonalds began to pull the plug on them after they experienced two regional outages. The outage was a matter of minutes, not hours, but payments weren’t affected. It only affected, as far as I know, kiosks - this was before they were everywhere. I can imagine this will cost a few heads at McDonalds who didn’t have a DRP or BCP which included secondary payment infrastructure. Even if you were paying double retail rates and ate the cost for your franchisees, it would have been worth it. For a company so dependent on external vendors, I’m surprised we don’t see more outages. I was always surprised to hear they outsourced their integration infrastructure for the core business. I’m sure it’s like that elsewhere given the number of customer stories floating around for McDonalds. They have the capital, brand, network and infrastructure to be a tech powerhouse. But they’re not. https://www.mastercardservices.com/en/dynamic-yield/case-studies/mcdonald-s-enhances-customer-experience-new-era-digital-menu-decisioning https://www.mastercardservices.com/en/dynamic-yield/case-stu... https://blogs.mulesoft.com/digital-transformation/business/mcdonalds-digital-innovation/ https://blogs.mulesoft.com/digital-transformation/business/m... https://aws.amazon.com/solutions/case-studies/mcdonalds/ https://aws.amazon.com/solutions/case-studies/mcdonalds/ https://www.evoke-creative.com/projects/mcdonalds https://www.evoke-creative.com/projects/mcdonalds https://www.abbyy.com/customer-stories/mcdonalds-relies-on-abbyy-ocr-technology-to-power-mobile-app/ https://www.abbyy.com/customer-stories/mcdonalds-relies-on-a... https://newsroom.accenture.com/news/2023/mcdonald-s-corporation-and-accenture-expand-partnership-to-accelerate-technology-adoption-and-reinvent-customer-and-crew-experiences https://newsroom.accenture.com/news/2023/mcdonald-s-corporat...

mort96 3y ago
You call it the "core business", but is it? Isn't McDonald's core business real estate? https://www.wallstreetsurvivor.com/mcdonalds-beyond-the-burger/ https://www.wallstreetsurvivor.com/mcdonalds-beyond-the-burg...
- joezydeco 3y ago
  They still need to make sure the restaurants on top of them are generating money. Otherwise they'd just build self-storage units on each one.

ChrisArchitect 3y ago

Some more discussion: https://news.ycombinator.com/item?id=39713489 https://news.ycombinator.com/item?id=39713489

FredPret 3y ago

When I was a kid in the 90s, I would excitedly rant to my dad about what a life-and-death issue cybersecurity was. He said "but the computer only does the accounting, nothing is connected to it - business can continue for a while without it." Interesting how you can't even get a hamburger without being connected now. I'm not sure if I'm happy about having been right.

RGamma 3y ago
Even hospitals outsource some core infrastructure to the cloud. Cut the internet connection and suddenly they're in the 80s.
- baud147258 3y ago
  worse, in the 80s, hospitals would know how to work without internet access.

hagbard_c 3y ago

My personal McDonald's data-security history: having been part of Hacking in Progress '97 in the Netherlands where I got a coupon for a free burger at McDonald's. Not being a regular - as in never going to - McDonald's I thought I'd give it a try as there happened to be a McDonald's drive-through in Lelystad which was on my route home. (Maybe this is where I should explain that I was cycling home with a cycle cart behind my bike filled with computer and camping hardware, wearing wooden shoes) I entered the drive-through lane behind a car but when it was my turn the woman behind the counter said 'only cars allowed'. That was close to the first and thus far the last time I went to a McDonald's. It is also a good example of correlation without causation...

password54321 3y ago

And another outage a day later: https://www.bbc.co.uk/news/uk-68584235 https://www.bbc.co.uk/news/uk-68584235 I can't even use an app from my Gym without it lagging / crashing. Many of us predicted this was going to start happening at an increasing rate because of AI and the over-saturation of under-qualified students with CS degrees. It is going to be a bad decade for software.