10 ms·

When will M&S take online orders again?

89 points by fredley 1y ago

nickdothutton 1y ago
I don't believe most (pre-internet) retailers should be building and operating their own sites. They already run core supply chain, distribution, and certain other apps (e.g. rostering and so on, accounting and payroll), but they probably shouldn't even be running some of those either.
- fredoralive 1y ago
  M&S tried that, Amazon used to run the website: https://www.theguardian.com/technology/2005/apr/19/business.marksspencer https://www.theguardian.com/technology/2005/apr/19/business.... But they eventually took control back, so it clearly didn't work for them: https://www.theguardian.com/business/2014/feb/18/marks-spencer-control-online-shopping-website-amazon https://www.theguardian.com/business/2014/feb/18/marks-spenc... M&S orders still use the same ###-#######-####### order number format as Amazon, so I'm not sure if it's still some sort of fork of whatever white-label Amazon technology they were using back then. I'm not sure if getting Amazon to run your own ecomerce website is really the greatest idea in the long term (Amazon kinda want your customers to use Amazon, not your website), but M&S using them isn't as mad as that bit in the early 2000's where Waterstone's website was just a subsection of Amazon.co.uk.
  - spacebanana7 1y ago
    > I'm not sure if getting Amazon to run your own e-commerce website is really the greatest idea in the long term Amazon has a clear conflict of interest with anyone in e-commerce. Shopify is probably a better example.
    - pinoy420 1y ago
      [dead]
  - xp84 1y ago
    Famously, Borders and Target both allowed Amazon to run their ecommerce operations on this side of the pond, until they realized what a bad idea it is to partner with your competitor on something important. Target can be forgiven, I suppose, as in those days Amazon was mainly a store for books, CDs, and DVDs. Unclear how Borders didn't see it coming, though!
    - fredoralive 1y ago
      The Waterstone’s example I gave is similar to Borders - if your not from the UK you might not know it’s a bookshop, and they were basically just routing traffic to Amazon as a glorified affiliate link for a while.
  - youngtaff 1y ago
    M&S’ online store is outsourced to someone like Tata, Publicis Sapient etc and last time I looked was built on one of the commercially available platforms (DemandWare and the like) Warehousing and delivery is probably contracted out to another third-party My guess is it’s one of these that’s been hacked
- arp242 1y ago
  This incident has little to do with website or web store as such, and the only reason those are impacted is because pretty much all of M&S's IT systems have been impacted. Even if someone else would be running all of that, chances are that would still interface with the M&S computer systems to accurately get inventory information and the like.
- deleted 1y ago
  [deleted]
- neepi 1y ago
  Err they were breached most likely through Tata Consultancy's helpdesk apparently which is literally the people they outsourced it to. Their approach was to sell the UK operation to Tata in 2018 and piss everyone off until they leave and replace them with Indian staff to save costs over time. You get what you pay for. They're now paying for it.
  - dangus 1y ago
    Honestly, reading your comment I'm not sure how I can be generous enough with it to not consider the main purpose of it to be xenophobia/racism/nationalism. As an example of Tata’s general competency, Tata owns Jaguar Range Rover group which reported their best profit in a decade for the fiscal year ending March 31. It’s certainly very possible for any help desk to be an attack vector regardless of the nationality of the employee. So it seems to me the main point of your comment is to hate on an Indian company solely for being Indian and to stereotype Indian companies as low quality and low cost. Tata Consultancy is a global company that includes offices in places like Chicago, Dallas, and Atlanta.
    - neepi 1y ago
      The point is if you're outsourcing to India you want low cost and are not concerned with quality. That's exactly why you go there. And India has plenty of that to give. No one there wants to work for those outsourcing chop shops, but it's an easy way into the industry so staff turnover is high as people move up the chain. So if you're outsourcing for cost cutting reasons, which they are, it's a constant cycle of low quality staff. That is the risk. Nothing to do with race or xenophobia. You put low costs above risk provision. We can all pick and choose good stories. What about Tata Steel's total mismanagement of Tata Steel Europe? And JLR isn't exactly in good shape as you say it is. You just picked some numbers that sound good. And wrapped it in a xenophobia straw man.
      - d1sxeyes 1y ago
        > The point is if you're outsourcing to India you want low cost and are not concerned with quality First part is a reasonable assumption, the second is not, and this is what’s opening you up to allegations of xenophobia. The allure (and promise) of outsourcing is the idea that you can pay less for a comparable service due to the cost of living disparity between your location and the outsourcing provider. Whether any individual provider achieves that or not is another question, but saying “if your service is provided from India you are not concerned about quality”, or “if your service is provided from India you will have a constant cycle of low quality staff” does sound a lot like xenophobia. > No one there wants to work for those outsourcing chop shops This is simply not true. There are a lot of benefits to working for a company like this, although as with any company, it’s not all upside. Of course turnover is high because you have a lot of entry level folks. Regardless of where you are in the world, no-one wants to work a level one help desk until they retire. > We can all pick and choose good stories. What about Tata Steel's total mismanagement of Tata Steel Europe? And JLR isn't exactly in good shape as you say it is. You just picked some numbers that sound good. Yeah odd example, JLR is not doing great, and Tata Steel is also struggling, but overall the Tata Group are doing well.
- benjaminwootton 1y ago
  They outsource as much as they can to the cheapest system integrators they can find, primarily TCS.
- ecshafer 1y ago
  This is the core thesis of a company like Shopify. Shopify will run everything else about being an e-commerce company (website, inventory, shipping, returns, ads, sales channels, etc) and then the merchant can focus on selling their product. But this is part of the larger thesis about running a business you hear in business school classes, to focus on your specialization and outsource your non-core expertise. Buy Workday/ADP/Paychex don't do payroll or HR. Don't build a data center, buy AWS/Azure/GCP. Don't build a sales database or marketing get Hubspot or Salesforce. Does your company take in a lot of mail? Outsource to a company that specializes in processing mail. Outsource your Technical Helpdesk. Outsource your customer support. This is why componentization is accelerating.
  - madeofpalk 1y ago
    I guess the question is whether e-commerce should be a core competency of a business with a significant e-commerce business. I’m not sure what it’s like in the US, but grocery delivery is a reasonably big deal in the UK.
  - runako 1y ago
    > to focus on your specialization and outsource your non-core expertise Most retailers will argue that connecting with their core customers and delivering delightful experiences to them is their core expertise. More practically, it will be tension between things like "our marketing department wants X on the site for summer" and "Shopify is planning on launching X in January." It will be less of a resistance to using a third-party provider and more that the third-party provider imposes constraints on the mode of contact with customers. That's a hard pill to swallow for a lot of consumer-focused companies.
    - xp84 1y ago
      > Most retailers will argue that connecting with their core customers and delivering delightful experiences to them is their core expertise. Having worked in e-commerce for most of my career, for individual retailers, I can assure you that the perpetual tension you describe is real. The problem as I see it is, every little retailer thinks that their two-bit designers and product managers are so uniquely visionary in designing interactions that they rightfully should have full control over the product that is the ecommerce website. Shopify employs God-knows-how-many engineers to build and maintain this experience, and probably thousands of SREs to be there 24/7 making sure a random DDOS or slow query doesn't take your site out. "But we think we can build a better site than Shopify with 10 engineers and a couple of managers," they say. They can build one that has the 3 cute whiz-bang features that their self-important product design staff thinks matter, but it will be unreliable, and they won't have sufficient expertise to get right the other 90% of what a "good" ecom site should have. And on top of it all, none of those gimmicks will likely improve conversion or order value enough to be worth doing. The smarter ones IMHO do use Shopify. It lacks so many things in its core that it's infuriating (decent search, any nontrivial filtering), but retailers who use it mostly patch over those flaws with plugins sold by third parties (which often introduce ghastly single points of failure that you have no visibility into, and you can't sue some random plugin vendor you pay $50 a month for your site going down on Black Friday). Ecommerce is hard tbh. But I do personally think that most of my previous employers probably should have done lightweight Shopify skins and made their core competence sourcing, merchandising, and advertising product rather than designing cute search filters, or their own product recommendations algorithm.
- dangus 1y ago
  This take doesn't make sense. If one of your core businesses is selling clothes online, and you're a large enough entity, you should write your own software to sell clothes online. Basically by your exact same logic you're asking Walmart and Target to outsource their websites, which is completely insane.
  - crop_rotation 1y ago
    > Basically by your exact same logic you're asking Walmart and Target to outsource their websites, which is completely insane. No, because Walmart website being down for months is just unthinkable. If a company is so incompetent/resource deficient/use your favorite phrase to describe it/ that their e-commerce is down for a month and outlook is it will be down for more time, then something is seriously wrong with the company. Such companies are 100% going to have a much better experience with Shopify. > If one of your core businesses is selling clothes online, and you're a large enough entity, you should write your own software to sell clothes online. If only this was so easy. For various reasons writing non trivial software is hard, and unless these companies can make some structural changes to hire and retain very good engineers (which also is hard for various reasons for these companies), they simply have no chance of doing better than shopify.
    - dangus 1y ago
      The site being down for months is just the effect of this breach, it doesn't mean that the company made the wrong risk assessment by choosing to write their own software in the first place. If I buy a car and I happen to crash it on the first day I drove it home, that doesn't mean I made the wrong choice to buy the car that day. I still bought the car based on the best information I had at the time. M&S basically lost a lottery ticket type of bad luck scenario where they are dealing with a breach that is far, far worse than a typical data breach's impact. Remember when the PlayStation Network was down for over a month due to a very serious breach? That breach didn't prove that PlayStation should have used some kind of external provider for its online services. In fact, an external provider for that sort o thing is not even practical for their business. Remember, there's an alternate timeline where Shopify itself could be breached in a similarly severe way and also go down for 3 months. It's very unlikely but it's possible. If it can happen to M&S it could happen to Shopify.
softwaredoug 1y ago
At the same time we’re talking about AI replacing developers we also see cases like this of organizational technical incompetency. How does one square those two realities?
- bradly 1y ago
  > How does one square those two realities? People eat terrible food because they are bombarded with messages to do so. People can use terrible software for the same reasons. It doesn't matter that the food tastes worse than it used to–food companies are having record profits.
- jacobsenscott 1y ago
  AI replacing devs talk is about short term stock pumping and short term COGS reduction. The long tail is someone else's problem.
  - School-Cotton 1y ago
    I’m pretty sure devs are not usually counted as part of COGS.
- imhoguy 1y ago
  We just need one event of C*O of critical/big company bragging about firing engineering and replacing it with AI and then followed by huge cyberattack like that. Then see how AI balloon pops across news outlets.
  - coliveira 1y ago
    They'll respond saying they need to invest the entire revenue of the company on new data centers to fix the issue, and the stock will double in price.
  - chgs 1y ago
    The market rewards failure. Look at crowdstrike. You aren’t buying technical service or even technical assurance. You are buying someone to blame so the stakeholders don’t hold you accountable.
benjaminwootton 1y ago
How can it take 3-4 months to get an eCommerce site back online? I assume you could redeploy everything from scratch in less time if you have source code and release assets. With backups and failover sites I can’t think of any world where this would happen?
- pavel_lishin 1y ago
  > with backups and failover sites What a fun pair of assumptions!
- internetter 1y ago
  There are no backups. There are no failovers. There is no git. There is no orchestration and deployment stratagies. Programmers ssh into the server and edit code there. Years and years of patchwork on top of patchwork with closely coupled code. Such is a taste of what needs to be done if you wish to have a service that takes months to set back up after any disruption.
  - 98codes 1y ago
    [citation needed]
    - internetter 1y ago
      Sorry if I phrased it poorly. I wasn’t definitively saying that all these things are the case. But what always is the case is that when an attack takes down an organization for months, it was employing a tremendous number of horrendous practices. My list was supposed to be some. M&S isn’t down for months because of something innocuous like a full security audit. As a public company losing tens of millions of dollars a week, their only priority is to stop the bleed, even if that means a hasty partial restoration. The fact they can’t even do that suggests they did stuff terribly wrong. There’s an infinite amount of things I didn’t list that could also be the case. Like if Amazon gave them proprietary blobs they lost after the attack and Amazon won’t provide again. But no matter what they are, things were wrong beyond belief. That is a given.
woah 1y ago
Holy shit why don't they just set up a Shopify
- wavemode 1y ago
  Bureaucracy is almost always the reason. They don't just need a website, they need -their- website back, because it was programmed with a million little business rules and pricing logic and regulatory requirements.
  - xp84 1y ago
    You're surely not wrong that a lot of things would need to be done without, but I'd like to think that if I were 'king of M&S' I could have identified a subset of merchandise that could be loaded into a suitable interim solution like Shopify within say, 4 weeks, if the only other option was forgoing all online sales for 12 weeks +. That would also take a lot of the pressure off of the "full recovery team." Of course, the real situation must be 100x more complex than I'm imagining it so "I'd like to think" != "I am confident"
    - NoImmatureAdHom 1y ago
      Four weeks? You mean two days??? The real situation is not 100x more complex, it's just that this is happening in Britain where everything is someone else's job and no one has any reason to care about the actual goals and everyone will go home at 5pm. Or, more likely, to the pub.
      - xp84 1y ago
        You have to keep in mind that you'd have to build out actual fulfillment capabilities on a brand new platform (so you need training of a ton of staff on a new system, as well as a bunch of new equipment to print orders from the "temporary" systems in the warehouses/stores hopefully without removing the original systems they're trying to restore), plus who knows what the database backups looked like. Plus inventory is going to be all screwy with sales happening to some degree in stores (what POS is that using? Is it decrementing inventory we can use?) I'd sure not bet my reputation on having a shoppable site up in less than several weeks without knowing if I have at minimum, reasonably accurate catalog and inventory data.
deleted 1y ago
[deleted]
wyager 1y ago
It's weird to me how it often seems like the US and China are the only countries capable of mega-scale tech infrastructure like this (and even then, only in some industries). Can you imagine Wal-mart's website going down for multiple months? I think a lot of companies (especially in Europe) have not internalized that, yes, you actually do need to expend apparently exorbitant amounts of money on highly-paid engineers if you want your tech to actually be good. Many countries, including the UK, are simply not wealthy enough to do it at scale. They produce plenty of engineers, but most of the ones capable of holding complicated stuff together probably end up working for US companies that can pay them market rates.
- tristor 1y ago
  With the case of M&S, and in many other cases in UK tech history that have gone poorly, it's mostly examples of the failure of hiring outside consultancies in India to do everything. Business executives continuously fall afoul of the fungibility myth. They believe that engineers are fungible, and that they should therefore simply pay for the cheapest engineers possible that meet the "requirements" on paper, usually set by someone who is not an engineer (HR, project manager, or a lower ranked middle-manager). Time and time and time again we have seen major failures globally, and especially in the UK, that prove that there is no fungibility of engineers, and that outsourcing the critical technical infrastructure for your core systems and services is doomed to failure. They'd rather save a dollar today and lose ten million dollars tomorrow by damaging their national economy and sending more money to India. India's GDP is basically entirely propped up by tech services, and most of that is /failed service delivery/, hard to differentiate from frauds and scams at scale.
  - theuppermiddle 1y ago
    Just because consultants are from India does not make them incompetent. Not to mention the attack vector was social engineering, which can happen to any person of any nationality. 70% of India's GDP is domestic consumption. 7% of the GDP is service export, of which only 3.5% is software services. While significant, it certainly is not "propping" up India's GDP.
- 1y ago
ryanbrick05 1y ago
[dead]
mystified5016 1y ago
> “There is no change to our strategy and our longer-term plans to reshape M&S for growth and, if anything, the incident allows us to accelerate the pace of change as we draw a line and move on.” I wonder if people like this ever hear themselves talking.
ojhughes 1y ago
M&S outsourced over half of its IT team to Tata Consultancy in 2018. Looks like they are now reaping what they sowed https://www.bbc.co.uk/news/business-42629522.amp https://www.bbc.co.uk/news/business-42629522.amp
- mrcsharp 1y ago
  So they saved £30 million a year and now losing estimated £40 a week (not to mention reputation and future opportunity loss due to lower customer confidence). They have almost wiped out all the savings. If this is the reason, then this was a very bad deal for them.

89 points by fredley 1y ago

I don't believe most (pre-internet) retailers should be building and operating their own sites. They already run core supply chain, distribution, and certain other apps (e.g. rostering and so on, accounting and payroll), but they probably shouldn't even be running some of those either.

fredoralive 1y ago
M&S tried that, Amazon used to run the website: https://www.theguardian.com/technology/2005/apr/19/business.marksspencer https://www.theguardian.com/technology/2005/apr/19/business.... But they eventually took control back, so it clearly didn't work for them: https://www.theguardian.com/business/2014/feb/18/marks-spencer-control-online-shopping-website-amazon https://www.theguardian.com/business/2014/feb/18/marks-spenc... M&S orders still use the same ###-#######-####### order number format as Amazon, so I'm not sure if it's still some sort of fork of whatever white-label Amazon technology they were using back then. I'm not sure if getting Amazon to run your own ecomerce website is really the greatest idea in the long term (Amazon kinda want your customers to use Amazon, not your website), but M&S using them isn't as mad as that bit in the early 2000's where Waterstone's website was just a subsection of Amazon.co.uk.
- spacebanana7 1y ago
  > I'm not sure if getting Amazon to run your own e-commerce website is really the greatest idea in the long term Amazon has a clear conflict of interest with anyone in e-commerce. Shopify is probably a better example.
  - pinoy420 1y ago
    [dead]
- xp84 1y ago
  Famously, Borders and Target both allowed Amazon to run their ecommerce operations on this side of the pond, until they realized what a bad idea it is to partner with your competitor on something important. Target can be forgiven, I suppose, as in those days Amazon was mainly a store for books, CDs, and DVDs. Unclear how Borders didn't see it coming, though!
  - fredoralive 1y ago
    The Waterstone’s example I gave is similar to Borders - if your not from the UK you might not know it’s a bookshop, and they were basically just routing traffic to Amazon as a glorified affiliate link for a while.
- youngtaff 1y ago
  M&S’ online store is outsourced to someone like Tata, Publicis Sapient etc and last time I looked was built on one of the commercially available platforms (DemandWare and the like) Warehousing and delivery is probably contracted out to another third-party My guess is it’s one of these that’s been hacked
arp242 1y ago
This incident has little to do with website or web store as such, and the only reason those are impacted is because pretty much all of M&S's IT systems have been impacted. Even if someone else would be running all of that, chances are that would still interface with the M&S computer systems to accurately get inventory information and the like.
deleted 1y ago
[deleted]
neepi 1y ago
Err they were breached most likely through Tata Consultancy's helpdesk apparently which is literally the people they outsourced it to. Their approach was to sell the UK operation to Tata in 2018 and piss everyone off until they leave and replace them with Indian staff to save costs over time. You get what you pay for. They're now paying for it.
- dangus 1y ago
  Honestly, reading your comment I'm not sure how I can be generous enough with it to not consider the main purpose of it to be xenophobia/racism/nationalism. As an example of Tata’s general competency, Tata owns Jaguar Range Rover group which reported their best profit in a decade for the fiscal year ending March 31. It’s certainly very possible for any help desk to be an attack vector regardless of the nationality of the employee. So it seems to me the main point of your comment is to hate on an Indian company solely for being Indian and to stereotype Indian companies as low quality and low cost. Tata Consultancy is a global company that includes offices in places like Chicago, Dallas, and Atlanta.
  - neepi 1y ago
    The point is if you're outsourcing to India you want low cost and are not concerned with quality. That's exactly why you go there. And India has plenty of that to give. No one there wants to work for those outsourcing chop shops, but it's an easy way into the industry so staff turnover is high as people move up the chain. So if you're outsourcing for cost cutting reasons, which they are, it's a constant cycle of low quality staff. That is the risk. Nothing to do with race or xenophobia. You put low costs above risk provision. We can all pick and choose good stories. What about Tata Steel's total mismanagement of Tata Steel Europe? And JLR isn't exactly in good shape as you say it is. You just picked some numbers that sound good. And wrapped it in a xenophobia straw man.
    - d1sxeyes 1y ago
      > The point is if you're outsourcing to India you want low cost and are not concerned with quality First part is a reasonable assumption, the second is not, and this is what’s opening you up to allegations of xenophobia. The allure (and promise) of outsourcing is the idea that you can pay less for a comparable service due to the cost of living disparity between your location and the outsourcing provider. Whether any individual provider achieves that or not is another question, but saying “if your service is provided from India you are not concerned about quality”, or “if your service is provided from India you will have a constant cycle of low quality staff” does sound a lot like xenophobia. > No one there wants to work for those outsourcing chop shops This is simply not true. There are a lot of benefits to working for a company like this, although as with any company, it’s not all upside. Of course turnover is high because you have a lot of entry level folks. Regardless of where you are in the world, no-one wants to work a level one help desk until they retire. > We can all pick and choose good stories. What about Tata Steel's total mismanagement of Tata Steel Europe? And JLR isn't exactly in good shape as you say it is. You just picked some numbers that sound good. Yeah odd example, JLR is not doing great, and Tata Steel is also struggling, but overall the Tata Group are doing well.
benjaminwootton 1y ago
They outsource as much as they can to the cheapest system integrators they can find, primarily TCS.
ecshafer 1y ago
This is the core thesis of a company like Shopify. Shopify will run everything else about being an e-commerce company (website, inventory, shipping, returns, ads, sales channels, etc) and then the merchant can focus on selling their product. But this is part of the larger thesis about running a business you hear in business school classes, to focus on your specialization and outsource your non-core expertise. Buy Workday/ADP/Paychex don't do payroll or HR. Don't build a data center, buy AWS/Azure/GCP. Don't build a sales database or marketing get Hubspot or Salesforce. Does your company take in a lot of mail? Outsource to a company that specializes in processing mail. Outsource your Technical Helpdesk. Outsource your customer support. This is why componentization is accelerating.
- madeofpalk 1y ago
  I guess the question is whether e-commerce should be a core competency of a business with a significant e-commerce business. I’m not sure what it’s like in the US, but grocery delivery is a reasonably big deal in the UK.
- runako 1y ago
  > to focus on your specialization and outsource your non-core expertise Most retailers will argue that connecting with their core customers and delivering delightful experiences to them is their core expertise. More practically, it will be tension between things like "our marketing department wants X on the site for summer" and "Shopify is planning on launching X in January." It will be less of a resistance to using a third-party provider and more that the third-party provider imposes constraints on the mode of contact with customers. That's a hard pill to swallow for a lot of consumer-focused companies.
  - xp84 1y ago
    > Most retailers will argue that connecting with their core customers and delivering delightful experiences to them is their core expertise. Having worked in e-commerce for most of my career, for individual retailers, I can assure you that the perpetual tension you describe is real. The problem as I see it is, every little retailer thinks that their two-bit designers and product managers are so uniquely visionary in designing interactions that they rightfully should have full control over the product that is the ecommerce website. Shopify employs God-knows-how-many engineers to build and maintain this experience, and probably thousands of SREs to be there 24/7 making sure a random DDOS or slow query doesn't take your site out. "But we think we can build a better site than Shopify with 10 engineers and a couple of managers," they say. They can build one that has the 3 cute whiz-bang features that their self-important product design staff thinks matter, but it will be unreliable, and they won't have sufficient expertise to get right the other 90% of what a "good" ecom site should have. And on top of it all, none of those gimmicks will likely improve conversion or order value enough to be worth doing. The smarter ones IMHO do use Shopify. It lacks so many things in its core that it's infuriating (decent search, any nontrivial filtering), but retailers who use it mostly patch over those flaws with plugins sold by third parties (which often introduce ghastly single points of failure that you have no visibility into, and you can't sue some random plugin vendor you pay $50 a month for your site going down on Black Friday). Ecommerce is hard tbh. But I do personally think that most of my previous employers probably should have done lightweight Shopify skins and made their core competence sourcing, merchandising, and advertising product rather than designing cute search filters, or their own product recommendations algorithm.
dangus 1y ago
This take doesn't make sense. If one of your core businesses is selling clothes online, and you're a large enough entity, you should write your own software to sell clothes online. Basically by your exact same logic you're asking Walmart and Target to outsource their websites, which is completely insane.
- crop_rotation 1y ago
  > Basically by your exact same logic you're asking Walmart and Target to outsource their websites, which is completely insane. No, because Walmart website being down for months is just unthinkable. If a company is so incompetent/resource deficient/use your favorite phrase to describe it/ that their e-commerce is down for a month and outlook is it will be down for more time, then something is seriously wrong with the company. Such companies are 100% going to have a much better experience with Shopify. > If one of your core businesses is selling clothes online, and you're a large enough entity, you should write your own software to sell clothes online. If only this was so easy. For various reasons writing non trivial software is hard, and unless these companies can make some structural changes to hire and retain very good engineers (which also is hard for various reasons for these companies), they simply have no chance of doing better than shopify.
  - dangus 1y ago
    The site being down for months is just the effect of this breach, it doesn't mean that the company made the wrong risk assessment by choosing to write their own software in the first place. If I buy a car and I happen to crash it on the first day I drove it home, that doesn't mean I made the wrong choice to buy the car that day. I still bought the car based on the best information I had at the time. M&S basically lost a lottery ticket type of bad luck scenario where they are dealing with a breach that is far, far worse than a typical data breach's impact. Remember when the PlayStation Network was down for over a month due to a very serious breach? That breach didn't prove that PlayStation should have used some kind of external provider for its online services. In fact, an external provider for that sort o thing is not even practical for their business. Remember, there's an alternate timeline where Shopify itself could be breached in a similarly severe way and also go down for 3 months. It's very unlikely but it's possible. If it can happen to M&S it could happen to Shopify.

softwaredoug 1y ago

At the same time we’re talking about AI replacing developers we also see cases like this of organizational technical incompetency. How does one square those two realities?

bradly 1y ago
> How does one square those two realities? People eat terrible food because they are bombarded with messages to do so. People can use terrible software for the same reasons. It doesn't matter that the food tastes worse than it used to–food companies are having record profits.
jacobsenscott 1y ago
AI replacing devs talk is about short term stock pumping and short term COGS reduction. The long tail is someone else's problem.
- School-Cotton 1y ago
  I’m pretty sure devs are not usually counted as part of COGS.
imhoguy 1y ago
We just need one event of C*O of critical/big company bragging about firing engineering and replacing it with AI and then followed by huge cyberattack like that. Then see how AI balloon pops across news outlets.
- coliveira 1y ago
  They'll respond saying they need to invest the entire revenue of the company on new data centers to fix the issue, and the stock will double in price.
- chgs 1y ago
  The market rewards failure. Look at crowdstrike. You aren’t buying technical service or even technical assurance. You are buying someone to blame so the stakeholders don’t hold you accountable.

benjaminwootton 1y ago

How can it take 3-4 months to get an eCommerce site back online? I assume you could redeploy everything from scratch in less time if you have source code and release assets. With backups and failover sites I can’t think of any world where this would happen?

pavel_lishin 1y ago
> with backups and failover sites What a fun pair of assumptions!
internetter 1y ago
There are no backups. There are no failovers. There is no git. There is no orchestration and deployment stratagies. Programmers ssh into the server and edit code there. Years and years of patchwork on top of patchwork with closely coupled code. Such is a taste of what needs to be done if you wish to have a service that takes months to set back up after any disruption.
- 98codes 1y ago
  [citation needed]
  - internetter 1y ago
    Sorry if I phrased it poorly. I wasn’t definitively saying that all these things are the case. But what always is the case is that when an attack takes down an organization for months, it was employing a tremendous number of horrendous practices. My list was supposed to be some. M&S isn’t down for months because of something innocuous like a full security audit. As a public company losing tens of millions of dollars a week, their only priority is to stop the bleed, even if that means a hasty partial restoration. The fact they can’t even do that suggests they did stuff terribly wrong. There’s an infinite amount of things I didn’t list that could also be the case. Like if Amazon gave them proprietary blobs they lost after the attack and Amazon won’t provide again. But no matter what they are, things were wrong beyond belief. That is a given.

woah 1y ago

Holy shit why don't they just set up a Shopify

wavemode 1y ago
Bureaucracy is almost always the reason. They don't just need a website, they need -their- website back, because it was programmed with a million little business rules and pricing logic and regulatory requirements.
- xp84 1y ago
  You're surely not wrong that a lot of things would need to be done without, but I'd like to think that if I were 'king of M&S' I could have identified a subset of merchandise that could be loaded into a suitable interim solution like Shopify within say, 4 weeks, if the only other option was forgoing all online sales for 12 weeks +. That would also take a lot of the pressure off of the "full recovery team." Of course, the real situation must be 100x more complex than I'm imagining it so "I'd like to think" != "I am confident"
  - NoImmatureAdHom 1y ago
    Four weeks? You mean two days??? The real situation is not 100x more complex, it's just that this is happening in Britain where everything is someone else's job and no one has any reason to care about the actual goals and everyone will go home at 5pm. Or, more likely, to the pub.
    - xp84 1y ago
      You have to keep in mind that you'd have to build out actual fulfillment capabilities on a brand new platform (so you need training of a ton of staff on a new system, as well as a bunch of new equipment to print orders from the "temporary" systems in the warehouses/stores hopefully without removing the original systems they're trying to restore), plus who knows what the database backups looked like. Plus inventory is going to be all screwy with sales happening to some degree in stores (what POS is that using? Is it decrementing inventory we can use?) I'd sure not bet my reputation on having a shoppable site up in less than several weeks without knowing if I have at minimum, reasonably accurate catalog and inventory data.

deleted 1y ago

[deleted]

wyager 1y ago

It's weird to me how it often seems like the US and China are the only countries capable of mega-scale tech infrastructure like this (and even then, only in some industries). Can you imagine Wal-mart's website going down for multiple months? I think a lot of companies (especially in Europe) have not internalized that, yes, you actually do need to expend apparently exorbitant amounts of money on highly-paid engineers if you want your tech to actually be good. Many countries, including the UK, are simply not wealthy enough to do it at scale. They produce plenty of engineers, but most of the ones capable of holding complicated stuff together probably end up working for US companies that can pay them market rates.

tristor 1y ago
With the case of M&S, and in many other cases in UK tech history that have gone poorly, it's mostly examples of the failure of hiring outside consultancies in India to do everything. Business executives continuously fall afoul of the fungibility myth. They believe that engineers are fungible, and that they should therefore simply pay for the cheapest engineers possible that meet the "requirements" on paper, usually set by someone who is not an engineer (HR, project manager, or a lower ranked middle-manager). Time and time and time again we have seen major failures globally, and especially in the UK, that prove that there is no fungibility of engineers, and that outsourcing the critical technical infrastructure for your core systems and services is doomed to failure. They'd rather save a dollar today and lose ten million dollars tomorrow by damaging their national economy and sending more money to India. India's GDP is basically entirely propped up by tech services, and most of that is /failed service delivery/, hard to differentiate from frauds and scams at scale.
- theuppermiddle 1y ago
  Just because consultants are from India does not make them incompetent. Not to mention the attack vector was social engineering, which can happen to any person of any nationality. 70% of India's GDP is domestic consumption. 7% of the GDP is service export, of which only 3.5% is software services. While significant, it certainly is not "propping" up India's GDP.
1y ago

ryanbrick05 1y ago

[dead]

mystified5016 1y ago

> “There is no change to our strategy and our longer-term plans to reshape M&S for growth and, if anything, the incident allows us to accelerate the pace of change as we draw a line and move on.” I wonder if people like this ever hear themselves talking.

ojhughes 1y ago

M&S outsourced over half of its IT team to Tata Consultancy in 2018. Looks like they are now reaping what they sowed https://www.bbc.co.uk/news/business-42629522.amp https://www.bbc.co.uk/news/business-42629522.amp

mrcsharp 1y ago
So they saved £30 million a year and now losing estimated £40 a week (not to mention reputation and future opportunity loss due to lower customer confidence). They have almost wiped out all the savings. If this is the reason, then this was a very bad deal for them.