10 ms·

How can it take 3-4 months to get an eCommerce site back online? I assume you could redeploy everything from scratch in less time if you have source code and r

How can it take 3-4 months to get an eCommerce site back online? I assume you could redeploy everything from scratch in less time if you have source code and release assets. With backups and failover sites I can’t think of any world where this would happen?

pavel_lishin 1y ago
> with backups and failover sites What a fun pair of assumptions!
internetter 1y ago
There are no backups. There are no failovers. There is no git. There is no orchestration and deployment stratagies. Programmers ssh into the server and edit code there. Years and years of patchwork on top of patchwork with closely coupled code. Such is a taste of what needs to be done if you wish to have a service that takes months to set back up after any disruption.
- 98codes 1y ago
  [citation needed]
  - internetter 1y ago
    Sorry if I phrased it poorly. I wasn’t definitively saying that all these things are the case. But what always is the case is that when an attack takes down an organization for months, it was employing a tremendous number of horrendous practices. My list was supposed to be some. M&S isn’t down for months because of something innocuous like a full security audit. As a public company losing tens of millions of dollars a week, their only priority is to stop the bleed, even if that means a hasty partial restoration. The fact they can’t even do that suggests they did stuff terribly wrong. There’s an infinite amount of things I didn’t list that could also be the case. Like if Amazon gave them proprietary blobs they lost after the attack and Amazon won’t provide again. But no matter what they are, things were wrong beyond belief. That is a given.
    - pavel_lishin 1y ago
      To be fair, I would be that nearly every organization employs a tremendous number of horrendous practices. We only gasp at the ones who get taken down for some reason.
      - internetter 1y ago
        Horrendous practices exist on a spectrum. Every org has bad code that somebody will fix someday™. It is reasonable to expect that after a catostrophic event like this, a full recovery takes some time. But at a "good" org, these practices are isolated. Not every org is entirely held together with masking tape. For the entire thing to be down for so long, the bad practices need to be widespread, seeping into every corner of the product. Ubiquitous. For instance, when Cloudflare all went down a while ago due to a bad regex, it took less than a hour to rollback the changes. Undoubtably there were bad practices that lead to a regex having the ability to take everything out, but the problem was isolatable and once adressed partial service was quickly restored, and shortly after preventative measures were employed. This bug didn't destroy cloudflare for months. P.S. in anticipation of the "but cloudflare has SLAs!!" that isn't really a distinction worth making because M&S has an implicit SLA with their customers — they are losing 40 million each week they can't offer service. Plenty of non-b2b companies that invest in quick recovery as well, like Netflix's monkey testing.
      - PaulHoule 1y ago
        No, best practice is that you have a checklist to bring up a copy of your system, better yet that checklist is "run a script". In the cloud age you ought to be able to bring a copy up in a new zone with a repeatable procedure. Makes a big difference in developer quality of life and improves productivity right away. If you onboard a new dev you give them a checklist and they are up and running that day. I had a coworker who taught me a lot about sysadmining, (social) networking, and vendor management. She told me that you'd better have your backup procedures tested. One time we were doing a software upgrade and I screwed up and dropped the Oracle database for a production system. She had a mirror in place so we had less than a minute of downtime.
- throwawaymgb123 1y ago
  This is a perfect description of how things work at one of the largest health care networks in the northeast US (speaking as someone who works there and keeps saying "where's the automation? where are the procedures?" and keeps being told to shut up, we don't have TIME for that sort of thing.
  - aspenmayer 1y ago
    If you don't have time to prepare for failure, then you'll have little time to invest in success, either, if/when failure strikes.
  - internetter 1y ago
    lol the healthcare industry was definitely in my mind as I wrote this. Never worked there but I read a lot of postmortems and it shows whenever I use their digital products. Recent example is CVS. Somehow, at some point, they decided that my CVS pharmacy account should be linked to my Mom's extracare. Couldn't find any menu to fix it online. So the next time I went to the register I asked to update it. They read the linked phone number. It was mine. Ok, it is fixed, I think. But then the reciept prints out and it is my mom's Extracare card number. So the next time I press harder. I ask them to read me the card number they have linked from their screen. They read my card number. Ok, it is fixed, I think. But then the reciept prints out and the card number is different—it is my mom's. Then I know the system is incredibly fucked. Being an engineer, I think about how this could happen. I'm guessing there are a hundred database fields where the extracare number is stored, and only one is set to my mom's or something. I poke around the CVS website and find countless different portals made with clearly different frameworks and design practices. Then I know all of CVS's tech looks like this and a disaster is waiting to happen. Goes like this for a lot of finance as well. E.g. I can say with confidence that Equifax is still as scuffed as it was back in 2017 when it was hacked. That is a story for another time. Nobody bothers to keep things clean until it is too late. The features you deliver give promotions, not the potential catastrophes you prevent. Humans have a tendency to be so short sighted, chasing endless earnings beats without anticipating future problems.
- squiffsquiff 1y ago
  This is an ignorant position. Look at e.g. https://engineering.marksandspencer.com/mobile/2024/09/05/releasing-apps.html https://engineering.marksandspencer.com/mobile/2024/09/05/re...
didroe 1y ago
How do you know it's safe to redeploy? If your entire operation may be compromised, how can you trust the code hasn't been modified, that some information the attackers have doesn't present a further threat, or that flaws that allowed the attack aren't still present in your services? It's a large company so likely has a mess of microservices and outsourced development where no-one really understands parts of it. Also, if they get compromised again it would be a PR disaster. They're probably having to audit everything, invest a lot of effort in additional hardening, and re-architect things to try and minimise the impact of any future attack. And via some bureaucratic organisational structure/outsourcing contract.
- ajb 1y ago
  You literally have some of your team buy new laptops and hang out in a temporary wework to set it up on entirely new infra, air-gapped from your ongoing forensic exercise. You just need to make sure none of the people you send are dumb enough to reuse their password. You need to take the domain name, but they will be using one of the high end domain companies so that can be handled. Bear in mind that this is a company which still sells physically and has retail and warehouse staff. All that the e-commerce side needs to do is issue orders of what skus to send to what addresses, and pause items that are out of stock. M&S is not Amazon and doesn't have that many SKUs, 5 people could probably walk round the store in a few days and photograph all of them for the new shopping site. Sure, customers will need to make a new account or buy as a guest. But this stuff is not hard on the technical side. There is no interaction between customers like a social media site, so horizontal scaling is easy. Now I get that there are loads of refinements that go into maximising profit, like analytics, price optimization, etc. But to get in revenue these guys don't even need to set up advertising on day one because they have customers that have been buying from them for decades. The time to set up all that stuff is when your revenue is nonzero
  - Oras 1y ago
    I don’t think you realise how complicated the e-commerce is for a company. You are thinking of a garage sale. With each order: - you need warehouse integration to keep the sync of physical to digital store. That has to happen fast or you’ll get orders with no stock. - You need to sync the payment to whatever ancient accounting system they use, again while issuing invoices, consolidating customers … etc. - Logistics management, where to get the order from, issuing a label, using the right fleet, making sure it is dispatched on time, arrive on time. - Customer support, refunds, partial refunds, adding items after order … etc. So yeah, 5 people!
chatmasta 1y ago
The Co-Op (grocery store chain) was hacked around the same time in likely the same incident. It took three weeks for them to get food back on the shelves at my local store. I don’t understand how that’s even possible… what happened to all the meat and vegetables in the supply chain? They just stopped flowing? They rotted? Why couldn’t they use pen and paper? It’s unbelievable to me that a business would go three weeks without stocking inventory.
- Henchman21 1y ago
  You forget we have entered the “Who the fuck cares?” era. When no one in the chain is incentivized to care, things just fall apart.
  - chatmasta 1y ago
    Interestingly Co-Op is so-called because it’s a cooperative business, which vaguely means it’s owned by its employees, and technically means it’s a “Registered Society” [0]. If you check CompaniesHouse [1], which normally has all financial documents for UK corporations, it points you to a separate “Public Register” for the Co-Op [2]. So, your comment has more basis in reality than simply being snark… the fact that “nobody is incentivized to care” is actually by design. That has some positive benefits but in this case we’re seeing how it breaks down for the same reasons nobody in a crowd calls an ambulance for someone hurt… it’s the bystander effect applied to corporate governance with diluted accountability. [0] https://www.gov.uk/hmrc-internal-manuals/company-taxation-manual/ctm40507 https://www.gov.uk/hmrc-internal-manuals/company-taxation-ma... [1] https://find-and-update.company-information.service.gov.uk/company/IP00525R https://find-and-update.company-information.service.gov.uk/c... [2] https://mutuals.fca.org.uk/Search/Society/7240 https://mutuals.fca.org.uk/Search/Society/7240
    - Henchman21 1y ago
      I guess this is more snark, but honestly I am genuinely shocked when people care about anything anymore. Sad times.
paxys 1y ago
It isn't surprising at all. There's a reason why tech companies have insanely large engineering teams even though it feels to an outsider (and inept management) that nobody is doing anything. It takes a lot of manpower and hours to keep a complex system working and up to date. Who validates the backups? Who writes the wikis? Who trains new hires? Who staffs all the on-call rotations? Who organizes disaster recovery drills? Who runs red team exercises? After the company has had repeated layoffs and fired, outsourced or otherwise pushed out all this "overhead" eventually there's no one remaining who actually understands how the system works. One small outage later, this is exactly the situation you end up in.
- CobrastanJorji 1y ago
  Yep. It takes way fewer people to operating a working system than to build a new one. And the nature of capitalism is that you will pare down your numbers until you have the absolute minimum staffing you need to keep the lights on. Then when everything explodes, you completely lack the know-how to fix it. Then the CEO yells as the tech executive who responds by demanding hourly updates from the two junior devs who operate the site, and nobody wants to admit that they aren't capable of fixing it, and nobody's gonna OK a really expensive "we're gonna spend a month emergency building a new thing" plan because nobody's okay with because a month is obviously way too much time you need to fix it right now, and then three months go by and here you are.
  - spacebanana7 1y ago
    I get the opposite impression. Stale software organisations with steady operating products seem to use massive headcounts, whereas startups building new products often get by with relatively few people.
    - esseph 1y ago
      Startups don't have to run a software stack for decades, hardware refreshes or SKU updates and replatforms, dealing with multiple types of turnover and reogs, knowledge transfer, etc. Plus at least monthly if not daily, even hourly system patching. Planting a garden is one thing. Weeding it is another.
wrs 1y ago
“If you have source code and release assets.” And a build process that works from a clean code base. And a deploy process that works on fresh servers. All of which assumes you even know what services exist, which in any company of this age and size you probably don’t.
deleted 1y ago
[deleted]
cjs_ac 1y ago
Your comment suggests that you're not familiar with the diversity in M&S' operation. Marks and Spencers started as a department store; they still have this operation. They sell clothes, beauty products, cookware, homeware and furniture. All these things are sold in physical shops and online. Most of this is straightforward for an e-commerce operation, but the furniture will involve separate warehousing and delivery systems. They also offer financial services (bank accounts, credit cards and insurance). These are white labelled products, but they are closely linked to their loyalty programme (the Sparks card). Finally, they have their food operation: M&S is also a high-end supermarket. You can't do your food shop on the M&S website (although their food products are available from online-only supermarket Ocado), but you can order some food products (sandwich platters and party food) and fresh flowers from the website. So M&S is a mid-tier department store and a high-end supermarket. These are very different styles of retail operation: supermarkets require a lot of data processing to ensure the right things get to the right shops at the right time to ensure that food doesn't go to waste but also shoppers aren't annoyed by the unavailability of staples like bread and milk. Finally, M&S is traditionally fairly strong in customer service; it's not exactly Harrod's or Fortnum and Mason's, but their bra-fitting service, for example, has a legendary reputation. The internet isn't their natural home. So all-in-all, you have a business doing complicated things online because they think they have to, not because they want to: a pretty clear recipe for disaster.
- neepi 1y ago
  Their banking op is a fucking mess as well. Had no end of problems with their card services which were rebranded HSBC.
dylan604 1y ago
For this particular audience, it's one of those things that could be rewritten in Rust over a weekend and then deployed on the cheap via Hetzner. At least then it'll be memory safe!
briffle 1y ago
of course, if you redeployed everything from the source code, you could very well still have the same vulnerabilities that caused the problem in the first place..
cameronh90 1y ago
The British Library still aren't fully back up and running after their cyberattack in Oct 2023: https://www.bl.uk/cyber-incident/ https://www.bl.uk/cyber-incident/
donnachangstein 1y ago
HN posters love talking gangster shit when something goes offline but never walked a mile in their boots. I most recently remember sifting through gloating that 4chan - a shoestring operation with basically no staff - was offline for a couple weeks after getting hacked. I've worked at a shop that had DR procedures for EVERYTHING. The recovery time for non-critical infra was measured in months. There are only so many hands to go around, and stuff takes time to rebuild. And that's assuming you have procedures on file! Not to mention if there was a major compromise you need to perform forensics to make sure you kick the bad guys out and patch the hole so the same thing doesn't happen again a week after your magical recovery. And if you don't know, you shut it down till it's deemed safe. How do you know the backups and failover sites aren't tainted? Nothing worse than running an e-commerce site processing customer payment card data when you know you're owned. That's a good way to get in deeper trouble.
kelnos 1y ago
I'm not that surprised, though 3-4 months does feel like a long time. When I was at early Twilio (2011? 2012? ish), we would completely tear down our dev and staging environments every month (quarter? can't remember), and build them back up from scratch. That was everything, including databases (which would get restored from backup during the re-bring-up) and even the deployment infrastructure itself. At that point we were still pretty small and didn't have a ton of services. Just bringing my product (Twilio Client) back up, plus some of the underlying voice services, took about 24 hours (spread across a few days). And the bits I handled were a) a small part of the whole, and b) some of the easier parts to bring up. We stopped doing those teardowns sometime later in 2012, or perhaps 2013, because they started taking way too much time away from doing Actual Work. People can't get things done when the staging environment is down for more than a week. Over the following 10 years or so, Twilio's backend exploded in complexity, number of services, and the dependencies between those services. I left Twilio in early 2022, and I wouldn't have been surprised if it would have taken several months to bring up Twilio (prod) from scratch at that point, though in their case it would be a situation where some products and features would be available earlier than others, so it's not really the same as an e-commerce site. And that was when I left; I'm sure complexity has increased further in the past 3 years. Also consider that institutional knowledge matters too. I would guess that for all the services running at Twilio, the people who first brought up many (most?) of them are long gone. So I wouldn't be surprised if the people at M&S right now just have no idea how to bring up an e-commerce site like theirs from scratch, and have to learn as they go.
tw04 1y ago
So you haven’t dealt with ransomware gangs yet? Because they have gotten sophisticated enough to nuke source code repos and backups and replicated copies. It’s part of the reason tape is literally never going to die for organizations with data that simply cannot be lost, regardless of rto.

pavel_lishin 1y ago

> with backups and failover sites What a fun pair of assumptions!

internetter 1y ago

There are no backups. There are no failovers. There is no git. There is no orchestration and deployment stratagies. Programmers ssh into the server and edit code there. Years and years of patchwork on top of patchwork with closely coupled code. Such is a taste of what needs to be done if you wish to have a service that takes months to set back up after any disruption.

98codes 1y ago
[citation needed]
- internetter 1y ago
  Sorry if I phrased it poorly. I wasn’t definitively saying that all these things are the case. But what always is the case is that when an attack takes down an organization for months, it was employing a tremendous number of horrendous practices. My list was supposed to be some. M&S isn’t down for months because of something innocuous like a full security audit. As a public company losing tens of millions of dollars a week, their only priority is to stop the bleed, even if that means a hasty partial restoration. The fact they can’t even do that suggests they did stuff terribly wrong. There’s an infinite amount of things I didn’t list that could also be the case. Like if Amazon gave them proprietary blobs they lost after the attack and Amazon won’t provide again. But no matter what they are, things were wrong beyond belief. That is a given.
  - pavel_lishin 1y ago
    To be fair, I would be that nearly every organization employs a tremendous number of horrendous practices. We only gasp at the ones who get taken down for some reason.
    - internetter 1y ago
      Horrendous practices exist on a spectrum. Every org has bad code that somebody will fix someday™. It is reasonable to expect that after a catostrophic event like this, a full recovery takes some time. But at a "good" org, these practices are isolated. Not every org is entirely held together with masking tape. For the entire thing to be down for so long, the bad practices need to be widespread, seeping into every corner of the product. Ubiquitous. For instance, when Cloudflare all went down a while ago due to a bad regex, it took less than a hour to rollback the changes. Undoubtably there were bad practices that lead to a regex having the ability to take everything out, but the problem was isolatable and once adressed partial service was quickly restored, and shortly after preventative measures were employed. This bug didn't destroy cloudflare for months. P.S. in anticipation of the "but cloudflare has SLAs!!" that isn't really a distinction worth making because M&S has an implicit SLA with their customers — they are losing 40 million each week they can't offer service. Plenty of non-b2b companies that invest in quick recovery as well, like Netflix's monkey testing.
    - PaulHoule 1y ago
      No, best practice is that you have a checklist to bring up a copy of your system, better yet that checklist is "run a script". In the cloud age you ought to be able to bring a copy up in a new zone with a repeatable procedure. Makes a big difference in developer quality of life and improves productivity right away. If you onboard a new dev you give them a checklist and they are up and running that day. I had a coworker who taught me a lot about sysadmining, (social) networking, and vendor management. She told me that you'd better have your backup procedures tested. One time we were doing a software upgrade and I screwed up and dropped the Oracle database for a production system. She had a mirror in place so we had less than a minute of downtime.
throwawaymgb123 1y ago
This is a perfect description of how things work at one of the largest health care networks in the northeast US (speaking as someone who works there and keeps saying "where's the automation? where are the procedures?" and keeps being told to shut up, we don't have TIME for that sort of thing.
- aspenmayer 1y ago
  If you don't have time to prepare for failure, then you'll have little time to invest in success, either, if/when failure strikes.
- internetter 1y ago
  lol the healthcare industry was definitely in my mind as I wrote this. Never worked there but I read a lot of postmortems and it shows whenever I use their digital products. Recent example is CVS. Somehow, at some point, they decided that my CVS pharmacy account should be linked to my Mom's extracare. Couldn't find any menu to fix it online. So the next time I went to the register I asked to update it. They read the linked phone number. It was mine. Ok, it is fixed, I think. But then the reciept prints out and it is my mom's Extracare card number. So the next time I press harder. I ask them to read me the card number they have linked from their screen. They read my card number. Ok, it is fixed, I think. But then the reciept prints out and the card number is different—it is my mom's. Then I know the system is incredibly fucked. Being an engineer, I think about how this could happen. I'm guessing there are a hundred database fields where the extracare number is stored, and only one is set to my mom's or something. I poke around the CVS website and find countless different portals made with clearly different frameworks and design practices. Then I know all of CVS's tech looks like this and a disaster is waiting to happen. Goes like this for a lot of finance as well. E.g. I can say with confidence that Equifax is still as scuffed as it was back in 2017 when it was hacked. That is a story for another time. Nobody bothers to keep things clean until it is too late. The features you deliver give promotions, not the potential catastrophes you prevent. Humans have a tendency to be so short sighted, chasing endless earnings beats without anticipating future problems.
squiffsquiff 1y ago
This is an ignorant position. Look at e.g. https://engineering.marksandspencer.com/mobile/2024/09/05/releasing-apps.html https://engineering.marksandspencer.com/mobile/2024/09/05/re...

didroe 1y ago

How do you know it's safe to redeploy? If your entire operation may be compromised, how can you trust the code hasn't been modified, that some information the attackers have doesn't present a further threat, or that flaws that allowed the attack aren't still present in your services? It's a large company so likely has a mess of microservices and outsourced development where no-one really understands parts of it. Also, if they get compromised again it would be a PR disaster. They're probably having to audit everything, invest a lot of effort in additional hardening, and re-architect things to try and minimise the impact of any future attack. And via some bureaucratic organisational structure/outsourcing contract.

ajb 1y ago
You literally have some of your team buy new laptops and hang out in a temporary wework to set it up on entirely new infra, air-gapped from your ongoing forensic exercise. You just need to make sure none of the people you send are dumb enough to reuse their password. You need to take the domain name, but they will be using one of the high end domain companies so that can be handled. Bear in mind that this is a company which still sells physically and has retail and warehouse staff. All that the e-commerce side needs to do is issue orders of what skus to send to what addresses, and pause items that are out of stock. M&S is not Amazon and doesn't have that many SKUs, 5 people could probably walk round the store in a few days and photograph all of them for the new shopping site. Sure, customers will need to make a new account or buy as a guest. But this stuff is not hard on the technical side. There is no interaction between customers like a social media site, so horizontal scaling is easy. Now I get that there are loads of refinements that go into maximising profit, like analytics, price optimization, etc. But to get in revenue these guys don't even need to set up advertising on day one because they have customers that have been buying from them for decades. The time to set up all that stuff is when your revenue is nonzero
- Oras 1y ago
  I don’t think you realise how complicated the e-commerce is for a company. You are thinking of a garage sale. With each order: - you need warehouse integration to keep the sync of physical to digital store. That has to happen fast or you’ll get orders with no stock. - You need to sync the payment to whatever ancient accounting system they use, again while issuing invoices, consolidating customers … etc. - Logistics management, where to get the order from, issuing a label, using the right fleet, making sure it is dispatched on time, arrive on time. - Customer support, refunds, partial refunds, adding items after order … etc. So yeah, 5 people!

chatmasta 1y ago

The Co-Op (grocery store chain) was hacked around the same time in likely the same incident. It took three weeks for them to get food back on the shelves at my local store. I don’t understand how that’s even possible… what happened to all the meat and vegetables in the supply chain? They just stopped flowing? They rotted? Why couldn’t they use pen and paper? It’s unbelievable to me that a business would go three weeks without stocking inventory.

Henchman21 1y ago
You forget we have entered the “Who the fuck cares?” era. When no one in the chain is incentivized to care, things just fall apart.
- chatmasta 1y ago
  Interestingly Co-Op is so-called because it’s a cooperative business, which vaguely means it’s owned by its employees, and technically means it’s a “Registered Society” [0]. If you check CompaniesHouse [1], which normally has all financial documents for UK corporations, it points you to a separate “Public Register” for the Co-Op [2]. So, your comment has more basis in reality than simply being snark… the fact that “nobody is incentivized to care” is actually by design. That has some positive benefits but in this case we’re seeing how it breaks down for the same reasons nobody in a crowd calls an ambulance for someone hurt… it’s the bystander effect applied to corporate governance with diluted accountability. [0] https://www.gov.uk/hmrc-internal-manuals/company-taxation-manual/ctm40507 https://www.gov.uk/hmrc-internal-manuals/company-taxation-ma... [1] https://find-and-update.company-information.service.gov.uk/company/IP00525R https://find-and-update.company-information.service.gov.uk/c... [2] https://mutuals.fca.org.uk/Search/Society/7240 https://mutuals.fca.org.uk/Search/Society/7240
  - Henchman21 1y ago
    I guess this is more snark, but honestly I am genuinely shocked when people care about anything anymore. Sad times.

paxys 1y ago

It isn't surprising at all. There's a reason why tech companies have insanely large engineering teams even though it feels to an outsider (and inept management) that nobody is doing anything. It takes a lot of manpower and hours to keep a complex system working and up to date. Who validates the backups? Who writes the wikis? Who trains new hires? Who staffs all the on-call rotations? Who organizes disaster recovery drills? Who runs red team exercises? After the company has had repeated layoffs and fired, outsourced or otherwise pushed out all this "overhead" eventually there's no one remaining who actually understands how the system works. One small outage later, this is exactly the situation you end up in.

CobrastanJorji 1y ago
Yep. It takes way fewer people to operating a working system than to build a new one. And the nature of capitalism is that you will pare down your numbers until you have the absolute minimum staffing you need to keep the lights on. Then when everything explodes, you completely lack the know-how to fix it. Then the CEO yells as the tech executive who responds by demanding hourly updates from the two junior devs who operate the site, and nobody wants to admit that they aren't capable of fixing it, and nobody's gonna OK a really expensive "we're gonna spend a month emergency building a new thing" plan because nobody's okay with because a month is obviously way too much time you need to fix it right now, and then three months go by and here you are.
- spacebanana7 1y ago
  I get the opposite impression. Stale software organisations with steady operating products seem to use massive headcounts, whereas startups building new products often get by with relatively few people.
  - esseph 1y ago
    Startups don't have to run a software stack for decades, hardware refreshes or SKU updates and replatforms, dealing with multiple types of turnover and reogs, knowledge transfer, etc. Plus at least monthly if not daily, even hourly system patching. Planting a garden is one thing. Weeding it is another.

wrs 1y ago

“If you have source code and release assets.” And a build process that works from a clean code base. And a deploy process that works on fresh servers. All of which assumes you even know what services exist, which in any company of this age and size you probably don’t.

deleted 1y ago

[deleted]

cjs_ac 1y ago

Your comment suggests that you're not familiar with the diversity in M&S' operation. Marks and Spencers started as a department store; they still have this operation. They sell clothes, beauty products, cookware, homeware and furniture. All these things are sold in physical shops and online. Most of this is straightforward for an e-commerce operation, but the furniture will involve separate warehousing and delivery systems. They also offer financial services (bank accounts, credit cards and insurance). These are white labelled products, but they are closely linked to their loyalty programme (the Sparks card). Finally, they have their food operation: M&S is also a high-end supermarket. You can't do your food shop on the M&S website (although their food products are available from online-only supermarket Ocado), but you can order some food products (sandwich platters and party food) and fresh flowers from the website. So M&S is a mid-tier department store and a high-end supermarket. These are very different styles of retail operation: supermarkets require a lot of data processing to ensure the right things get to the right shops at the right time to ensure that food doesn't go to waste but also shoppers aren't annoyed by the unavailability of staples like bread and milk. Finally, M&S is traditionally fairly strong in customer service; it's not exactly Harrod's or Fortnum and Mason's, but their bra-fitting service, for example, has a legendary reputation. The internet isn't their natural home. So all-in-all, you have a business doing complicated things online because they think they have to, not because they want to: a pretty clear recipe for disaster.

neepi 1y ago
Their banking op is a fucking mess as well. Had no end of problems with their card services which were rebranded HSBC.

dylan604 1y ago

For this particular audience, it's one of those things that could be rewritten in Rust over a weekend and then deployed on the cheap via Hetzner. At least then it'll be memory safe!

briffle 1y ago

of course, if you redeployed everything from the source code, you could very well still have the same vulnerabilities that caused the problem in the first place..

cameronh90 1y ago

The British Library still aren't fully back up and running after their cyberattack in Oct 2023: https://www.bl.uk/cyber-incident/ https://www.bl.uk/cyber-incident/

donnachangstein 1y ago

HN posters love talking gangster shit when something goes offline but never walked a mile in their boots. I most recently remember sifting through gloating that 4chan - a shoestring operation with basically no staff - was offline for a couple weeks after getting hacked. I've worked at a shop that had DR procedures for EVERYTHING. The recovery time for non-critical infra was measured in months. There are only so many hands to go around, and stuff takes time to rebuild. And that's assuming you have procedures on file! Not to mention if there was a major compromise you need to perform forensics to make sure you kick the bad guys out and patch the hole so the same thing doesn't happen again a week after your magical recovery. And if you don't know, you shut it down till it's deemed safe. How do you know the backups and failover sites aren't tainted? Nothing worse than running an e-commerce site processing customer payment card data when you know you're owned. That's a good way to get in deeper trouble.

kelnos 1y ago

I'm not that surprised, though 3-4 months does feel like a long time. When I was at early Twilio (2011? 2012? ish), we would completely tear down our dev and staging environments every month (quarter? can't remember), and build them back up from scratch. That was everything, including databases (which would get restored from backup during the re-bring-up) and even the deployment infrastructure itself. At that point we were still pretty small and didn't have a ton of services. Just bringing my product (Twilio Client) back up, plus some of the underlying voice services, took about 24 hours (spread across a few days). And the bits I handled were a) a small part of the whole, and b) some of the easier parts to bring up. We stopped doing those teardowns sometime later in 2012, or perhaps 2013, because they started taking way too much time away from doing Actual Work. People can't get things done when the staging environment is down for more than a week. Over the following 10 years or so, Twilio's backend exploded in complexity, number of services, and the dependencies between those services. I left Twilio in early 2022, and I wouldn't have been surprised if it would have taken several months to bring up Twilio (prod) from scratch at that point, though in their case it would be a situation where some products and features would be available earlier than others, so it's not really the same as an e-commerce site. And that was when I left; I'm sure complexity has increased further in the past 3 years. Also consider that institutional knowledge matters too. I would guess that for all the services running at Twilio, the people who first brought up many (most?) of them are long gone. So I wouldn't be surprised if the people at M&S right now just have no idea how to bring up an e-commerce site like theirs from scratch, and have to learn as they go.

tw04 1y ago

So you haven’t dealt with ransomware gangs yet? Because they have gotten sophisticated enough to nuke source code repos and backups and replicated copies. It’s part of the reason tape is literally never going to die for organizations with data that simply cannot be lost, regardless of rto.