5 ms·

O2 'to seek millions' in damages over data outage

70 points by bauc 8y ago

tiemand 8y ago
Ericsson said last week that "an initial root cause analysis" had indicated that the "main issue was an expired certificate in the software versions installed with these customers".
- tiemand 8y ago
  Wow!
- 7ewis 8y ago
  Wonder whose responsibility it was to update that, O2 or Ericsson.
  - shshhdhs 8y ago
    Ericsson generally offers managed services, and so it's likely it was their responsibility. Also, this is reflected by O2 seeking damages.
    - walshemj 8y ago
      It apparently only effected a new software release - so its down to Ericson in this case.
    - lovemenot 8y ago
      This is supported by the fact that Softbank in Japan experiened a similar network outage at the same time. They too blame Ericsson.
- tialaramex 8y ago
  I suspect I'll never know, but I would be really interested to know what sort of certificate. Was this part of some larger public system or was it a purely internal PKI? If internal, did the certificates _do_ anything of value or did they purely make the system more fragile because somebody thought it ought to have certificates? One of the arguments in favour of abusing the Web PKI to do other things is that tooling for the Web PKI is in a relatively good place. And a lot of other things that you might ideally hope exist actually DO exist for the Web PKI. There's independent oversight, somebody is actually reading those yawn-making audit reports, it isn't perfect but it's not just make believe either. But on the other hand, the Web PKI is ours, and so if it suits us to change it we don't really care that this is annoying for your payment card systems, jet aeroplanes, nuclear submarines or whatever else you've duct-taped the Web PKI into. This was fun to watch with SHA-1 for example and is currently causing some fallout for names with underscores in (DNS can handle a name with an underscore in it but they're prohibited for hostnames. Lots of people ignored that, but that's their problem, not ours and they are not happy about that).
  - dzhiurgis 8y ago
    Can you explain/link more about sha1 and underscores? Can't seem to find anything on web.
    - joecool1029 8y ago
      Both underscores (because against standard) and SHA-1 (because old and weak) were removed from browsers. On underscores: https://www.digicert.com/blog/digicert-pushes-underscore-extension/ https://www.digicert.com/blog/digicert-pushes-underscore-ext... On SHA-1: https://blog.mozilla.org/security/2015/10/20/continuing-to-phase-out-sha-1-certificates/ https://blog.mozilla.org/security/2015/10/20/continuing-to-p...
      - tialaramex 8y ago
        Mmm, rather than the behaviour being "removed from browsers" the important thing is that Certificate Authorities were told that issuing such certificates could cause them to be distrusted as a whole by browsers. Part of the reason to do this is that if CAs issue the certificates then their customers buy them, and create pressure to accept them. If none of the CAs are willing to issue this never comes up. Another is "unknown unknowns". Security systems don't play well with undefined behaviour, if we explicitly forbid everything we don't want to exist that minimises the risk of such undefined behaviour anywhere in the certificate handling code getting exploited. It's a defence in depth.
  - becauseiam 8y ago
    Ericsson have both their own PKI infrastructure[0], at least for software integrity checking (however that certificate is valid since March this year, and the CRL[1] it refers to is empty), in addition to using other certificate authorities for everything such as the hosting of websites to internal infrastructure[2]. I suspect it wasn't any of the above - rather it is the PKI that is used in running the IPSec networking done between carrier's RANs and other parts of core network[3], which is probably Ericsson's own internal CA. [0] https://www.ericsson.com/en/about-us/enterprise-security/pki https://www.ericsson.com/en/about-us/enterprise-security/pki [1] http://crl.ericsson.net/Ericsson_Software_Deliverable_Integrity_Protection_Root_CA_A1.crl http://crl.ericsson.net/Ericsson_Software_Deliverable_Integr... [2] https://crt.sh/?q=%.ericsson.net https://crt.sh/?q=%.ericsson.net [3] https://en.wikipedia.org/wiki/System_Architecture_Evolution https://en.wikipedia.org/wiki/System_Architecture_Evolution
stingraycharles 8y ago
This seems like a publicity stunt more than anything else. There's no way they actually lost 100MM here, and doing this ensures the public sees Ericsson as the real villain, and paints O2 as the victim. It's a very clever strategy, nonetheless.
- djhworld 8y ago
  I suspect business customers might have more of a stick to beat O2 over the head with, rather than average people with a cell phone contract. e.g. Transport for London's dot matrix displays could not show bus times for the 14 or so hours the network was down, apparently iPads issued to NHS staff to manage patient reports used O2 sim cards so that couldn't work either. Probably a whole range of small to medium enterprises affected too.
  - isostatic 8y ago
    Our yodel driver reported they were struggling with both signing the packages as delivered and with drivers unable to cope without google maps.
- jw1224 8y ago
  Just some dirty back-of-the-napkin numbers, but theoretically, £100MM doesn't seem unreasonable. O2 have 25MM direct customers, plus another 7MM through reseller networks — so that's 32MM people directly affected by the outage. Presuming O2 receives an average of £30 per customer per month (which I think is a reasonable estimate), that's £940MM/month, working out to £32MM per day in a 30-day month. O2 are refunding customers 2 days' worth of service for the outage, so that would be £64MM in lost revenue. Add on top of that brand and reputation damage — and other factors I've no doubt forgotten to include — and £100MM doesn't seem unreasonable.
  - chipperyman573 8y ago
    What does MM mean? Google wasn't able to help.
fogetti 8y ago
At the same time Softbank's network was also down at the same time in Japan for the same Ericsson issue. The scale is quite surprising.
- kakkun 8y ago
  Softbank's press release [0] mentions that a total of 11 countries were affected by the outage. I can't seem to find any information on what other countries were affected though. [0] https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20181206_02/ https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20...
  - sondh 8y ago
    Vietnam's second largest mobile network operator Mobifone was affected for 3 hours[1]. Now we know 3: UK, Japan and Vietnam. I'm sure it's possible track down the other 8. [1] (link in Vietnamese) https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nhieu-mang-di-dong-tren-toan-cau-20181207123405213.htm https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nh...
    - ksec 8y ago
      Wow, Hong Kong's Smartone uses Ericsson and wasn't effected. I am surprised at how wide spread the problem is. How could Ericsson messed this up when the it has the biggest chance to grab business from Huawei.

70 points by bauc 8y ago

Ericsson said last week that "an initial root cause analysis" had indicated that the "main issue was an expired certificate in the software versions installed with these customers".

tiemand 8y ago
Wow!
7ewis 8y ago
Wonder whose responsibility it was to update that, O2 or Ericsson.
- shshhdhs 8y ago
  Ericsson generally offers managed services, and so it's likely it was their responsibility. Also, this is reflected by O2 seeking damages.
  - walshemj 8y ago
    It apparently only effected a new software release - so its down to Ericson in this case.
  - lovemenot 8y ago
    This is supported by the fact that Softbank in Japan experiened a similar network outage at the same time. They too blame Ericsson.
tialaramex 8y ago
I suspect I'll never know, but I would be really interested to know what sort of certificate. Was this part of some larger public system or was it a purely internal PKI? If internal, did the certificates _do_ anything of value or did they purely make the system more fragile because somebody thought it ought to have certificates? One of the arguments in favour of abusing the Web PKI to do other things is that tooling for the Web PKI is in a relatively good place. And a lot of other things that you might ideally hope exist actually DO exist for the Web PKI. There's independent oversight, somebody is actually reading those yawn-making audit reports, it isn't perfect but it's not just make believe either. But on the other hand, the Web PKI is ours, and so if it suits us to change it we don't really care that this is annoying for your payment card systems, jet aeroplanes, nuclear submarines or whatever else you've duct-taped the Web PKI into. This was fun to watch with SHA-1 for example and is currently causing some fallout for names with underscores in (DNS can handle a name with an underscore in it but they're prohibited for hostnames. Lots of people ignored that, but that's their problem, not ours and they are not happy about that).
- dzhiurgis 8y ago
  Can you explain/link more about sha1 and underscores? Can't seem to find anything on web.
  - joecool1029 8y ago
    Both underscores (because against standard) and SHA-1 (because old and weak) were removed from browsers. On underscores: https://www.digicert.com/blog/digicert-pushes-underscore-extension/ https://www.digicert.com/blog/digicert-pushes-underscore-ext... On SHA-1: https://blog.mozilla.org/security/2015/10/20/continuing-to-phase-out-sha-1-certificates/ https://blog.mozilla.org/security/2015/10/20/continuing-to-p...
    - tialaramex 8y ago
      Mmm, rather than the behaviour being "removed from browsers" the important thing is that Certificate Authorities were told that issuing such certificates could cause them to be distrusted as a whole by browsers. Part of the reason to do this is that if CAs issue the certificates then their customers buy them, and create pressure to accept them. If none of the CAs are willing to issue this never comes up. Another is "unknown unknowns". Security systems don't play well with undefined behaviour, if we explicitly forbid everything we don't want to exist that minimises the risk of such undefined behaviour anywhere in the certificate handling code getting exploited. It's a defence in depth.
- becauseiam 8y ago
  Ericsson have both their own PKI infrastructure[0], at least for software integrity checking (however that certificate is valid since March this year, and the CRL[1] it refers to is empty), in addition to using other certificate authorities for everything such as the hosting of websites to internal infrastructure[2]. I suspect it wasn't any of the above - rather it is the PKI that is used in running the IPSec networking done between carrier's RANs and other parts of core network[3], which is probably Ericsson's own internal CA. [0] https://www.ericsson.com/en/about-us/enterprise-security/pki https://www.ericsson.com/en/about-us/enterprise-security/pki [1] http://crl.ericsson.net/Ericsson_Software_Deliverable_Integrity_Protection_Root_CA_A1.crl http://crl.ericsson.net/Ericsson_Software_Deliverable_Integr... [2] https://crt.sh/?q=%.ericsson.net https://crt.sh/?q=%.ericsson.net [3] https://en.wikipedia.org/wiki/System_Architecture_Evolution https://en.wikipedia.org/wiki/System_Architecture_Evolution

stingraycharles 8y ago

This seems like a publicity stunt more than anything else. There's no way they actually lost 100MM here, and doing this ensures the public sees Ericsson as the real villain, and paints O2 as the victim. It's a very clever strategy, nonetheless.

djhworld 8y ago
I suspect business customers might have more of a stick to beat O2 over the head with, rather than average people with a cell phone contract. e.g. Transport for London's dot matrix displays could not show bus times for the 14 or so hours the network was down, apparently iPads issued to NHS staff to manage patient reports used O2 sim cards so that couldn't work either. Probably a whole range of small to medium enterprises affected too.
- isostatic 8y ago
  Our yodel driver reported they were struggling with both signing the packages as delivered and with drivers unable to cope without google maps.
jw1224 8y ago
Just some dirty back-of-the-napkin numbers, but theoretically, £100MM doesn't seem unreasonable. O2 have 25MM direct customers, plus another 7MM through reseller networks — so that's 32MM people directly affected by the outage. Presuming O2 receives an average of £30 per customer per month (which I think is a reasonable estimate), that's £940MM/month, working out to £32MM per day in a 30-day month. O2 are refunding customers 2 days' worth of service for the outage, so that would be £64MM in lost revenue. Add on top of that brand and reputation damage — and other factors I've no doubt forgotten to include — and £100MM doesn't seem unreasonable.
- chipperyman573 8y ago
  What does MM mean? Google wasn't able to help.

fogetti 8y ago

At the same time Softbank's network was also down at the same time in Japan for the same Ericsson issue. The scale is quite surprising.

kakkun 8y ago
Softbank's press release [0] mentions that a total of 11 countries were affected by the outage. I can't seem to find any information on what other countries were affected though. [0] https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20181206_02/ https://www.softbank.jp/en/corp/group/sbm/news/press/2018/20...
- sondh 8y ago
  Vietnam's second largest mobile network operator Mobifone was affected for 3 hours[1]. Now we know 3: UK, Japan and Vietnam. I'm sure it's possible track down the other 8. [1] (link in Vietnamese) https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nhieu-mang-di-dong-tren-toan-cau-20181207123405213.htm https://congnghe.tuoitre.vn/ericsson-xin-loi-ve-su-co-sap-nh...
  - ksec 8y ago
    Wow, Hong Kong's Smartone uses Ericsson and wasn't effected. I am surprised at how wide spread the problem is. How could Ericsson messed this up when the it has the biggest chance to grab business from Huawei.