6 ms·

Millions of customers in Australia hit by mobile phone and internet outage

72 points by langfo 3y ago

PebblesHD 3y ago
Roughly a third of the entire country is currently without mobile service. The communication from Optus has been rather lacklustre.
asdefghyk 3y ago
Apparently about 10 million customers. or roughly 40% of population. More concerningly, the cause has not been identified yet, after 7 hours. Was identified as down at 3am? ( Maybe some maintenance / upgrade gone wrong? My company used to do such critical upgrades / changes at 2am on Tuesdays ... )
harrylepotter 3y ago
just went to the store to buy a Vodafone prepaid sim card. There were about 60 people in line. Point of sales are out in a lot of coffee shops. Utter chaos!
- sherry-sherry 3y ago
  Yeah but then you have to deal with Vodafone coverage. Get a full coverage (Telstra/Boost) or Telstra MVNO SIM (Belong/Aldi/Woolworths).
- usr1106 3y ago
  If you work in an emergency room or similar this could be a major nightmare. If you are a software developer it's best to take a day off or read a book (good for those who have bought a physical one). No reason to stress, isn't your project delayed by more than a day anyway?
  - sherry-sherry 3y ago
    Many phones support a second line with an SIM/eSIM setup. As most people still use a SIM in Australia, I wonder if having a second line with an eSIM on a very basic pre-paid plan would work for those people. In theory, you could have line 1 forward to line 2 when 'unreached'. Of course you'd want the second line to be different network.
    - seb1204 3y ago
      As far as I know you only get and eSIM with a full on plan with the 3 big telcos. So all prepaid users can't get an eSIM
      - sherry-sherry 3y ago
        That's true, even Telstra was slow getting to it (was only available for prepaid services for a long while). I think Aussie telcos need a bit of kick to really go in on eSIM, such as Apple releasing eSIM only iPhone's in the USA. You can't even get a add-on SIM for an Apple/Android Smartwatch from anyone except the big three here still. It would be good to have regulation surrounding eSIM, because telcos will always find a way to make it awful. E.g. able to do 24x7, max 1hr processing time, no cost to user, able to do via self-service options.
      - hooch 3y ago
        I have a Telstra prepaid esim.
sherry-sherry 3y ago
From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO --- ABC: Can you tell us why the system's gone down? Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible. ABC: Do they know what happened though? Who's there the team that you've got working on it? Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can. --- Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach https://en.wikipedia.org/wiki/2022_Optus_data_breach )
- usr1106 3y ago
  Not a surprise the CEO does not understand why their network is down. Many CEOs of companies of that size think business is moving money around, doing acquisitions etc. They have no clue how their production works or where it could be vulnerable. The information they get has been filtered through several levels of management, the cynic would say through several levels of of increasing incompetence in production matters.
- grecy 3y ago
  I worked for a large Telco (not Optus) and was constantly shocked by how little the management knew about anything. They hired a new Director of "Core Network", and in her first all-hands meeting she loudly asked what IPV6 was. (serious, she had never heard of it). The Director of IT had me setup her home wifi because she had no idea what DHCP was. The VP of "IT/Internet" literally had other people make her Powerpoints because she couldn't. She had never done anything more on a computer than send emails. It was shocking, but also really eye-opening to realize how important business acumen and playing the political game were.
superchroma 3y ago
I dropped optus like a hot coal after they leaked all my data and then lied to me that they hadn't. I'm surprised anyone is still with them.
- viraptor 3y ago
  Because having any competing company which fails sometimes it's still better than having just Telstra with free reign. Monopolies are not fun and will lose your data too.
  - superchroma 3y ago
    Optus cost me hundreds and made me wait for hours at the DMV to get a new driver's license. Telstra made me miserable but never that miserable.
unsignednoop 3y ago
[flagged]
- ta93754829 3y ago
  why would Israel want to bring down Optus in Australia? Seems like they've got better things to do at the moment
- grandinteg3 3y ago
  This person is being sarcastic - they have Israel on the mind. This is coming from a Jew who does not support Israel by the way (before anyone thinks I'm on their side).
justinclift 3y ago
If anyone in Aust is looking for a better internet service than Optus, then Launtel is very, very good: * https://launtel.net.au https://launtel.net.au * https://www.productreview.com.au/listings/launtel https://www.productreview.com.au/listings/launtel Note - I don't work for them or anything, am just a happy customer. :)
- aldonius 3y ago
  OK, but what about for mobile?
  - justinclift 3y ago
    No idea. I barely use mine, so don't have any solid recommendations. :/
clouddrover 3y ago
More comments here: https://news.ycombinator.com/item?id=38181986 https://news.ycombinator.com/item?id=38181986
dhx 3y ago
“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.” “Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.” [1] https://www.smh.com.au/technology/what-caused-the-optus-outage-20231108-p5eiep.html https://www.smh.com.au/technology/what-caused-the-optus-outa...
- __turbobrew__ 3y ago
  So there is one dude in a datacenter trying to plug in a db9 connector to get a serial connection in order to fix this?
  - sonicanatidae 3y ago
    Yes. lol #SysAdminLife
  - dhx 3y ago
    I haven't been able to find information on what part of their overall network is impacted. Per [1] they use a mixture of leased fibre and owned fibre for connectivity between Australian cities, and per [2] (note: zoom in) many point-to-point microwave links connecting mobile phone tower sites to the fibre footprint. It's potentially as bad as Optus needing to send a technician on a flight to a remote airfield somewhere in rural Australia, then driving 5 hours to a microwave tower to fix a router within the communications hut. Multiply by thousands of routers which are no doubt very much spread out. If Optus are lucky, it's just impacting routers for part of their overall network. For example, only impacting routers within data centres of the major Australian cities where you'd expect Optus to have technicians nearby who could physically access the router with 1-2 hours notice. [1] https://www.optus.com.au/dafiles/OCA/Wholesale/ProductAndServices/DataSolutions/StaticFiles/Documents/OP4213%20OW%20Int%20Backhaul%20HR.pdf https://www.optus.com.au/dafiles/OCA/Wholesale/ProductAndSer... [2] https://maprad.io/us/search/text/optus?filters=WyJkdl9saWNlbmNlX3N1YnR5cGU6ZjpQb2ludCB0byBQb2ludCJd&source=AU#app https://maprad.io/us/search/text/optus?filters=WyJkdl9saWNlb...
AnotherGoodName 3y ago
The same provider leaked all customer data in very recent history
kiwijamo 3y ago
"One Optus customer, Annie, told ABC Radio she found out about the issues through her cat, who is fed through an automatic Wi-Fi feeder, and missed being fed due to the outage." Source: https://www.stuff.co.nz/world/australia/301004965/major-optus-outage-millions-unable-to-make-calls-including-000-or-use-internet-in-australia https://www.stuff.co.nz/world/australia/301004965/major-optu...
- swader999 3y ago
  Catastrophic failure.
- lloydatkinson 3y ago
  Internet of Shit strikes again. It’s just insane to me that anyone makes an embedded system that isn’t capable of tracking time and it’s own schedule. I’m sure some useless agile product owner scrum type person came up with the brilliant idea of needing an internet connection just to turn the motor on and off.
mikelward 3y ago
> All of Melbourne's train services stopped temporarily due to a communications outage, and Metro Trains said it was unable to rely on its back-up system, which used the Optus mobile network. Their primary and their backup were both Optus. Not much of a backup.
- Gigachad 3y ago
  I wonder what they ended up doing to resolve it. There was a disruption but it was for less than an hour. Someone wifi hotspot the train network from their phone or something.

72 points by langfo 3y ago

Roughly a third of the entire country is currently without mobile service. The communication from Optus has been rather lacklustre.

asdefghyk 3y ago

Apparently about 10 million customers. or roughly 40% of population. More concerningly, the cause has not been identified yet, after 7 hours. Was identified as down at 3am? ( Maybe some maintenance / upgrade gone wrong? My company used to do such critical upgrades / changes at 2am on Tuesdays ... )

harrylepotter 3y ago

just went to the store to buy a Vodafone prepaid sim card. There were about 60 people in line. Point of sales are out in a lot of coffee shops. Utter chaos!

sherry-sherry 3y ago
Yeah but then you have to deal with Vodafone coverage. Get a full coverage (Telstra/Boost) or Telstra MVNO SIM (Belong/Aldi/Woolworths).
usr1106 3y ago
If you work in an emergency room or similar this could be a major nightmare. If you are a software developer it's best to take a day off or read a book (good for those who have bought a physical one). No reason to stress, isn't your project delayed by more than a day anyway?
- sherry-sherry 3y ago
  Many phones support a second line with an SIM/eSIM setup. As most people still use a SIM in Australia, I wonder if having a second line with an eSIM on a very basic pre-paid plan would work for those people. In theory, you could have line 1 forward to line 2 when 'unreached'. Of course you'd want the second line to be different network.
  - seb1204 3y ago
    As far as I know you only get and eSIM with a full on plan with the 3 big telcos. So all prepaid users can't get an eSIM
    - sherry-sherry 3y ago
      That's true, even Telstra was slow getting to it (was only available for prepaid services for a long while). I think Aussie telcos need a bit of kick to really go in on eSIM, such as Apple releasing eSIM only iPhone's in the USA. You can't even get a add-on SIM for an Apple/Android Smartwatch from anyone except the big three here still. It would be good to have regulation surrounding eSIM, because telcos will always find a way to make it awful. E.g. able to do 24x7, max 1hr processing time, no cost to user, able to do via self-service options.
    - hooch 3y ago
      I have a Telstra prepaid esim.

sherry-sherry 3y ago

From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO --- ABC: Can you tell us why the system's gone down? Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible. ABC: Do they know what happened though? Who's there the team that you've got working on it? Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can. --- Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach https://en.wikipedia.org/wiki/2022_Optus_data_breach )

usr1106 3y ago
Not a surprise the CEO does not understand why their network is down. Many CEOs of companies of that size think business is moving money around, doing acquisitions etc. They have no clue how their production works or where it could be vulnerable. The information they get has been filtered through several levels of management, the cynic would say through several levels of of increasing incompetence in production matters.
grecy 3y ago
I worked for a large Telco (not Optus) and was constantly shocked by how little the management knew about anything. They hired a new Director of "Core Network", and in her first all-hands meeting she loudly asked what IPV6 was. (serious, she had never heard of it). The Director of IT had me setup her home wifi because she had no idea what DHCP was. The VP of "IT/Internet" literally had other people make her Powerpoints because she couldn't. She had never done anything more on a computer than send emails. It was shocking, but also really eye-opening to realize how important business acumen and playing the political game were.

superchroma 3y ago

I dropped optus like a hot coal after they leaked all my data and then lied to me that they hadn't. I'm surprised anyone is still with them.

viraptor 3y ago
Because having any competing company which fails sometimes it's still better than having just Telstra with free reign. Monopolies are not fun and will lose your data too.
- superchroma 3y ago
  Optus cost me hundreds and made me wait for hours at the DMV to get a new driver's license. Telstra made me miserable but never that miserable.

unsignednoop 3y ago

[flagged]

ta93754829 3y ago
why would Israel want to bring down Optus in Australia? Seems like they've got better things to do at the moment
grandinteg3 3y ago
This person is being sarcastic - they have Israel on the mind. This is coming from a Jew who does not support Israel by the way (before anyone thinks I'm on their side).

justinclift 3y ago

If anyone in Aust is looking for a better internet service than Optus, then Launtel is very, very good: * https://launtel.net.au https://launtel.net.au * https://www.productreview.com.au/listings/launtel https://www.productreview.com.au/listings/launtel Note - I don't work for them or anything, am just a happy customer. :)

aldonius 3y ago
OK, but what about for mobile?
- justinclift 3y ago
  No idea. I barely use mine, so don't have any solid recommendations. :/

clouddrover 3y ago

More comments here: https://news.ycombinator.com/item?id=38181986 https://news.ycombinator.com/item?id=38181986

dhx 3y ago

“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.” “Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.” [1] https://www.smh.com.au/technology/what-caused-the-optus-outage-20231108-p5eiep.html https://www.smh.com.au/technology/what-caused-the-optus-outa...

__turbobrew__ 3y ago
So there is one dude in a datacenter trying to plug in a db9 connector to get a serial connection in order to fix this?
- sonicanatidae 3y ago
  Yes. lol #SysAdminLife
- dhx 3y ago
  I haven't been able to find information on what part of their overall network is impacted. Per [1] they use a mixture of leased fibre and owned fibre for connectivity between Australian cities, and per [2] (note: zoom in) many point-to-point microwave links connecting mobile phone tower sites to the fibre footprint. It's potentially as bad as Optus needing to send a technician on a flight to a remote airfield somewhere in rural Australia, then driving 5 hours to a microwave tower to fix a router within the communications hut. Multiply by thousands of routers which are no doubt very much spread out. If Optus are lucky, it's just impacting routers for part of their overall network. For example, only impacting routers within data centres of the major Australian cities where you'd expect Optus to have technicians nearby who could physically access the router with 1-2 hours notice. [1] https://www.optus.com.au/dafiles/OCA/Wholesale/ProductAndServices/DataSolutions/StaticFiles/Documents/OP4213%20OW%20Int%20Backhaul%20HR.pdf https://www.optus.com.au/dafiles/OCA/Wholesale/ProductAndSer... [2] https://maprad.io/us/search/text/optus?filters=WyJkdl9saWNlbmNlX3N1YnR5cGU6ZjpQb2ludCB0byBQb2ludCJd&source=AU#app https://maprad.io/us/search/text/optus?filters=WyJkdl9saWNlb...

AnotherGoodName 3y ago

The same provider leaked all customer data in very recent history

kiwijamo 3y ago

"One Optus customer, Annie, told ABC Radio she found out about the issues through her cat, who is fed through an automatic Wi-Fi feeder, and missed being fed due to the outage." Source: https://www.stuff.co.nz/world/australia/301004965/major-optus-outage-millions-unable-to-make-calls-including-000-or-use-internet-in-australia https://www.stuff.co.nz/world/australia/301004965/major-optu...

swader999 3y ago
Catastrophic failure.
lloydatkinson 3y ago
Internet of Shit strikes again. It’s just insane to me that anyone makes an embedded system that isn’t capable of tracking time and it’s own schedule. I’m sure some useless agile product owner scrum type person came up with the brilliant idea of needing an internet connection just to turn the motor on and off.

mikelward 3y ago

> All of Melbourne's train services stopped temporarily due to a communications outage, and Metro Trains said it was unable to rely on its back-up system, which used the Optus mobile network. Their primary and their backup were both Optus. Not much of a backup.

Gigachad 3y ago
I wonder what they ended up doing to resolve it. There was a disruption but it was for less than an hour. Someone wifi hotspot the train network from their phone or something.