Optus Outage - How is it affecting you?

[RooFlyer]

The cause of the outage is described in The Oz on-line:

Botched upgrade shut down Optus network, telco reveals, leaving it with a compo bill of up to $400m

Probably paywalled, sorry, so a few snippets:

Optus has finally revealed what caused its entire network to collapse, leaving more than 10 million Australians cut off from essential telecommunication services, as it faces a $400m compensation bill.

A routine software upgrade triggered a mass shutdown of routers across its network, effectively unplugging phone and internet services across the country. Some people couldn’t dial triple-zero for emergency services on fixed lines, while Melbourne’s train network was paralysed and phone lines at some hospitals were blocked among broader economic disruption.

Chief executive Kelly Bayer Rosmarin initially denied the outage, which began at 4:05am on Wednesday, was the result of a botched network upgrade.

“I can tell you that is not true,” she told The Australian later that day.

But an investigation from its engineers revealed in otherwise.

Good grief.

In an effort to cauterise the fallout, Optus has offered customers the equivalent of more than $100 in free data for eligible postpaid customers — an offer that has been roundly criticised after some small businesses lost $10,000 from the outage.

Optus says the offer is fair given the value in lost services would have equated to $1 per customer based on what it charges for its plans. But the small business ombudsman Bruce Billson said this only reflects what Optus charges — not how much customers lost from not being able to access its services.

 

[Pushka]

Optus have now offered our business a month credit for our phones and internet which equates to around $300. This was following a TIO complaint on Thursday. We had to purchase A Telstra plan and modem due to numerous planned business zooms that day.

 

[tomlee1986]

The cause of the outage is described in The Oz on-line:

Botched upgrade shut down Optus network, telco reveals, leaving it with a compo bill of up to $400m

Probably paywalled, sorry, so a few snippets:



Good grief.

free version from news.com.au

https://www.news.com.au/finance/economy/interest-rates/bumpy-road-ahead-on-inflation-senior-reserve-bank-official-warns/news-story/173eaba697bfef6a98472555761f62d9

Post automatically merged: Nov 13, 2023

Optus have now offered our business a month credit for our phones and internet which equates to around $300. This was following a TIO complaint on Thursday. We had to purchase A Telstra plan and modem due to numerous planned business zooms that day.

thats a start but the business lost cant be regained sadly

 

[Lynda2475]

"A routine software upgrade triggered a mass shutdown of routers across its network"

So exactly as I posted hours after it happened. Anyone who has ever worked closely with or for Optus knows router software changes happen over night Tues-Wed. The towers were up (one reason why failed to correctly fail-over to Telstra for emergency calls) but traffic couldnt get to the data centre because routers were failing to allow it, so couldnt validate services, patch connections etc.

"Optus has offered customers the equivalent of more than $100 in free data for eligible postpaid customers

Laughable, not valued anywhere near $100. More value would be received from refunding 1/30th of the monthly fee for consumers which would be somewhere around $1-2 for most but actual value as opposed to more data when most customers dont use anywhere the data inclusions already provided.

For businesses that couldnt take payments they lost far more than a months bill worth of sales.

 

[AIRwin]

I received the "We’re very sorry for the outage" email from Optus this morning offering at least 200GB of extra data

It's live on the app - if you activate before 31 December 2023 you get 100GB per billing cycle for three billing cycles:

 

[MARTINE]

I will post here as the Optus debacle prompted the discussion. No nbn in our area but we have/are v happy with our mobile broadband using amasym (aka Optus). We are coastal on a hill and thankfully a neighbor behind has the massive (cough) Optus tower that we don't see but gives us v v good fast (usually predictable) 5G. Have Telstra mobiles.
Q? - I am sorting out a backup for when this kind of outage happens (beyond my hotspot on phone).
Some great IT minds on this thread
Ideas for cheap modem/sim combo from Telstra?
Anyone have experience with StarLink - we have uninterrupted sky access!
Other ideas?

 

[downgraded]

I will post here as the Optus debacle prompted the discussion. No nbn in our area but we have/are v happy with our mobile broadband using amasym (aka Optus). We are coastal on a hill and thankfully a neighbor behind has the massive (cough) Optus tower that we don't see but gives us v v good fast (usually predictable) 5G. Have Telstra mobiles.
Q? - I am sorting out a backup for when this kind of outage happens (beyond my hotspot on phone).
Some great IT minds on this thread
Ideas for cheap modem/sim combo from Telstra?
Anyone have experience with StarLink - we have uninterrupted sky access!
Other ideas?

Boost is on the full Telstra network and offers low volume long expiry (12month) prepaid options.

 

[Pushka]

I will post here as the Optus debacle prompted the discussion. No nbn in our area but we have/are v happy with our mobile broadband using amasym (aka Optus). We are coastal on a hill and thankfully a neighbor behind has the massive (cough) Optus tower that we don't see but gives us v v good fast (usually predictable) 5G. Have Telstra mobiles.
Q? - I am sorting out a backup for when this kind of outage happens (beyond my hotspot on phone).
Some great IT minds on this thread
Ideas for cheap modem/sim combo from Telstra?
Anyone have experience with StarLink - we have uninterrupted sky access!
Other ideas?

It's quite interesting you ask this as thought I'd do a speed test on both Optus and our new Telstra. Optus speeds - I have never seen it go past 5mbps. Today it's 30mbps. Interesting. For comparison, Telstra, 245 mbps. Currently Telstra on low data but it will be cheaper than Optus when we upgrade for superior service.

 

[downgraded]

It's quite interesting you ask this as thought I'd do a speed test on both Optus and our new Telstra. Optus speeds - I have never seen it go past 5mbps. Today it's 30mbps. Interesting. For comparison, Telstra, 245 mbps. Currently Telstra on low data but it will be cheaper than Optus when we upgrade for superior service.

We have retained "old" Optus plans because they are not rate limited and still have more data than we use. I did a speed test on 5G the other day and got >600Mbps. However I'm now switching to Boost to give us the diversity (Mrs D will stay on Optus) but it still offers a rate cap off 150Mbps which is easily more than the average person needs anyway. The con that is the NBN sales pitch of faster is better is only generally so for the Telco in the vast majority of cases.

 

[Pushka]

We have retained "old" Optus plans because they are not rate limited and still have more data than we use. I did a speed test on 5G the other day and got >600Mbps. However I'm now switching to Boost to give us the diversity (Mrs D will stay on Optus) but it still offers a rate cap off 150Mbps which is easily more than the average person needs anyway. The con that is the NBN sales pitch of faster is better is only generally so for the Telco in the vast majority of cases.

Our Optus plan is execrable. They are upgrading it in our area. And have been doing so for, the year so far. Might just dump them.

 

[Must...Fly!]

Boost is on the full Telstra network and offers low volume long expiry (12month) prepaid options.

Yes, it's a good option. I've been on Boost for years, good discounts on the regular for the 12mth SIMs. Just keep an eye on OzBargain.

 

[Pushka]

Maybe she's completely bombed out with stress but goodness she is hardly an inspiring leader right now.

 

[RooFlyer]

Maybe she's completely bombed out with stress but goodness she is hardly an inspiring leader right now.

But I though she responded stoically under hostile questions from the 'Chair' of the Senate committee (not sure who that is).

Asked antagonistically again and again if she was trying to blame someone else for the 000 failure in mobiles, she calmly stated that the 000 system, where it should divert to an available network if the phone owner's chosen network is unavailable, was not run by Optus, so the failure is somewhere else.

And that makes sense. Say I have a phone on Optus. Their network goes down. I call 000. How is Optus involved at that point? They have lost connection with the handset. The phone should have thought 'Oh, 000 - I know that one. Can't connect to my paid network Optus, so what else can I see? Oh, a Telstra signal. I'll connect through that.' Sounds like a software fault in the phone.

BTW, does anyone know how the 000-hand-over mechanism is supposed to work (in layman's terms)?

And under the circumstances of the outage morning, you can see why she wasn't ringing around ministers etc early in the morning. Like the cyber attack a year ago, the SingTel board was visiting. So it wasn't unreasonable to suspect another cyber attack and to find out if that's the case before she started phoning Ministers. Because if she did call earlier, the first question would be "Is this another attack?" "Don't know yet" "Well, call me back when you do know something!!".

 

[Lynda2475]

Noting that at Senate Estimates KBR clarified that the 000 calls that failed were from Optus landlines (not Optus mobiles).

I would like to know the type of landline. Positive these were not the old school copper landlines which are virtually non-existent except in a handful of regional areas because these are not part of Optus' network. Therefore they had to be VOIP services; and one needs to understand if they were VOIP over nbn or VOIP over 4G/5G modem?

When the regulation re being able to make 000 from mobile even if no credit or sim card inserted was introduced in the 90s I worked on program that tested it and fail-over worked provided you were in range of a working mobile network.

Very few places in Aus where Optus would be the only mobile network within range. Generally f Optus has coverage so does Telstra.

 

[TheRealTMA]

While the outage is complete incompetence on the part of the technical section, as CEO I thought she did a much better job and explaining and responding than Joyce or any of the QF execs.

Presumably the 000 outages were not from mobile lines where even a sim is not required.

 

[Pushka]

Noting that at Senate Estimates KBR clarified that the 000 calls that failed were from Optus landlines (not Optus mobiles).

I would like to know the type of landline. Positive these were not the old school copper landlines which are virtually non-existent except in a handful of regional areas because these are not part of Optus' network. Therefore they had to be VOIP services; and one needs to understand if they were VOIP over nbn or VOIP over 4G/5G modem?

When the regulation re being able to make 000 from mobile even if no credit or sim card inserted was introduced in the 90s I worked on program that tested it and fail-over worked provided you were in range of a working mobile network.

Very few places in Aus where Optus would be the only mobile network within range. Generally f Optus has coverage so does Telstra.

I thought they were mobiles as well. They knew prior about the landline issue but not their mobile.

 

[RooFlyer]

While the outage is complete incompetence on the part of the technical section, as CEO I thought she did a much better job and explaining and responding than Joyce or any of the QF execs.

Presumably the 000 outages were not from mobile lines where even a sim is not required.

I've just gone through the SMH 'rolling coverage' - I agree, she handled it quite well, given (like Qantas) she was in front of a baying pack, seeking blood. The time line of discovery/recovery/advice she described, and what is in more detail in their submission (on-line at SMH, at least), was quite reasonable, considering.

Also, their argument about not automatically $ compensating customers for an outage seems more reasonable - not done by anyone else, including power companies etc. With power, we've grown to recognise that outages happen - and that's a monopoly supplier situation. At least with telcos most can have a back-up if they want (and which I am now arranging).

I thought they were mobiles as well. They knew prior about the landline issue but not their mobile.

Yes, I'm pretty sure it was initially reported that there were mobile 000 failures - and that was the 'scandal'. Moral: don't believe everything you read/hear in the media!!

 

[RooFlyer]

Link to Optus' written submission to the Senate Committee, via the SMH (opens PDF )

https://www.smh.com.au/interactive/hub/media/tearout-excerpt/23010/1.-Optus.pdf

 

[Pushka]

I've just gone through the SMH 'rolling coverage' - I agree, she handled it quite well, given (like Qantas) she was in front of a baying pack, seeking blood. The time line of discovery/recovery/advice she described, and what is in more detail in their submission (on-line at SMH, at least), was quite reasonable, considering.

Also, their argument about not automatically $ compensating customers for an outage seems more reasonable - not done by anyone else, including power companies etc. With power, we've grown to recognise that outages happen - and that's a monopoly supplier situation. At least with telcos most can have a back-up if they want (and which I am now arranging).



Yes, I'm pretty sure it was initially reported that there were mobile 000 failures - and that was the 'scandal'. Moral: don't believe everything you read/hear in the media!!

Or by what Optus itself announces then

 

[MelMel]

I've just gone through the SMH 'rolling coverage' - I agree, she handled it quite well, given (like Qantas) she was in front of a baying pack, seeking blood. The time line of discovery/recovery/advice she described, and what is in more detail in their submission (on-line at SMH, at least), was quite reasonable, considering.

Also, their argument about not automatically $ compensating customers for an outage seems more reasonable - not done by anyone else, including power companies etc. With power, we've grown to recognise that outages happen - and that's a monopoly supplier situation. At least with telcos most can have a back-up if they want (and which I am now arranging).



Yes, I'm pretty sure it was initially reported that there were mobile 000 failures - and that was the 'scandal'. Moral: don't believe everything you read/hear in the media!!

Off topic but Power companies do occasionally provide compensation for outages……