Report - IT 'glitch' grounds Qantas (resolved)

[RooFlyer]

In the Oz, on-line Qantas planes stranded after computer glitch

Thousands of Qantas passengers have been stuck in airports around the country after a nationwide computer glitch.

A large number of furious travellers have taken to social media to show their frustration on Sunday evening, desperate to get home in time for the working week, as packed planes were stranded on tarmacs across the country.

Cabin crew told passengers the “airline is working to fix the technical issue but that flights cannot take off until the system is back up and running”, passengers tweeted

Another passenger waiting to board added: ‘Well it just gets better from Qantas. They just announced to the whole lounge that there is a ‘nationwide IT issue’ and that everyone should wait in the lounge until further notice. Gulp.’

Qantas confirmed there was “a computer glitch, which has since been resolved”.

 

[SeatBackForward]

Did they try re-starting their airline?

 

[jpp42]

They really shouldn't let the work experience kids into the data centres...

 

[justinbrett]

90 minute downtime is pretty good for this kind of thing. Happens to everyone - VA and JQ have had them in recent years. Fact of life when you run large IT systems, they go down from time to time - even the banks.

 

[jb747]

Did they try re-starting their airline?

But it hung....and is now black screening.

 

[Quickstatus]

black

blue

 

[Quickstatus]

Fact of life

Yes.
They did get it up pretty quickly in the context of IT glitches. Unfortunately the airline is not getting any luck to go their way . Basically as they say still flying in IMC.

The banks and telcos are the ones we often hear of IT glitches. Whatever they are they often last for several hours. Or they can make you richer (at least temporarily)

Edit: Richer

 

[jpp42]

Or they can make you richer (at least temporarily)

Copy/paste error, that seems to be an eBay listing for QF pajamas?

Fact of life when you run large IT systems, they go down from time to time - even the banks.

No modern system should go down with current highly available architectures. I have worked in site reliability and if you are designing a new system from the ground up, this is an outdated mentality and should be avoided - there are plenty of ways to ensure pretty much 100% uptime if well-managed. However, this doesn't really apply to banks and airlines due to the legacy systems they are saddled with. Eventually we'll get there, but the industry is still another decade way, probably, at least for the airlines that lead in IT - Qantas will only come along kicking and screaming only after they are forced to. Certain banks have finally managed to replace their core banking with modern systems and you won't expect to see major outages again, as a result. But airlines are still on the back foot here.

 

[RichardMEL]

Copy/paste error, that seems to be an eBay listing for QF pajamas?



No modern system should go down with current highly available architectures. I have worked in site reliability and if you are designing a new system from the ground up, this is an outdated mentality and should be avoided - there are plenty of ways to ensure pretty much 100% uptime if well-managed. However, this doesn't really apply to banks and airlines due to the legacy systems they are saddled with. Eventually we'll get there, but the industry is still another decade way, probably, at least for the airlines that lead in IT - Qantas will only come along kicking and screaming only after they are forced to. Certain banks have finally managed to replace their core banking with modern systems and you won't expect to see major outages again, as a result. But airlines are still on the back foot here.

I wonder if it was an Amadeus outage as opposed to a more QF specific issue. Details seem a bit thin on the ground.. but most of these outages appear to have been caused more by vendors outside of the direct control of the airlines themselves (JQ,VA,QF etc) and the legacy is the key word. some of these things are held together by the virtual equivalent of duct tape. Specially as time goes on....

I have a friend who 20+ years ago was doing work on SABRE on a project for AA(and other client airlines) and it was super legacy back then... even the things they had to implment post 9/11 (eg: scanning PNRs for watchlist names and the like) - which would seem like reasonable trivial tasks.. were not. And that was yeas ago. Who knows what it's like now. Probably a nightmare.

the problem is the LCC's out there who can start up with their own platforms have an advantage, but when they decide to participate in the GDS to integrate with others then it all becomes far more difficult. Southwest (WN) is a classic example of this. They shunned the legacy platforms for decades and only relatively recently joined in and that gave all kinds of headaches to them. I was listening to a podcast interview with someone from WN who said that they decided to remove all expiry dates from their credits (a very customer friendly move) and while they could manage it on their systems reasonably easily, when it came to integration with the GDS... the implication this was not easy at all.. just due to the legacy nature.

This in turn is why JQ, for example, in general has vastly better systems that work more seamlessly for customers - they are built on different platforms and in the main are not tied to many things the legacy carriers are (but then you get the issues trying to integrate into QF and others as we all know).

tbh it must be a nightmare to work in this area to try and resolve all of the various issues that would be piling on, specially with things like covid and having to extend validity of credits over years and the like.

Not to defend anyone here, just as someone who is in the IT field I can absolutely relate in some respects and have some level of sympathy for some of the problems that must be constant and ongoing...

 

[Quickstatus]

Copy/paste error

Thank you - corrected in original post (see edit)

 

[OATEK]

Copy/paste error, that seems to be an eBay listing for QF pajamas?



No modern system should go down with current highly available architectures. I have worked in site reliability and if you are designing a new system from the ground up, this is an outdated mentality and should be avoided - there are plenty of ways to ensure pretty much 100% uptime if well-managed. However, this doesn't really apply to banks and airlines due to the legacy systems they are saddled with. Eventually we'll get there, but the industry is still another decade way, probably, at least for the airlines that lead in IT - Qantas will only come along kicking and screaming only after they are forced to. Certain banks have finally managed to replace their core banking with modern systems and you won't expect to see major outages again, as a result. But airlines are still on the back foot here.

Even AWS has been known to have outages. We are not at Nirvana yet.

The big issue is that big organisations (business and Government) don't just have legacy application systems, they have networks that have grown like topsy. Just takes a single static route to be missing after a reboot or change, and bingo!

QF is no different to so many organisations that have struggled to transition legacy industry-specific systems to transition to high availability platforms - keeps techies in business. But it would be interesting to know if it was an in house issue, or third party problem for QF.

 

[Quickstatus]

But it would be interesting to know if it was an in house issue, or third party problem for QF.

I think most of the QF IT department has been outsourced to Tata consultancy.

While some would decry the airline for outsourcing lots of its functions, I think it would be difficult even for a major public company to keep IT in-house.

 

[RichardMEL]

Even AWS has been known to have outages. We are not at Nirvana yet.

Yup. I was looking at some comments from a manager about how everything would go cloud based, clouds were reliable, robust and secure.. when Microsoft had a teams outage...... and that was a few days after CloudFlare had a outage too.

nothing's perfect. Getting better.. but not there yet.

The big issue is that big organisations (business and Government) don't just have legacy application systems, they have networks that have grown like topsy. Just takes a single static route to be missing after a reboot or change, and bingo!

QF is no different to so many organisations that have struggled to transition legacy industry-specific systems to transition to high availability platforms - keeps techies in business. But it would be interesting to know if it was an in house issue, or third party problem for QF.

Very much so.. and in environments where costs are being cut left right and centre and tech resources, brain drain means those who do know things are gone (and documentation can sometimes be.. interesting) that sometimes half the problem is locating the fault - specially when symptoms may produce red herrings (eg: your corporate website is failing to load - first thought is probably the server(s) behind the website, or a code issue... it may take time to figure out the web side is fine, but that back end tomcat connection to a third party database server has lost its mind and needs a kick up the backside.... or some network routing failed over to a backup which was blocked by a firewall rule change... so on and so forth). Even with strict change management policies and so on these things can still creep through.. and often times it comes down to those few people who know things inside out - that kind of knowledge time and experience yields that documentation can't replace.


Many moons ago I worked for a major Australian telco on a contract throhgh a third party that acted between the client telco and their systems provider (a large company with a well known three letter name... ). Anyway we had an outage one day where it turned out the CEO's morning briefing came up with all zeros. One does not like to see their $billions company suddenly seemingly reduced to zeros over their wheet-bix and coffee. Anyway it was all hands on deck to figure out the problem. I forgot what it was in the end, but it came down to the guy being paid $absurdamount on contract as basically "the guru" who knew EVERYTHING. And they paid this individual this insane amount of money simply to retain his knowledge and experience, because without him it would be 100x more difficult.

I bet this is not an isolated example. Maybe (hopefully!) less so these days!

 

[RooFlyer]

"Qantas. Our IT is just as bad as all the others".

"Remember when you couldn't use EFTPOS for a few hours? Well, us not being able to fly and your missing your international connection in Sydney, then not being able to get a substitute flight for a few days etc etc ... is just like that. No biggie."

While some would decry the airline for outsourcing lots of its functions, I think it would be difficult even for a major public company to keep IT in-house.

I guess it depends on the consequences one way or another. In-house, its there, the knowledge is there, at the direction of the airline when and where its needed. Outsource - presumably the out-sourced company has other clients; who gets the priority? Specialised airline knowledge, or just programmers? If there is a 'dedicated' outsourced team, why not have the dedicated team in-house. It could still be based in India.

 

[OATEK]

"Qantas. Our IT is just as bad as all the others".

"Remember when you couldn't use EFTPOS for a few hours? Well, us not being able to fly and your missing your international connection in Sydney, then not being able to get a substitute flight for a few days etc etc ... is just like that. No biggie."



I guess it depends on the consequences one way or another. In-house, its there, the knowledge is there, at the direction of the airline when and where its needed. Outsource - presumably the out-sourced company has other clients; who gets the priority? Specialised airline knowledge, or just programmers? If there is a 'dedicated' outsourced team, why not have the dedicated team in-house. It could still be based in India.

There is an old saying in IT - You can't outsource risk!

 

[justinbrett]

"Qantas. Our IT is just as bad as all the others".

"Remember when you couldn't use EFTPOS for a few hours? Well, us not being able to fly and your missing your international connection in Sydney, then not being able to get a substitute flight for a few days etc etc ... is just like that. No biggie."



I guess it depends on the consequences one way or another. In-house, its there, the knowledge is there, at the direction of the airline when and where its needed. Outsource - presumably the out-sourced company has other clients; who gets the priority? Specialised airline knowledge, or just programmers? If there is a 'dedicated' outsourced team, why not have the dedicated team in-house. It could still be based in India.

Outage finally resolved after Virgin booking system crashed causing travel chaos

Virgin Australia terminals around the country were faced with frustrated passengers after a global system outage.

7news.com.au

All the IT is as bad as the others.

 

[Pom-DownUnder]

The Multi-City is failing for me last night and today, this the same for everyone? goes straight to error 500

 

[RooFlyer]

The Multi-City is failing for me last night and today, this the same for everyone? goes straight to error 500

Might be better posted here:

The definitive thread on Qantas IT gripes.

First the good... the missing flights online claim form has been update to request both PNR and ticket number, which seems much more robust in allowing someone to track down the missing flight. My experience in the past has been a fail for the online form when they didn't request this info; not...

www.australianfrequentflyer.com.au

 

[Vic]

Can't see behind the paywall, was it a customer facing fail or a staff facing fail? The headline made me think it was something like can't do the aircraft loads. But that was a few days ago, not sure how I got that imporession.

 

[RooFlyer]

No mention of what the actual failing was, but it halted all of its aircraft movements (in Australia, I assume).