Qantas check-in goes down (Merged)

[yo yo ma]

Check-in tech problems delay Qantas flights

After VB systems went down a few weeks ago, QF had troubles this morning.

On one of the busiest travel days of the year, the plans of thousands of domestic air travellers have been thrown into chaos at airports Australia-wide this morning, after another outage with check-in systems at Qantas.

A Qantas spokesman confirmed there was an intermittent outage with check-in systems this morning between 8.25am and 9.30am.

The global system, called Amadeus, is now working again but delays of up to an hour are still expected until mid-afternoon.

 

[YQY]

QF domestic baggage system down

The baggage system is down at qf domestic. Ques are mile long. Will post more from the lounge. In Sydney I meant

 

[YQY]

Re: QF domestic baggage system down

Update - you can only check in baggage for called flights. I'm at the j check in - approx 40 people are ahead. Baggage check in has stopped in the que. :evil:

 

[YQY]

Re: QF domestic baggage system down

Flights delayed by 40minutes.

 

[anat0l]

Re: QF domestic baggage system down

Oh this sucks royally.

Well, first DJ in MEL took a hit from a cut cable. Now QF takes a hit in SYD due to the baggage system going kaput.

 

[YQY]

Re: QF domestic baggage system down

Been infomed by an ABC journo - the computer check in has been down three times since this morning. Check in has restarted. No updates by qf on Twitter & SMS

 

[oz_mark]

Re: QF domestic baggage system down

Reported on another thread that there was a problem with Amadeus

 

[samh004]

Re: Qantas check-in goes down

Amadeus sounds like they need to improve their infrastructure a bit (assuming by global system there's a backend somewhere). Late-2009 and early-2010 outages aren't going to make their customers happy.

 

[YQY]

Re: QF domestic baggage system down

Finally checked in after 50 minutes. Apparently my flight QF443 is only 10 mins late....

Some pic's attached.

 

[drron]

Probably hasnt helped with BNE having just reopened after closing for a storm.Fortunately I am on DJ.

 

[anat0l]

What does Qantas rely on when communicating with Amadeus?

If it's a continuous link to maintain a license to use software, that's just dumb and I'd take that issue to Amadeus with a baseball bat.

If it's the flight data for customer reservations, then why couldn't this data be stored redundantly at some local servers (i.e. not one central Amadeus server) - a bit like mirroring?

And what are Amadeus - arguably one of the biggest GDSes in the world - doing with faulty servers without appropriate backup links and hardware? This isn't a Telstra severed cable thing like DJ's problem. Although the required redundancy equipment is probably very, very expensive and extensive, when you're a lifeblood for several major airlines around the world I think there is a case to be made, especially when your hardware fails and you cause ridiculous damage to all your customers!

Yes, I am satirising the complaints made when DJ lost the Telstra link to some degree....

 

[YQY]

Boarding now - only 30 mins late - good on you Qantas.

 

[drewbles]

What does Qantas rely on when communicating with Amadeus?

If it's a continuous link to maintain a license to use software, that's just dumb and I'd take that issue to Amadeus with a baseball bat.

If it's the flight data for customer reservations, then why couldn't this data be stored redundantly at some local servers (i.e. not one central Amadeus server) - a bit like mirroring?

And what are Amadeus - arguably one of the biggest GDSes in the world - doing with faulty servers without appropriate backup links and hardware? This isn't a Telstra severed cable thing like DJ's problem. Although the required redundancy equipment is probably very, very expensive and extensive, when you're a lifeblood for several major airlines around the world I think there is a case to be made, especially when your hardware fails and you cause ridiculous damage to all your customers!

Yes, I am satirising the complaints made when DJ lost the Telstra link to some degree....

Having worked on large data networks for many years, I can safely say, no matter how much redundancy you have in place, things can and do happen.

That said, who knows what is really the cause here. I do not know enough about distributed GDS systems (nor their design) to really say what's happened. Something as simple as a router dying somewhere can cause these issues. You can have multiple links for backup, but all it takes is one device upstream to fail for the proverbial to hit the fan.

Still, it's really not good enough. Something as mission-critical as the ability to get a pax onto a plane should see them having pretty much 0% downtime.

 

[anat0l]

Still, it's really not good enough. Something as mission-critical as the ability to get a pax onto a plane should see them having pretty much 0% downtime.

Indeed. Same with bank accounts and medical records. OK those other examples are more likely to be mission-critical than a mere airline's business.....

Not many tweets on the article posted by The Age, although this one did come through:

#fdlmw0 Chaos as check-in problems affect Qantas LOL!Why am I not surprised?what DU expect frm World's worst airline.But they'ii find an excuse

 

[markis10]

Will we see a PR piece from the engineers union pointing out that the computers were serviced overseas and that local engineer serviced computers would not have the same problem

 

[oz_mark]

And what are Amadeus - arguably one of the biggest GDSes in the world - doing with faulty servers without appropriate backup links and hardware?

I don't think there is enough information to know what happened. Given that it is the weekend, and the the middle of the night in the USA, I wouldn't discount the possibility of some maintenance happening that caused the problem.

 

[drewbles]

Indeed. Same with bank accounts and medical records. OK those other examples are more likely to be mission-critical than a mere airline's business.....

Speaking of banks, was just reading that the Bank of Queensland had an EFTPOS problem for people with their terminals. Some merchants were unable to accept electronic EFTPOS payments for 2 days. Now that's just not on either.

 

[NM]

I don't think there is enough information to know what happened. Given that it is the weekend, and the the middle of the night in the USA, I wouldn't discount the possibility of some maintenance happening that caused the problem.

I believe the Amadeus primary data centre is in Germany, not USA.

Maintenance on such systems will require notification to all the "customers" and should not involve any down-time. But if things do go wring during some scheduled maintenance/change/upgrade process, then unscheduled outage can result.

Having been involved with "mission-critical" systems, I know how difficult it is to predict and then mitigate all possible failure modes. At some point you have to accept a risk analysis and minimise the probability of a disruptive failure and then ensure the impact (both in duration and number of systems/people affected) of such a failure also minimised.

Amadeus claim 99.99% up-time, whoch equates to just under 1 hours per year of down-times. Looks like they have used their quota for 2010, so should all be good for the rest of the year

The RCA for the recent outages would make interesting reading, but I am sure it won't be made public.

 

[oz_mark]

Having worked on high availability systems, and had to write or contribute to more than one RCA, I am well aware that sometimes things don't go to plan. Suffice it to say, that in my experience most (but by no means all)outages were during or very shortly after some form of maintenance.

(or in one case where we did the regular checks to prove that the failover systems worked. We proved they didn't )

 

[Nigelinoz]

From Chaos as check-in problems affect Qantas

"It's an Amadeus server issue that meant that our check-in system was only intermittently available between 8.25am and 9.30am this morning," a Qantas spokesman said.
"The similar issue would have been experienced by other airlines globally using the same Amadeus system.
"We'll certainly be talking to Amadeus [management] and seeking assurances today's issue will be addressed."

I bet they will :!: (and rather forcefully I would expect).