VA/Sabre IT outage & major delays 21 May 2021

[kyle]

The Age reports that they found the cause being a hardware redundancy failure. Does that mean there were 2 failures?

 

[eastwest101]

Couldnt have been much worse timing for VA given that there would have been a lot of voluntary and involuntary redundancies at the airline, thus taking the 'corporate knowledge' of how to revert to manual systems with them out the door, I suspect a lot of those staff who had seen Sabre outages before in the period 2015 to 2018 may not have been on duty last night.

 

[henrus]

The Age reports that they found the cause being a hardware redundancy failure. Does that mean there were 2 failures?

This is what I heard (Google sabre outage), apparently some dell server issue. Question is why didn't the backup system kick in?

 

[Inlander]

I was booked on VA1401 ex Adelaide to Bne today and received an email at 0600 this morning telling me it was cancelled and they have rebooked Me on VA1393 on Sunday! After waiting one hour on the phone ( Gold status), they cancelled and said they will issue a refund. Could not offer me another alternative back to Bne today so I booked with Skippy.

 

[dk4]

... thus taking the 'corporate knowledge' of how to revert to manual systems with them out the door, I suspect a lot of those staff who had seen Sabre outages before in the period 2015 to 2018 may not have been on duty last night.

I remember checking in during an outage a few years back, and it was interesting to watch. From what I could ascertain, as a bored bystander in the queue;

• whatever flights had not closed check-in prior to the system crash, had to be re-checked in manually. So passengers air-side and in the lounge, as well as those who had checked-in online, had to go to the check-in counters to get new boarding passes.
• one land-side check-in counter is allocated per flight, and every passenger on that flight has to check-in through that single counter.
• someone at HQ sends through a file containing all the passenger data for each individual flight, to the departure port. (So many files being sent out, one for each flight.)
• departure ports deploy laptops to the check-in counters, and a runner brings through a flash drive with downloaded data file containing the specific flights information on it, which is inserted into the lap top. The slow process of check-in for that specific flight then begins.
• There aren't many people in HQ available to download the passenger information from the back-ups, so flights seem to be prioritised by ticketed departure time, and across a national scale. Which means that each data file takes a while to slowly drip through to every departure port.
• There are no public announcements about what is happening, as all local staff remain hopeful that the system will reboot and they won't have to implement manual processing.
• Someone brings out water bottles for those stuck in the building lines, and if you are lucky an annoucement is made to stay in your specific check-in line, even if it is after your flight's ticketed departure time.
• the same lap top that was used at the check-in counter, is brought to the gate for boarding management. (Which means that anyone who had a boarding pass issued before the system crash and didn't go back and get a new one, gets a new one at the gate anyway, hehe).
• Flight departs - late.

 

[Carrots]

I wonder what the license agreements with Sabre contain... could be rather costly for Sabre's professional indemnity insurer (or them if self insured).

 

[33kft]

apparently some dell server issue

My guess from reading the article is that it would be storage related. They referenced Dell/EMC and EMC is the storage company that Dell acquired back in 2016. It's a subsidiary of Dell Technologies and isnt how you would refer to most of Dell's server products. It's also a technology that could easily cause a several hour outage if something went wrong in either the storage network or in the arrays themselves.

 

[kookaburra75]

And to add insult to injury, at Brisbane Airport this morning when I arrived from Canberra, there was an announcement there had been a power spike at 7am which was affecting the Qantas and Virgin Check-In areas. There were some long queues at Virgin Check-In when I walked past. Deja vu for the poor Virgin staff.

 

[ozstamps]

Hrdlicka said that some will die... yesterday that just so happened to be their computer system.

Missed this by a day. TV reports had some planes sitting or tarmac for 2 or 3 hours, and not going back to gate.

Same first-hand reports said zero info passed onto pax via PA as to what was going on. Unacceptable.

 

[henrus]

Same first-hand reports said zero info passed onto pax via PA as to what was going on. Unacceptable.

This seems to happen most times delays occur. VA has been getting better given the flight status page says the reason for the delay but I guess it doesn't help when your computers aren't working. At the end of the day, front-line staff can't really pass on info if they've got no idea themselves.

They do manual check-in and boarding practices throughout the year but clearly not enough. dk4 gave a recount of how they witnessed the manual process occurring, I suspect if they did it more often (as practice) then a few things could be improved in the process.

At most VA flights have 176 people so checking them in via a single counter/laptop shouldn't be that difficult. Things like receipt printers and barcode scanners are cheap these days so keeping a supply at airports wouldn't cost and greatly speed up the process.

That'd allow them to use the laptop to print receipt-style boarding passes that could be scanned during boarding and receipt-style bag tags that are stuck to bags instead. The amount of handwriting during these manual processes is shocking and whilst I've got no idea how these offline flight files come to the laptop, it'd take an IT intern to create some program that could be used offline to speed up this process.

 

[Must...Fly!]

Manual check-in, boarding, processing, loading, balancing is so unbelievably difficult. At a port the size of the capital cities with such limited gates, it's literally impossible to keep anything close to on-time. It's just the nature of the business. You can keep things moving but at such a pace that it almost feels like it isn't.

Back in my day we used to pick one flight a fortnight and do a completely manual process for it just for the experience. Even with a minimal load it was painful. Must say I was grateful to have never experienced a major outage.

 

[AIRwin]

This afternoon's outage has been resolved:

Virgin Australia statement on IT system outage

Virgin Australia is pleased to confirm that an IT outage has been resolved and our website and guest contact centres are operating as normal.

newsroom.virginaustralia.com

 

[eastwest101]

It wasn't just Virgin Australia that got hit, its no coincidence that Telstra and many banks all got hit today.

http://amp.abc.net.au/article/100223900&ved=2ahUKEwjc3f36wJ7xAhV9xzgGHTAXCbIQ0PADegQIMBAB&usg=AOvVaw0oXH3i32fKJfpjWhZxDYV6

 

[AIRwin]

Meanwhile, VA has renewed the global distribution agreement with Sabre:

Sabre Global Office Locations & Corporate Headquarters | Sabre

Find Sabre Corporation office locations and contact details around the world. Connect with our regional headquarters in the US, EMEA, APAC, and Latin America.

www.sabre.com

 

[henrus]

Meanwhile, VA has renewed the global distribution agreement with Sabre:

Sabre Global Office Locations & Corporate Headquarters | Sabre

Find Sabre Corporation office locations and contact details around the world. Connect with our regional headquarters in the US, EMEA, APAC, and Latin America.

www.sabre.com

I wonder what happened during the administration? Is Sabre a creditor of the previous VA?

I guess they're both questions we'll never know but given the renewal nows the time for VA to invest a bit more in IT... being able to easily change bookings on a mobile or fly ahead shouldn't require assistance.

 

[Bagpuss]

I wonder what happened during the administration? Is Sabre a creditor of the previous VA?

I guess they're both questions we'll never know but given the renewal nows the time for VA to invest a bit more in IT... being able to easily change bookings on a mobile or fly ahead shouldn't require assistance.

My assumption is that they are paid an upfront annual fee rather.

When going into voluntary administration some creditors arent overly affected as they affectively paid upto date. (Obviously salaries fall into this category.) Companies kinda sense what is happening and actually their last payment run will keep their key creditors good.

Virgin did used to chop and change with this and I doubt they would have wanted to go through another additional expense to change.