VA/Sabre IT outage & major delays 21 May 2021

kyle

Established Member
Joined
Mar 8, 2006
Messages
1,365
Solutions
1
Points
420
The Age reports that they found the cause being a hardware redundancy failure. Does that mean there were 2 failures?
 

eastwest101

Established Member
Joined
Oct 26, 2010
Messages
3,259
Points
720
Qantas
Gold
Virgin
Gold
Couldnt have been much worse timing for VA given that there would have been a lot of voluntary and involuntary redundancies at the airline, thus taking the 'corporate knowledge' of how to revert to manual systems with them out the door, I suspect a lot of those staff who had seen Sabre outages before in the period 2015 to 2018 may not have been on duty last night.
 

henrus

Established Member
Joined
Jun 23, 2016
Messages
1,436
Solutions
8
Points
575
Qantas
Gold
Virgin
Gold
The Age reports that they found the cause being a hardware redundancy failure. Does that mean there were 2 failures?
This is what I heard (Google sabre outage), apparently some dell server issue. Question is why didn't the backup system kick in?
 
Get paid up to 25% in real cash from your everyday purchases from leading companies such as Virgin Australia, Booking.com, Coles, Apple, Microsoft and much more. Free to join and no catches!

AFF Supporters can remove this and all advertisements

Inlander

Member
Joined
Aug 19, 2019
Messages
127
Points
205
I was booked on VA1401 ex Adelaide to Bne today and received an email at 0600 this morning telling me it was cancelled and they have rebooked Me on VA1393 on Sunday! After waiting one hour on the phone ( Gold status), they cancelled and said they will issue a refund. Could not offer me another alternative back to Bne today so I booked with Skippy.
 

dk4

Established Member
Joined
Mar 23, 2010
Messages
1,616
Solutions
1
Points
480
... thus taking the 'corporate knowledge' of how to revert to manual systems with them out the door, I suspect a lot of those staff who had seen Sabre outages before in the period 2015 to 2018 may not have been on duty last night.
I remember checking in during an outage a few years back, and it was interesting to watch. From what I could ascertain, as a bored bystander in the queue;
  • whatever flights had not closed check-in prior to the system crash, had to be re-checked in manually. So passengers air-side and in the lounge, as well as those who had checked-in online, had to go to the check-in counters to get new boarding passes.
  • one land-side check-in counter is allocated per flight, and every passenger on that flight has to check-in through that single counter.
  • someone at HQ sends through a file containing all the passenger data for each individual flight, to the departure port. (So many files being sent out, one for each flight.)
  • departure ports deploy laptops to the check-in counters, and a runner brings through a flash drive with downloaded data file containing the specific flights information on it, which is inserted into the lap top. The slow process of check-in for that specific flight then begins.
  • There aren't many people in HQ available to download the passenger information from the back-ups, so flights seem to be prioritised by ticketed departure time, and across a national scale. Which means that each data file takes a while to slowly drip through to every departure port.
  • There are no public announcements about what is happening, as all local staff remain hopeful that the system will reboot and they won't have to implement manual processing.
  • Someone brings out water bottles for those stuck in the building lines, and if you are lucky an annoucement is made to stay in your specific check-in line, even if it is after your flight's ticketed departure time.
  • the same lap top that was used at the check-in counter, is brought to the gate for boarding management. (Which means that anyone who had a boarding pass issued before the system crash and didn't go back and get a new one, gets a new one at the gate anyway, hehe).
  • Flight departs - late.
 
Joined
Jul 18, 2013
Messages
418
Solutions
1
Points
220
Qantas
Gold
Virgin
Platinum
I wonder what the license agreements with Sabre contain... could be rather costly for Sabre's professional indemnity insurer (or them if self insured).
 

33kft

Established Member
Joined
Jun 19, 2018
Messages
1,241
Solutions
1
Points
575
apparently some dell server issue
My guess from reading the article is that it would be storage related. They referenced Dell/EMC and EMC is the storage company that Dell acquired back in 2016. It's a subsidiary of Dell Technologies and isnt how you would refer to most of Dell's server products. It's also a technology that could easily cause a several hour outage if something went wrong in either the storage network or in the arrays themselves.
 

kookaburra75

Established Member
Joined
Mar 26, 2017
Messages
1,266
Solutions
1
Points
695
Qantas
Platinum
Virgin
Gold
And to add insult to injury, at Brisbane Airport this morning when I arrived from Canberra, there was an announcement there had been a power spike at 7am which was affecting the Qantas and Virgin Check-In areas. There were some long queues at Virgin Check-In when I walked past. Deja vu for the poor Virgin staff.
 

ozstamps

Active Member
Joined
Aug 30, 2003
Messages
979
Points
320
Virgin
Platinum
Hrdlicka said that some will die... yesterday that just so happened to be their computer system.

Missed this by a day. TV reports had some planes sitting or tarmac for 2 or 3 hours, and not going back to gate.

Same first-hand reports said zero info passed onto pax via PA as to what was going on. Unacceptable.
 

henrus

Established Member
Joined
Jun 23, 2016
Messages
1,436
Solutions
8
Points
575
Qantas
Gold
Virgin
Gold
Same first-hand reports said zero info passed onto pax via PA as to what was going on. Unacceptable.
This seems to happen most times delays occur. VA has been getting better given the flight status page says the reason for the delay but I guess it doesn't help when your computers aren't working. At the end of the day, front-line staff can't really pass on info if they've got no idea themselves.

They do manual check-in and boarding practices throughout the year but clearly not enough. dk4 gave a recount of how they witnessed the manual process occurring, I suspect if they did it more often (as practice) then a few things could be improved in the process.

At most VA flights have 176 people so checking them in via a single counter/laptop shouldn't be that difficult. Things like receipt printers and barcode scanners are cheap these days so keeping a supply at airports wouldn't cost and greatly speed up the process.

That'd allow them to use the laptop to print receipt-style boarding passes that could be scanned during boarding and receipt-style bag tags that are stuck to bags instead. The amount of handwriting during these manual processes is shocking and whilst I've got no idea how these offline flight files come to the laptop, it'd take an IT intern to create some program that could be used offline to speed up this process.
 

Must...Fly!

Senior Member
Joined
Jan 12, 2010
Messages
5,893
Points
990
Qantas
Gold
Virgin
Platinum
Manual check-in, boarding, processing, loading, balancing is so unbelievably difficult. At a port the size of the capital cities with such limited gates, it's literally impossible to keep anything close to on-time. It's just the nature of the business. You can keep things moving but at such a pace that it almost feels like it isn't.

Back in my day we used to pick one flight a fortnight and do a completely manual process for it just for the experience. Even with a minimal load it was painful. Must say I was grateful to have never experienced a major outage.
 

AIRwin

Senior Member
Joined
Apr 13, 2013
Messages
5,080
Solutions
10
Points
985
Qantas
Gold
Virgin
Gold
Meanwhile, VA has renewed the global distribution agreement with Sabre:
 

henrus

Established Member
Joined
Jun 23, 2016
Messages
1,436
Solutions
8
Points
575
Qantas
Gold
Virgin
Gold
Meanwhile, VA has renewed the global distribution agreement with Sabre:
I wonder what happened during the administration? Is Sabre a creditor of the previous VA?

I guess they're both questions we'll never know but given the renewal nows the time for VA to invest a bit more in IT... being able to easily change bookings on a mobile or fly ahead shouldn't require assistance.
 

Bagpuss

Established Member
Joined
Feb 24, 2014
Messages
1,159
Solutions
3
Points
395
I wonder what happened during the administration? Is Sabre a creditor of the previous VA?

I guess they're both questions we'll never know but given the renewal nows the time for VA to invest a bit more in IT... being able to easily change bookings on a mobile or fly ahead shouldn't require assistance.
My assumption is that they are paid an upfront annual fee rather.

When going into voluntary administration some creditors arent overly affected as they affectively paid upto date. (Obviously salaries fall into this category.) Companies kinda sense what is happening and actually their last payment run will keep their key creditors good.

Virgin did used to chop and change with this and I doubt they would have wanted to go through another additional expense to change.
 
Top