5 ms·

Virgin America's Web Meltdown: Four Weeks and Counting

40 points by technologizer 15y ago

jtchang 15y ago
As a technologist my sympathy goes out to Virgin. Airline reservation systems are some of the most complicated beasts we've ever created. The risks of migrating are enormous and with the busiest travel days ahead you can bet that David Cush has every resource available trying to fix the issues. That said it is never easy. The changes may amount to just a handful of lines but understanding such systems is key. My most recent encounter with bugs was that the system couldn't find a reservation for me even though I made one while authenticated. I just plugged in the confirmation number manually and that seemed to work. I didn't think too much of it though...maybe because I am use to mediocrity in the airline industry as a whole.
- namdnay 15y ago
  It's funny how until you are in a business, you don't realise how monstrously complicated it can be. I work for another GDS (if Sabre is an "800 pound gorilla", I suppose that makes us a small elephant), and the amount of work that goes into a cutover is just mind-boggling. And if you think this type of transition is dangerous, consider the fact that this is a switch of the reservation system, not the DCS (departure control). That's when things can get really messy...
  - numlocked 15y ago
    A deep-linked article had a good quote about the complexity of cut-over transition like this: "It is the business equivalent of taking out the brain of a human being, while they are laying on the table and you're having a conversation with them."
- waterside81 15y ago
  Having worked with Sabre and Gallileo (did they merge?), I, too, can attest to the fun and joy of working with airline res systems. Wacky XML, fixed length fields, unique fields that aren't really unique. It's a right of passage that most devs should take so that they (1) Promise never to build something like this and (2) Learn to never work in travel.
  - namdnay 15y ago
    The problem is that we're having to keep perfect compatibility with stuff that was done in the seventies. So you quickly end up with something just as (if not more) horrendous as the Windows or Office codebase. Take EDIFACT for example...
    - waterside81 15y ago
      Oh for sure compatibility is a huge issue and getting airlines to update and keep in step with things like herding cats. I don't envy the role of a dev who works for one of the large GDS's
  - r00fus 15y ago
    > Wacky XML, fixed length fields, unique fields that aren't really unique. If you've ever done enterprise integration, sadly, this is far from exceptional - in fact, it's the norm.
- mirkules 15y ago
  Take note: this is a great example of how customer service should be handled. 1) Explain what went wrong and why 2) Explain the countermeasures and how long it will take (obviously, they didn't meet their goal -- in that case, a note explaining why more time is needed) 3) Increase support staff, especially if there is an influx of callers because of the problems 4) Offer your customers something of value as an apology 5) Bonus: CEO makes himself available to small-time reporters. I think this step alone shows incredible goodwill. The first post on this thread starts with "As a technologist my sympathy goes out to Virgin." I can't imagine the same response if the company was not so open through the entire process (Delta comes to mind).
  - silencio 15y ago
    Re: point number 3 I especially love how their support is actually fixing problems. When I had a lot of trouble booking my flight and idly complained on Twitter in the middle of the night, I had a response within hours and eventually after a couple emails got to the point where I was on the phone with someone who called, at a time convenient for me, to book my flight for practically free for the inconvenience I had dealt with with exactly the flight and seats I wanted. The reason I try not to fly with any non-Virgin America airline is because I don't believe that any traditional airline would have listened on Twitter and then called me at my leisure to fix their problem and book a $300 flight for chump change in taxes and frequent flier points in order to book the flight for me. I don't know if I'm just getting the special treatment for someone that has flown over 2x roundtrips a month with Virgin, but I've gotten this kind of amazing treatment from them and their Twitter team when I only flew a couple times a year too. Despite the ups and downs, I couldn't be happier.
raganwald 15y ago
Ah, happy memories of making a big mistake: providing an online airline reservation system with an email address containing a “+”. http://weblog.raganwald.com/2007/09/you-suck.html http://weblog.raganwald.com/2007/09/you-suck.html
- davedx 15y ago
  Beautiful. Several times in my career I've passionately resisted that kind of stupidity, most recently concerning phone numbers (we should only accept numeric characters, right?!). Sigh...
- pavel_lishin 15y ago
  Running into this type of problem finally convinced me to set up a catch-all for my domain. I'm no longer registering as pavel.lishin+example@example.com, but as example@lishin.org.
erickhill 15y ago
It's their entire system, not just the customer-facing site. "The woes stem from Virgin’s switchover from its old reservation system, iFly, to the industry’s 800-pound gorilla, Sabre." As stated in the article, "...modern businesses, of all sorts, don’t have Web sites. To a remarkable degree, they are Web sites." Even at the airport if you go to a kiosk to check-in with a bag, you won't be able to print out a boarding pass. You'll have to go stand in line.
- funkah 15y ago
  You can print out the boarding pass, but you need to stand in the line for the bag check ticket, and so they can check your ID before putting the bag on the conveyor belt (or telling you to lug your bag to the conveyor belt around the corner).
- namdnay 15y ago
  On most systems, reservation is more or less independent from departure control. The kiosks might be plugged straight into the DCS, in which case there would be no issue son that side.
- nlh 15y ago
  The woes stem from Virgin’s switchover from its old reservation system, iFly, to the industry’s 800-pound gorilla, Sabre Ah. I have a hyphenated last name (it's been in my family since the late 1800s), and I was wondering why, on my most recent VA reservation, I was suddenly greeted with the all-too-familiar "Your last name contains invalid characters". Thanks, Sabre. Virgin's old system supported my real name (as most on the web do, as they should). It's remarkable to me that the industry's 800lb gorilla can't support something as simple as a "-" character in a last name, something that's fairly common (or at least not unheard of).
ibejoeb 15y ago
Thanks for the link; I was wondering what the hell has been going on. I've been flying Virgina for a while, and I flew on the weekend of the transition. I think I slipped my online ticket sale in just in time, but check-in, seat selection, frequent flyer info: all screwed. I'm sure this is a tough job, but I wish they would have disabled the features rather than let them fail. The first several times I tried and failed, I assumed it was a fluke. That these thing sometimes worked and sometimes didn't led me to believe that these were ephemeral problems, and I must have wasted hours retrying things. If that had been the case, I would have just arrived early and done it the old-fashioned. Ultimately, that's what I did, although their internal systems are just as screwed (and I don't think hand-written passenger manifests cut the muster anymore). Anyway, on my second pass through this mess, I just skipped the website entirely and did everything at the airport. Took a little longer, but it was pretty stress-free. I'd recommend doing so if you're flying with them soon. To their credit, they did give us miles and candy, and the staff is still the best in the biz. And, if you're flying through SFO, you can make the best of your residual time with a solid burger and a decent drink at Lark Creek Grill.
bengl3rt 15y ago
It seems to me that Sabre et al are all overly complex, left over from the days when you had to physically go to a travel agent's office to watch them clatter incantations into a monochrome terminal as they manually assembled your itinerary... The amount of complexity and arbitrage that goes on in selling plane tickets today is just nuts. I don't like actually flying Southwest, but I think they have the right idea with regards to booking - always, and only, on their website. Ensures a consistent, branded experience and saves them the headache of listing in a GDS.
- namdnay 15y ago
  For "regional" airlines such as SouthWest or Ryanair, managing your own system is fine. But if you want to be able to do code-sharing, ground-handling and all the rest of the madness that comes with being a global airline, you don't have a choice. The single website solution also means you're cutting yourself off from people who are using fare search services. And let's not forget that in some regions (Asia notably), travel agents with their "cryptic" terminals and "valuable" tickets are still the norm.
zbruhnke 15y ago
Been back and forth to DFW and SFO about 6 times in the last few weeks all on virgin ... for those flights I have probably spent a combined 5 hours on hold for one small change or a misbooked flight etc. but their $139 first class upgrades and friendly staff almost always make it all better for me. I'll be glad when they get this all worked out thats for sure
gujk 15y ago
Why would Virgin migrate to SABRE??

40 points by technologizer 15y ago

As a technologist my sympathy goes out to Virgin. Airline reservation systems are some of the most complicated beasts we've ever created. The risks of migrating are enormous and with the busiest travel days ahead you can bet that David Cush has every resource available trying to fix the issues. That said it is never easy. The changes may amount to just a handful of lines but understanding such systems is key. My most recent encounter with bugs was that the system couldn't find a reservation for me even though I made one while authenticated. I just plugged in the confirmation number manually and that seemed to work. I didn't think too much of it though...maybe because I am use to mediocrity in the airline industry as a whole.

namdnay 15y ago
It's funny how until you are in a business, you don't realise how monstrously complicated it can be. I work for another GDS (if Sabre is an "800 pound gorilla", I suppose that makes us a small elephant), and the amount of work that goes into a cutover is just mind-boggling. And if you think this type of transition is dangerous, consider the fact that this is a switch of the reservation system, not the DCS (departure control). That's when things can get really messy...
- numlocked 15y ago
  A deep-linked article had a good quote about the complexity of cut-over transition like this: "It is the business equivalent of taking out the brain of a human being, while they are laying on the table and you're having a conversation with them."
waterside81 15y ago
Having worked with Sabre and Gallileo (did they merge?), I, too, can attest to the fun and joy of working with airline res systems. Wacky XML, fixed length fields, unique fields that aren't really unique. It's a right of passage that most devs should take so that they (1) Promise never to build something like this and (2) Learn to never work in travel.
- namdnay 15y ago
  The problem is that we're having to keep perfect compatibility with stuff that was done in the seventies. So you quickly end up with something just as (if not more) horrendous as the Windows or Office codebase. Take EDIFACT for example...
  - waterside81 15y ago
    Oh for sure compatibility is a huge issue and getting airlines to update and keep in step with things like herding cats. I don't envy the role of a dev who works for one of the large GDS's
- r00fus 15y ago
  > Wacky XML, fixed length fields, unique fields that aren't really unique. If you've ever done enterprise integration, sadly, this is far from exceptional - in fact, it's the norm.
mirkules 15y ago
Take note: this is a great example of how customer service should be handled. 1) Explain what went wrong and why 2) Explain the countermeasures and how long it will take (obviously, they didn't meet their goal -- in that case, a note explaining why more time is needed) 3) Increase support staff, especially if there is an influx of callers because of the problems 4) Offer your customers something of value as an apology 5) Bonus: CEO makes himself available to small-time reporters. I think this step alone shows incredible goodwill. The first post on this thread starts with "As a technologist my sympathy goes out to Virgin." I can't imagine the same response if the company was not so open through the entire process (Delta comes to mind).
- silencio 15y ago
  Re: point number 3 I especially love how their support is actually fixing problems. When I had a lot of trouble booking my flight and idly complained on Twitter in the middle of the night, I had a response within hours and eventually after a couple emails got to the point where I was on the phone with someone who called, at a time convenient for me, to book my flight for practically free for the inconvenience I had dealt with with exactly the flight and seats I wanted. The reason I try not to fly with any non-Virgin America airline is because I don't believe that any traditional airline would have listened on Twitter and then called me at my leisure to fix their problem and book a $300 flight for chump change in taxes and frequent flier points in order to book the flight for me. I don't know if I'm just getting the special treatment for someone that has flown over 2x roundtrips a month with Virgin, but I've gotten this kind of amazing treatment from them and their Twitter team when I only flew a couple times a year too. Despite the ups and downs, I couldn't be happier.

raganwald 15y ago

Ah, happy memories of making a big mistake: providing an online airline reservation system with an email address containing a “+”. http://weblog.raganwald.com/2007/09/you-suck.html http://weblog.raganwald.com/2007/09/you-suck.html

davedx 15y ago
Beautiful. Several times in my career I've passionately resisted that kind of stupidity, most recently concerning phone numbers (we should only accept numeric characters, right?!). Sigh...
pavel_lishin 15y ago
Running into this type of problem finally convinced me to set up a catch-all for my domain. I'm no longer registering as pavel.lishin+example@example.com, but as example@lishin.org.

erickhill 15y ago

It's their entire system, not just the customer-facing site. "The woes stem from Virgin’s switchover from its old reservation system, iFly, to the industry’s 800-pound gorilla, Sabre." As stated in the article, "...modern businesses, of all sorts, don’t have Web sites. To a remarkable degree, they are Web sites." Even at the airport if you go to a kiosk to check-in with a bag, you won't be able to print out a boarding pass. You'll have to go stand in line.

funkah 15y ago
You can print out the boarding pass, but you need to stand in the line for the bag check ticket, and so they can check your ID before putting the bag on the conveyor belt (or telling you to lug your bag to the conveyor belt around the corner).
namdnay 15y ago
On most systems, reservation is more or less independent from departure control. The kiosks might be plugged straight into the DCS, in which case there would be no issue son that side.
nlh 15y ago
The woes stem from Virgin’s switchover from its old reservation system, iFly, to the industry’s 800-pound gorilla, Sabre Ah. I have a hyphenated last name (it's been in my family since the late 1800s), and I was wondering why, on my most recent VA reservation, I was suddenly greeted with the all-too-familiar "Your last name contains invalid characters". Thanks, Sabre. Virgin's old system supported my real name (as most on the web do, as they should). It's remarkable to me that the industry's 800lb gorilla can't support something as simple as a "-" character in a last name, something that's fairly common (or at least not unheard of).

ibejoeb 15y ago

Thanks for the link; I was wondering what the hell has been going on. I've been flying Virgina for a while, and I flew on the weekend of the transition. I think I slipped my online ticket sale in just in time, but check-in, seat selection, frequent flyer info: all screwed. I'm sure this is a tough job, but I wish they would have disabled the features rather than let them fail. The first several times I tried and failed, I assumed it was a fluke. That these thing sometimes worked and sometimes didn't led me to believe that these were ephemeral problems, and I must have wasted hours retrying things. If that had been the case, I would have just arrived early and done it the old-fashioned. Ultimately, that's what I did, although their internal systems are just as screwed (and I don't think hand-written passenger manifests cut the muster anymore). Anyway, on my second pass through this mess, I just skipped the website entirely and did everything at the airport. Took a little longer, but it was pretty stress-free. I'd recommend doing so if you're flying with them soon. To their credit, they did give us miles and candy, and the staff is still the best in the biz. And, if you're flying through SFO, you can make the best of your residual time with a solid burger and a decent drink at Lark Creek Grill.

bengl3rt 15y ago

It seems to me that Sabre et al are all overly complex, left over from the days when you had to physically go to a travel agent's office to watch them clatter incantations into a monochrome terminal as they manually assembled your itinerary... The amount of complexity and arbitrage that goes on in selling plane tickets today is just nuts. I don't like actually flying Southwest, but I think they have the right idea with regards to booking - always, and only, on their website. Ensures a consistent, branded experience and saves them the headache of listing in a GDS.

namdnay 15y ago
For "regional" airlines such as SouthWest or Ryanair, managing your own system is fine. But if you want to be able to do code-sharing, ground-handling and all the rest of the madness that comes with being a global airline, you don't have a choice. The single website solution also means you're cutting yourself off from people who are using fare search services. And let's not forget that in some regions (Asia notably), travel agents with their "cryptic" terminals and "valuable" tickets are still the norm.

zbruhnke 15y ago

Been back and forth to DFW and SFO about 6 times in the last few weeks all on virgin ... for those flights I have probably spent a combined 5 hours on hold for one small change or a misbooked flight etc. but their $139 first class upgrades and friendly staff almost always make it all better for me. I'll be glad when they get this all worked out thats for sure

gujk 15y ago

Why would Virgin migrate to SABRE??