7 ms·

Frankfurt rail work damaged fibre, causes global Lufthansa check-in system fault

77 points by L_226 4y ago

james_pm 4y ago
Crazy that an airline that is very used to the need for double- and triple-redundancy in the aircraft it flies fails to have any sort of connectivity backup for it's ground systems.
- deleted 4y ago
  [deleted]
- cactusplant7374 4y ago
  Does anyone know if these systems are such high bandwidth that they couldn't run with 4G or 5G temporarily? Perhaps some flights, say 25%, could actually still be processed?
  - oarsinsync 4y ago
    Attaching a 4G modem to the end of an existing enterprise network is a non-trivial feat.
    - KMag 4y ago
      Right. I'm not even sure the interconnect is IP-based. It could very well be a legacy X.25 network. A quick check of Wikipedia says that X.25 is still used in the aviation industry. Routing X.25 over 4G probably involves adding a layer to encapsulate in IP.
      - TeMPOraL 4y ago
        This is Germany. I bet they have at least one company nobody heard of that makes money hand over fist specifically by doing X.25 over IP for industrial plants.
- JaimeThompson 4y ago
  They have the legal obligation to have those double/triple redundant systems in their aircraft. Otherwise it is probable they would not have as many safety systems.
cactusplant7374 4y ago
There was a fiber cut last year near Chicago. It was near a railroad so a bunch of authorizations were required and it took about three days to complete the work. My parents live in a rural area that has fiber but didn't have internet because the local ISP didn't have any redundancy. We switched from microwave antenna which has its own issues to fiber and my dad is thinking, "Well, you said this would be better." You can blame the local ISP, but am I wrong to think that it's hard for a rural ISP providing fiber to afford redundancy?
- thinkindie 4y ago
  this could have happened to the microwave antenna anyway - the other side of your antenna is most probably connected with a fiber cable, which can also be cut.
  - Am4TIfIsER0ppos 4y ago
    Clearly the internet should be wireless all the way to the server which even has wireless power. No outages(!)
  - cactusplant7374 4y ago
    The previous outages with the microwave antenna were weather related. Now the problems are much more interesting. :)
- zrail 4y ago
  A rural ISP might not even have the possibility of redundancy. There might be just one backbone fiber within reasonable distance. For example, there's a local rural fiber ISP near me that had to invest quite a lot into digging a fiber trench several miles to the nearest backbone connection before they even got off the ground.
midasuni 4y ago
“ Lufthansa said construction work on a rail line in Frankfurt was to blame for the massive IT failure,. ” No, Lufthansa not having sufficient resilience was to blame.
- davidkuennen 4y ago
  I'm from Germany and it's really crazy to think just one cable is holding almost the entirety of our flight system. Hopefully they fix it until next Monday. I'm having a flight to Frankfurt then.
  - jessekv 4y ago
    > fix it until next Monday Off topic, sorry. I'm curious why Germans often use this phrasing. Is there a "false friend" [1] in English? (I assume you mean 'before', not 'until') 1. https://en.m.wikipedia.org/wiki/False_friend https://en.m.wikipedia.org/wiki/False_friend
    - toast0 4y ago
      Before and until are sometimes interchangable in English. Don't go there before Monday. Don't go there until Monday. You can get this deal until Monday. You can get this deal before Monday. (Some lack of clarity about whether the offer is valid on Monday with until though) But I agree, it doesn't work in this context. English is weird.
    - cjpearson 4y ago
      In English one would use 'by' for a deadline and 'until' when the state or action continues up to a point. e.g. "I must finish the task by the end of the day" or "I'll be working on the task until the end of the day". In German 'bis' is used for both cases.
    - trunc8 4y ago
      To continue the tangent, I'm currently learning Spanish and appreciate the really long tail involved in language learning. I often work with people who, while they do not speak English as their first language, speak it pretty much perfectly. Except for maybe a few minor "glitches", which would just never occur with a native speaker. I often wonder how pointing these out would be received, and though I presume many would be appreciative, maybe not all...
ho_schi 4y ago
What an irony that the Deutsche Bahn caused an outage of Deutsche Lufthansa. And Deutsche Lufthansa now tries to shift domestic flights to Deutsche Bahn[0] (see also for general advice/help). And who got recently a Star Alliance member? Deutsche Bahn. And who has experience with interrupted cut-off fibers[1]? Deutsche Bahn! *Please* All critical system shall be working autonomous. Cache flight/passenger data locally at airport for next weeks. Servers are for synchronizing, not for keeping your data away. With a local cache you can keep going for some weeks, adding/changing bookings will be harder but you can remain in the air. That is pretty complicated and requires more work but is important. Git or IMAP are examples how it should be. Outages will happen and will keep happening and when the get more seldom, the will become more serious because it will hit inexperienced staff. Especially British Airways is known for issues[2][3]. [0] https://www.lufthansa.com/xx/en/flight-information.html https://www.lufthansa.com/xx/en/flight-information.html [1] https://www.heise.de/news/Sabotage-bei-der-Bahn-Viele-vertrauliche-Infos-sind-offen-zugaenglich-7307277.html https://www.heise.de/news/Sabotage-bei-der-Bahn-Viele-vertra... [2] https://thepointsguy.co.uk/2017/09/amadeus-network-issue-causes-delays/ https://thepointsguy.co.uk/2017/09/amadeus-network-issue-cau... [3] https://www.datacenterdynamics.com/en/news/british-airways-experiences-major-outage-says-it-is-not-a-cyberattack/ https://www.datacenterdynamics.com/en/news/british-airways-e... PS: Deutsche Lufthansa is known for reliable transport. Deutsche Bahn is known for unreliable transport.
- zrail 4y ago
  Cacheing at the edge might help in certain situations but a global reservation and check-in system seems like the kind of thing that needs to be centralized. I think a more workable solution would have been to treat transit for this data center how most other data centers are organized, with multiple independent uplinks that take different routes into the building. A last-resort point-to-point wireless link would probably be wise as well.
  - deleted 4y ago
    [deleted]
  - ho_schi 4y ago
    Most of the IT-Plans focus on mitigating the outage itself (e.g. multiple uplinks, backup server, fallback datacenter). I argue that critical system shall gracefully handling the outage. If a client caches the flight/passenger data (lets say two weeks) you can print list. Grab a pen and keep going. That is enough for some hours and even days. If you're going advanced you can allow to add (right at the gate/counter) one person to more passengers and so on. You can also go all-in and add code which allows merging of external data i.e. allowing headquarters and airport personnel working at same time. Die Fluggäste wurden auch nicht per Strichliste in die bereitstehenden Maschinen gelassen, weil nach Angaben des Personals wichtige Informationen zum Abflug fehlten. If it works that is an appropriate solution. PS: We cannot see circumstances. Maybe a hack, a burning datacenter, war, co-worker going crazy, international sanctions, storm, flooding, whatever. For example Deutsche Bahn itself suffered recently a sabotage, two entirely independent glass fibers get cut off at same time (one in west-germany, one in east-germany).
atonse 4y ago
How is this possible? Presumably whichever datacenter it was hosted in, would have multiple fiber lines connecting to it? Or am I just spoiled by the major cloud providers?
- kleiba 4y ago
  Obviously you're not familiar with how IT is done in Germany.
  - wellanyway 4y ago
    Hehe I'm having flashbacks
- PurpleRamen 4y ago
  It was a major outtake of the whole area, and there is not that much competition in terms of cables for historical reasons. Though, it's also possible that the other connections just failed, or could not take the sudden traffic, or the system crashed for some nonsical reason, because nobody really tested this scenario.
JCM9 4y ago
Everything fails all the time. The fiber cut was not responsible for this. Lufthansa having woefully inadequate resiliency plans in place with their infrastructure is to blame. This is 100% on poor management practices by Lufthansa, but of course they’re going to point the press to that shiny object over there in the form of a fiber cut. The press, as usual, took the bait.
- ta1243 4y ago
  push out the PR and remember that journalists are not technical people. Had there been multiple cuts in multiple different locations I'd be more sympathetic. Shetland being a recent example [0], where one fibre was cut, then the other one in a completely different direction was also cut. Assuming they had 2 cables and both are broken, whether something as critical as the entire LH booking system deserves more resilience than just 2 geographically diverse cables, I'm not sure -- ultimately it's what's the cost, what's the damage, and what's the likelihood of it happening. [0] https://www.bbc.co.uk/news/uk-scotland-north-east-orkney-shetland-63364346 https://www.bbc.co.uk/news/uk-scotland-north-east-orkney-she...
ginko 4y ago
Guess DB isn't content with just delaying their own trains these days..
sokols 4y ago
Interesting, most of the Nord-West area of Frankfurt was affected, residential cable internet failed but LTE/5G network of Deutsche Telekom didn't function as well. edit: related tweet from Deutsche Telekom https://twitter.com/deutschetelekom/status/1625824840924950528 https://twitter.com/deutschetelekom/status/16258248409249505...
- ho_schi 4y ago
  Some people suggested adding {2,3,4,5}G as another fallback. This often fails because the cell tower is connected to the same cables. Or the backbone burns down. Or the connection between the datacenters fails (see recent AWS failures). Or coworker is mad... It is good to prevent failures. But autonomous local systems shall remain (basically) usable for some time. And if the local system is a paper, that's at least something. We've also ABS, ESP and ASR in cars. We still close the seatbelts.
durnygbur 4y ago
Why don't they communicate over DSL like most German households? Maybe 6Mbps but it's a sicher 6Mbps.
- coldcode 4y ago
  I would consider a move to Germany, but not giving up my 500Mbps internet connection.
  - rad_gruchalski 4y ago
    Some places in Germany have that too.
    - durnygbur 4y ago
      Except when you rent and the landlord says "Internet is internet, why do you need different one?" then a next desperate tenant will rent whatever there is.
  - tiluha 4y ago
    Internet Speeds in Germany are lacking in some areas, but fast internet is generally available. I have a symmetrical 1 gig FTTH connection in the countryside
slac 4y ago
This is data sovereignty at work.

77 points by L_226 4y ago

Crazy that an airline that is very used to the need for double- and triple-redundancy in the aircraft it flies fails to have any sort of connectivity backup for it's ground systems.

deleted 4y ago
[deleted]
cactusplant7374 4y ago
Does anyone know if these systems are such high bandwidth that they couldn't run with 4G or 5G temporarily? Perhaps some flights, say 25%, could actually still be processed?
- oarsinsync 4y ago
  Attaching a 4G modem to the end of an existing enterprise network is a non-trivial feat.
  - KMag 4y ago
    Right. I'm not even sure the interconnect is IP-based. It could very well be a legacy X.25 network. A quick check of Wikipedia says that X.25 is still used in the aviation industry. Routing X.25 over 4G probably involves adding a layer to encapsulate in IP.
    - TeMPOraL 4y ago
      This is Germany. I bet they have at least one company nobody heard of that makes money hand over fist specifically by doing X.25 over IP for industrial plants.
JaimeThompson 4y ago
They have the legal obligation to have those double/triple redundant systems in their aircraft. Otherwise it is probable they would not have as many safety systems.

cactusplant7374 4y ago

There was a fiber cut last year near Chicago. It was near a railroad so a bunch of authorizations were required and it took about three days to complete the work. My parents live in a rural area that has fiber but didn't have internet because the local ISP didn't have any redundancy. We switched from microwave antenna which has its own issues to fiber and my dad is thinking, "Well, you said this would be better." You can blame the local ISP, but am I wrong to think that it's hard for a rural ISP providing fiber to afford redundancy?

thinkindie 4y ago
this could have happened to the microwave antenna anyway - the other side of your antenna is most probably connected with a fiber cable, which can also be cut.
- Am4TIfIsER0ppos 4y ago
  Clearly the internet should be wireless all the way to the server which even has wireless power. No outages(!)
- cactusplant7374 4y ago
  The previous outages with the microwave antenna were weather related. Now the problems are much more interesting. :)
zrail 4y ago
A rural ISP might not even have the possibility of redundancy. There might be just one backbone fiber within reasonable distance. For example, there's a local rural fiber ISP near me that had to invest quite a lot into digging a fiber trench several miles to the nearest backbone connection before they even got off the ground.

midasuni 4y ago

“ Lufthansa said construction work on a rail line in Frankfurt was to blame for the massive IT failure,. ” No, Lufthansa not having sufficient resilience was to blame.

davidkuennen 4y ago
I'm from Germany and it's really crazy to think just one cable is holding almost the entirety of our flight system. Hopefully they fix it until next Monday. I'm having a flight to Frankfurt then.
- jessekv 4y ago
  > fix it until next Monday Off topic, sorry. I'm curious why Germans often use this phrasing. Is there a "false friend" [1] in English? (I assume you mean 'before', not 'until') 1. https://en.m.wikipedia.org/wiki/False_friend https://en.m.wikipedia.org/wiki/False_friend
  - toast0 4y ago
    Before and until are sometimes interchangable in English. Don't go there before Monday. Don't go there until Monday. You can get this deal until Monday. You can get this deal before Monday. (Some lack of clarity about whether the offer is valid on Monday with until though) But I agree, it doesn't work in this context. English is weird.
  - cjpearson 4y ago
    In English one would use 'by' for a deadline and 'until' when the state or action continues up to a point. e.g. "I must finish the task by the end of the day" or "I'll be working on the task until the end of the day". In German 'bis' is used for both cases.
  - trunc8 4y ago
    To continue the tangent, I'm currently learning Spanish and appreciate the really long tail involved in language learning. I often work with people who, while they do not speak English as their first language, speak it pretty much perfectly. Except for maybe a few minor "glitches", which would just never occur with a native speaker. I often wonder how pointing these out would be received, and though I presume many would be appreciative, maybe not all...

ho_schi 4y ago

What an irony that the Deutsche Bahn caused an outage of Deutsche Lufthansa. And Deutsche Lufthansa now tries to shift domestic flights to Deutsche Bahn[0] (see also for general advice/help). And who got recently a Star Alliance member? Deutsche Bahn. And who has experience with interrupted cut-off fibers[1]? Deutsche Bahn! *Please* All critical system shall be working autonomous. Cache flight/passenger data locally at airport for next weeks. Servers are for synchronizing, not for keeping your data away. With a local cache you can keep going for some weeks, adding/changing bookings will be harder but you can remain in the air. That is pretty complicated and requires more work but is important. Git or IMAP are examples how it should be. Outages will happen and will keep happening and when the get more seldom, the will become more serious because it will hit inexperienced staff. Especially British Airways is known for issues[2][3]. [0] https://www.lufthansa.com/xx/en/flight-information.html https://www.lufthansa.com/xx/en/flight-information.html [1] https://www.heise.de/news/Sabotage-bei-der-Bahn-Viele-vertrauliche-Infos-sind-offen-zugaenglich-7307277.html https://www.heise.de/news/Sabotage-bei-der-Bahn-Viele-vertra... [2] https://thepointsguy.co.uk/2017/09/amadeus-network-issue-causes-delays/ https://thepointsguy.co.uk/2017/09/amadeus-network-issue-cau... [3] https://www.datacenterdynamics.com/en/news/british-airways-experiences-major-outage-says-it-is-not-a-cyberattack/ https://www.datacenterdynamics.com/en/news/british-airways-e... PS: Deutsche Lufthansa is known for reliable transport. Deutsche Bahn is known for unreliable transport.

zrail 4y ago
Cacheing at the edge might help in certain situations but a global reservation and check-in system seems like the kind of thing that needs to be centralized. I think a more workable solution would have been to treat transit for this data center how most other data centers are organized, with multiple independent uplinks that take different routes into the building. A last-resort point-to-point wireless link would probably be wise as well.
- deleted 4y ago
  [deleted]
- ho_schi 4y ago
  Most of the IT-Plans focus on mitigating the outage itself (e.g. multiple uplinks, backup server, fallback datacenter). I argue that critical system shall gracefully handling the outage. If a client caches the flight/passenger data (lets say two weeks) you can print list. Grab a pen and keep going. That is enough for some hours and even days. If you're going advanced you can allow to add (right at the gate/counter) one person to more passengers and so on. You can also go all-in and add code which allows merging of external data i.e. allowing headquarters and airport personnel working at same time. Die Fluggäste wurden auch nicht per Strichliste in die bereitstehenden Maschinen gelassen, weil nach Angaben des Personals wichtige Informationen zum Abflug fehlten. If it works that is an appropriate solution. PS: We cannot see circumstances. Maybe a hack, a burning datacenter, war, co-worker going crazy, international sanctions, storm, flooding, whatever. For example Deutsche Bahn itself suffered recently a sabotage, two entirely independent glass fibers get cut off at same time (one in west-germany, one in east-germany).

atonse 4y ago

How is this possible? Presumably whichever datacenter it was hosted in, would have multiple fiber lines connecting to it? Or am I just spoiled by the major cloud providers?

kleiba 4y ago
Obviously you're not familiar with how IT is done in Germany.
- wellanyway 4y ago
  Hehe I'm having flashbacks
PurpleRamen 4y ago
It was a major outtake of the whole area, and there is not that much competition in terms of cables for historical reasons. Though, it's also possible that the other connections just failed, or could not take the sudden traffic, or the system crashed for some nonsical reason, because nobody really tested this scenario.

JCM9 4y ago

Everything fails all the time. The fiber cut was not responsible for this. Lufthansa having woefully inadequate resiliency plans in place with their infrastructure is to blame. This is 100% on poor management practices by Lufthansa, but of course they’re going to point the press to that shiny object over there in the form of a fiber cut. The press, as usual, took the bait.

ta1243 4y ago
push out the PR and remember that journalists are not technical people. Had there been multiple cuts in multiple different locations I'd be more sympathetic. Shetland being a recent example [0], where one fibre was cut, then the other one in a completely different direction was also cut. Assuming they had 2 cables and both are broken, whether something as critical as the entire LH booking system deserves more resilience than just 2 geographically diverse cables, I'm not sure -- ultimately it's what's the cost, what's the damage, and what's the likelihood of it happening. [0] https://www.bbc.co.uk/news/uk-scotland-north-east-orkney-shetland-63364346 https://www.bbc.co.uk/news/uk-scotland-north-east-orkney-she...

ginko 4y ago

Guess DB isn't content with just delaying their own trains these days..

sokols 4y ago

Interesting, most of the Nord-West area of Frankfurt was affected, residential cable internet failed but LTE/5G network of Deutsche Telekom didn't function as well. edit: related tweet from Deutsche Telekom https://twitter.com/deutschetelekom/status/1625824840924950528 https://twitter.com/deutschetelekom/status/16258248409249505...

ho_schi 4y ago
Some people suggested adding {2,3,4,5}G as another fallback. This often fails because the cell tower is connected to the same cables. Or the backbone burns down. Or the connection between the datacenters fails (see recent AWS failures). Or coworker is mad... It is good to prevent failures. But autonomous local systems shall remain (basically) usable for some time. And if the local system is a paper, that's at least something. We've also ABS, ESP and ASR in cars. We still close the seatbelts.

durnygbur 4y ago

Why don't they communicate over DSL like most German households? Maybe 6Mbps but it's a sicher 6Mbps.

coldcode 4y ago
I would consider a move to Germany, but not giving up my 500Mbps internet connection.
- rad_gruchalski 4y ago
  Some places in Germany have that too.
  - durnygbur 4y ago
    Except when you rent and the landlord says "Internet is internet, why do you need different one?" then a next desperate tenant will rent whatever there is.
- tiluha 4y ago
  Internet Speeds in Germany are lacking in some areas, but fast internet is generally available. I have a symmetrical 1 gig FTTH connection in the countryside

slac 4y ago

This is data sovereignty at work.