4 ms·

What an irony that the Deutsche Bahn caused an outage of Deutsche Lufthansa. And Deutsche Lufthansa now tries to shift domestic flights to Deutsche Bahn[0] (see

What an irony that the Deutsche Bahn caused an outage of Deutsche Lufthansa. And Deutsche Lufthansa now tries to shift domestic flights to Deutsche Bahn[0] (see also for general advice/help). And who got recently a Star Alliance member? Deutsche Bahn. And who has experience with interrupted cut-off fibers[1]? Deutsche Bahn! *Please* All critical system shall be working autonomous. Cache flight/passenger data locally at airport for next weeks. Servers are for synchronizing, not for keeping your data away. With a local cache you can keep going for some weeks, adding/changing bookings will be harder but you can remain in the air. That is pretty complicated and requires more work but is important. Git or IMAP are examples how it should be. Outages will happen and will keep happening and when the get more seldom, the will become more serious because it will hit inexperienced staff. Especially British Airways is known for issues[2][3]. [0] https://www.lufthansa.com/xx/en/flight-information.html https://www.lufthansa.com/xx/en/flight-information.html [1] https://www.heise.de/news/Sabotage-bei-der-Bahn-Viele-vertrauliche-Infos-sind-offen-zugaenglich-7307277.html https://www.heise.de/news/Sabotage-bei-der-Bahn-Viele-vertra... [2] https://thepointsguy.co.uk/2017/09/amadeus-network-issue-causes-delays/ https://thepointsguy.co.uk/2017/09/amadeus-network-issue-cau... [3] https://www.datacenterdynamics.com/en/news/british-airways-experiences-major-outage-says-it-is-not-a-cyberattack/ https://www.datacenterdynamics.com/en/news/british-airways-e... PS: Deutsche Lufthansa is known for reliable transport. Deutsche Bahn is known for unreliable transport.

zrail 4y ago
Cacheing at the edge might help in certain situations but a global reservation and check-in system seems like the kind of thing that needs to be centralized. I think a more workable solution would have been to treat transit for this data center how most other data centers are organized, with multiple independent uplinks that take different routes into the building. A last-resort point-to-point wireless link would probably be wise as well.
- deleted 4y ago
  [deleted]
- ho_schi 4y ago
  Most of the IT-Plans focus on mitigating the outage itself (e.g. multiple uplinks, backup server, fallback datacenter). I argue that critical system shall gracefully handling the outage. If a client caches the flight/passenger data (lets say two weeks) you can print list. Grab a pen and keep going. That is enough for some hours and even days. If you're going advanced you can allow to add (right at the gate/counter) one person to more passengers and so on. You can also go all-in and add code which allows merging of external data i.e. allowing headquarters and airport personnel working at same time. Die Fluggäste wurden auch nicht per Strichliste in die bereitstehenden Maschinen gelassen, weil nach Angaben des Personals wichtige Informationen zum Abflug fehlten. If it works that is an appropriate solution. PS: We cannot see circumstances. Maybe a hack, a burning datacenter, war, co-worker going crazy, international sanctions, storm, flooding, whatever. For example Deutsche Bahn itself suffered recently a sabotage, two entirely independent glass fibers get cut off at same time (one in west-germany, one in east-germany).
  - ta1243 4y ago
    A large number of flights are booked with less than two weeks notice, and that's before changes which occur hours or even minutes before flights.
    - rtkwe 4y ago
      Not to mention planes are intentionally overbooked (at least here in the US I presume the same tactic exists to a lesser extent in places like Europe with presumably better consumer protections) to ensure they're flying as close to full as possible so you're constantly needing to access the larger booking information to rebook flights for people bumped or to see who's been moved on to your flight. Two weeks is way too long, a day is probably too long to have a fresh passenger manifest.
ho_schi 4y ago
The meme: https://pbs.twimg.com/media/FpAJFXoWAAEnjq7?format=png&name=small https://pbs.twimg.com/media/FpAJFXoWAAEnjq7?format=png&name=... German source: https://www.faz.net/agenturmeldungen/dpa/it-probleme-bei-lufthansa-flugausfaelle-und-verspaetungen-18679862.html https://www.faz.net/agenturmeldungen/dpa/it-probleme-bei-luf... They cut off four cables at once. It didn't failed immediately.
- ta1243 4y ago
  " four fiber optic cables were severed by an excavator on Tuesday." An excavator. One. That's not diversity. Diversity would require cables cuts at two separate sites.
  - est31 4y ago
    This tweet [0] points out that the cable break happened at tuesday afternoon, but the lufthansa problems only occured wednesday morning. So it's more complicated than just one breakage, something else failed too on top of that. [0]: https://twitter.com/deutschetelekom/status/1625865666111709186 https://twitter.com/deutschetelekom/status/16258656661117091...
PurpleRamen 4y ago
> What an irony that the Deutsche Bahn caused an outage of Deutsche Lufthansa. Technically it wasn't the Deutsche Bahn, but a hired construction company. And they just happen to cut some cable from Deutsche Telekom, which also broke the connection from Lufthansa and some others. Those things happen with constructions. But generally, they all are kinda at fault. > All critical system shall be working autonomous. It's not always obvious what is a critical system. I mean it was a semi-important sub-system of a bigger company, not something where you would assume it could cause a national impact.
- bogomipz 4y ago
  >"It's not always obvious what is a critical system. I mean it was a semi-important sub-system of a bigger company, not something where you would assume it could cause a national impact." Why would you not assume it was something that could cause national impact when cutting through fiber cables was the very thing that impacted rail travel for Deutsche Bahn 4 months ago in Northern Germany?[1] DB uses GSM-R(Global System for Mobile Communications Railway)which has fiber backhaul at regular intervals along the tracks.[2] Given the incident in October is still fresh in memory surely there should have been a heightened awareness to this. [1] https://archive.is/oOPcp https://archive.is/oOPcp [2] https://www.globalrailwayreview.com/article/1088/current-gsm-r-situation-in-germany/ https://www.globalrailwayreview.com/article/1088/current-gsm...
  - PurpleRamen 4y ago
    You really think 4 Months is enough time to check all your system for unlikely problems outside your control, redesign them, implement and execute it, and even convince your boss to waste all the money for something which in the worst case will cost most likely less than a solution for preventing it? All assuming that it's even possible to have a solution for every potential problem.
    - bogomipz 4y ago
      You don't need to check all your systems just the area where you are deploying bulldozers and backhoes into. Further enumerating all base stations and their geo coordinates in a GSM network is trivial.
ho_schi 4y ago
Update Source: https://www.sueddeutsche.de/wirtschaft/computer-warum-der-schutz-der-digitalen-infrastruktur-kompliziert-ist-dpa.urn-newsml-dpa-com-20090101-230216-99-618147 https://www.sueddeutsche.de/wirtschaft/computer-warum-der-sc... Article says that Lufthansa actually has a backup lane and it initially worked. It finally failed later [sic!] on Wednesday, maybe because the load was too much for the backup lane. So they tried - as usual - to lower the chance of an outage. But it is crucial to actually being able to keep working with an outage. PS: The construction workers seem to have drilled through four cables (each ~ 900 fibres) in 4-5 meter deep below the surface.

zrail 4y ago

Cacheing at the edge might help in certain situations but a global reservation and check-in system seems like the kind of thing that needs to be centralized. I think a more workable solution would have been to treat transit for this data center how most other data centers are organized, with multiple independent uplinks that take different routes into the building. A last-resort point-to-point wireless link would probably be wise as well.

deleted 4y ago
[deleted]
ho_schi 4y ago
Most of the IT-Plans focus on mitigating the outage itself (e.g. multiple uplinks, backup server, fallback datacenter). I argue that critical system shall gracefully handling the outage. If a client caches the flight/passenger data (lets say two weeks) you can print list. Grab a pen and keep going. That is enough for some hours and even days. If you're going advanced you can allow to add (right at the gate/counter) one person to more passengers and so on. You can also go all-in and add code which allows merging of external data i.e. allowing headquarters and airport personnel working at same time. Die Fluggäste wurden auch nicht per Strichliste in die bereitstehenden Maschinen gelassen, weil nach Angaben des Personals wichtige Informationen zum Abflug fehlten. If it works that is an appropriate solution. PS: We cannot see circumstances. Maybe a hack, a burning datacenter, war, co-worker going crazy, international sanctions, storm, flooding, whatever. For example Deutsche Bahn itself suffered recently a sabotage, two entirely independent glass fibers get cut off at same time (one in west-germany, one in east-germany).
- ta1243 4y ago
  A large number of flights are booked with less than two weeks notice, and that's before changes which occur hours or even minutes before flights.
  - rtkwe 4y ago
    Not to mention planes are intentionally overbooked (at least here in the US I presume the same tactic exists to a lesser extent in places like Europe with presumably better consumer protections) to ensure they're flying as close to full as possible so you're constantly needing to access the larger booking information to rebook flights for people bumped or to see who's been moved on to your flight. Two weeks is way too long, a day is probably too long to have a fresh passenger manifest.

ho_schi 4y ago

The meme: https://pbs.twimg.com/media/FpAJFXoWAAEnjq7?format=png&name=small https://pbs.twimg.com/media/FpAJFXoWAAEnjq7?format=png&name=... German source: https://www.faz.net/agenturmeldungen/dpa/it-probleme-bei-lufthansa-flugausfaelle-und-verspaetungen-18679862.html https://www.faz.net/agenturmeldungen/dpa/it-probleme-bei-luf... They cut off four cables at once. It didn't failed immediately.

ta1243 4y ago
" four fiber optic cables were severed by an excavator on Tuesday." An excavator. One. That's not diversity. Diversity would require cables cuts at two separate sites.
- est31 4y ago
  This tweet [0] points out that the cable break happened at tuesday afternoon, but the lufthansa problems only occured wednesday morning. So it's more complicated than just one breakage, something else failed too on top of that. [0]: https://twitter.com/deutschetelekom/status/1625865666111709186 https://twitter.com/deutschetelekom/status/16258656661117091...

PurpleRamen 4y ago

> What an irony that the Deutsche Bahn caused an outage of Deutsche Lufthansa. Technically it wasn't the Deutsche Bahn, but a hired construction company. And they just happen to cut some cable from Deutsche Telekom, which also broke the connection from Lufthansa and some others. Those things happen with constructions. But generally, they all are kinda at fault. > All critical system shall be working autonomous. It's not always obvious what is a critical system. I mean it was a semi-important sub-system of a bigger company, not something where you would assume it could cause a national impact.

bogomipz 4y ago
>"It's not always obvious what is a critical system. I mean it was a semi-important sub-system of a bigger company, not something where you would assume it could cause a national impact." Why would you not assume it was something that could cause national impact when cutting through fiber cables was the very thing that impacted rail travel for Deutsche Bahn 4 months ago in Northern Germany?[1] DB uses GSM-R(Global System for Mobile Communications Railway)which has fiber backhaul at regular intervals along the tracks.[2] Given the incident in October is still fresh in memory surely there should have been a heightened awareness to this. [1] https://archive.is/oOPcp https://archive.is/oOPcp [2] https://www.globalrailwayreview.com/article/1088/current-gsm-r-situation-in-germany/ https://www.globalrailwayreview.com/article/1088/current-gsm...
- PurpleRamen 4y ago
  You really think 4 Months is enough time to check all your system for unlikely problems outside your control, redesign them, implement and execute it, and even convince your boss to waste all the money for something which in the worst case will cost most likely less than a solution for preventing it? All assuming that it's even possible to have a solution for every potential problem.
  - bogomipz 4y ago
    You don't need to check all your systems just the area where you are deploying bulldozers and backhoes into. Further enumerating all base stations and their geo coordinates in a GSM network is trivial.

ho_schi 4y ago

Update Source: https://www.sueddeutsche.de/wirtschaft/computer-warum-der-schutz-der-digitalen-infrastruktur-kompliziert-ist-dpa.urn-newsml-dpa-com-20090101-230216-99-618147 https://www.sueddeutsche.de/wirtschaft/computer-warum-der-sc... Article says that Lufthansa actually has a backup lane and it initially worked. It finally failed later [sic!] on Wednesday, maybe because the load was too much for the backup lane. So they tried - as usual - to lower the chance of an outage. But it is crucial to actually being able to keep working with an outage. PS: The construction workers seem to have drilled through four cables (each ~ 900 fibres) in 4-5 meter deep below the surface.