10 ms·

I struggle a little to see how there is a single point of failure here (a fibre optic cable). I’d have thought the screens were powered by a device plugged into

I struggle a little to see how there is a single point of failure here (a fibre optic cable). I’d have thought the screens were powered by a device plugged into a LAN. Would be interested if anyone knows the architecture of such screens and why there is such a single point of failure in the whole airport.

Someone1234 8y ago
These screens build information from multiple sources inc. airlines, ground control, and in some cases airport management. They're typically run off-site at a data center since that provides high resiliency and some data is sourced from outside the airport anyway. If the airport has a dedicated fiber channel between their data center and airport itself and it was damaged, this could be the result. The question I'd ask is why they have no fallback strategies (e.g. cellular, VPN over a less secure internet channel, etc). This might be an area where they can explore graceful degradation, meaning that gate information would continue to display (from airport ground control), but some flight times might be missing.
- MBCook 8y ago
  It seems so strange that they wouldn’t have a redundant link (like the various options you suggested) or at least fall back to easily available public data (flight information sites) that would at least help some.
  - gaius 8y ago
    Perhaps they could have gone with a connectivity vendor with an extensive 4G network for backup if the (inexplicably) non-redundant fixed line was cut.
    - Someone1234 8y ago
      They're with Vodafone who has that.
      - gaius 8y ago
        Indeed. How embarrassing!
      - ljm 8y ago
        The problem is that it's the UK and mobile coverage isn't exactly world class here. You can get more consistent and reliable coverage towards the Russian border than you can around London.
    - heartbreak 8y ago
      I used to work for a major digital signage company and we had a huge contract with Verizon for cellular fallback at every site.
  - lakisy 8y ago
    Some years ago, we lost access to our data center although it had triple network redundancy. What happened was that three cables were leaving the DC leased by three different companies. But a mile down the road, all three different cable entered the same physical pipe. When the physical pipe was severed, all three network cables went dark. We should wait until they announced details of what happened before we decide that we know what happened and offer solution ("If they only had redundant links")
    - lgeorget 8y ago
      If my memory serves, one requirement of high-tier datacenters is that they have redundant electric lines leaving the building from opposite sides and in opposite directions. I guess it must be the same for optic fibers. I fully agree with you on your last sentence. One should wait for a bit more than a vague declaration reported in a newspaper before giving unsolicited advice.
      - hermitdev 8y ago
        Yes, they leave different ends/sides of the building, but you don't have much control on where they have utility right of way and whether or not at some point all of your lines share the same right of way (which is likely).
      - mmt 8y ago
        > high-tier datacenters How many people (even tech ops profesionals), these days, know how to tell the difference? Of those that do, how many don't just keep their mouth shut because their bosses don't want to hear it? These details are just trusted to "the cloud", instead. Of course, the airport itself isn't a datacenter of any tier, and that "last mile" can be the biggest redundancy challenge, anyway.
    - fulafel 8y ago
      I have heard this story before, with a twist that initially it was checked for, but the telcos changed it back some years down the road. I wonder what's a good way to audit this.
  - rleigh 8y ago
    Or even the option of inputting the data manually on a local system, so they could use the big displays rather than some whiteboards!
    - mbreese 8y ago
      As someone above already mentioned — the whiteboards themselves also convey the message of “something is wrong, you may want to verify this info” very well. The medium is the message, etc...
  - Moggie100 8y ago
    While I agree that it's odd that there was no fallback, the whiteboards do immediately convey 'stuff is broken, you should check this board' pretty decently, as they would stand out like a sore thumb in the airport normally. If there was a digital fallback, it would absolutely have to have something to tell commuters that the information might be stale (as someone frantically types up flight updates in a back room somewhere)
- JumpCrisscross 8y ago
  > why they have no fallback strategies They do. It's the whiteboards. Whiteboards are cheap, resilient and robust. Exactly what one wants for a backup.
  - tomc1985 8y ago
    Well you also need to have people run around and update them.
    - bhhaskin 8y ago
      Whats wrong with that? It seems to have worked. Sure not ideal, but better than canceling flights and shutting down the airport.
      - antidesitter 8y ago
        Obviously it’s better than shutting down the airport. But is it better than the alternatives Someone1234 proposed? That was the question.
        williamdclt 8y ago
        I don't have the data of course but it does seem better. Those alternative are pretty expensive to build and maintain, have to be tested regularly and are prone to failure. For a situation that has a very low probability of happening, whiteboards do seem more reasonable, that's a very lean approach!
        Dylan16807 8y ago
        The chance of a single fiber line getting cut is not "very low", it's something that a big entity should expect to happen. Much like a power outage; I'm sure they have very good plans for that case. An extra modem is cheap as dirt.
      - tomc1985 8y ago
- ams6110 8y ago
  The data on the screens themselves could be represented by nothing more than a simple CSV file. A dial-up modem could handle the bandwidth to keep that up to date.
- walshemj 8y ago
  Or maybe the backup was in the same conduit and they both suffered backhoe fade - diverse routing is your friend.
7952 8y ago
According to this they have just finished rebuilding their network... https://www.computerweekly.com/news/252441044/Gatwick-Airport-completes-15m-network-upgrade-to-support-expansion-plans https://www.computerweekly.com/news/252441044/Gatwick-Airpor...
- isostatic 8y ago
  Oh I love it [Gatwick has launched an app] for internal staff at the airport, such as shop assistants, baggage handlers and train station employees. ... He says the need for the app stems from some passengers entering “panic mode” and expecting to get flight information from anyone who appears to work at the airport. They may approach an employee in a non-passenger-facing department who has no more information than they do. ... “Passengers sometimes get more information from Twitter and Facebook feeds than from staff on the ground,” says Chacko. “This is an attempt to bridge that communication gap and make everyone look smarter and more up to date.” I wonder how well it worked today https://www.computerweekly.com/news/4500269843/Gatwicks-Airport-Community-App-using-technology-to-make-life-simpler https://www.computerweekly.com/news/4500269843/Gatwicks-Airp...
duxup 8y ago
As someone who used to work in networking. Single points of failure are a way of life...
londons_explore 8y ago
As is common in cases like this, there probably was a redundant link, but the link and its failover procedure probably wasn't recently tested.
outworlder 8y ago
Instead of redundant links, couldnt they have an option to update data manually?

Someone1234 8y ago

These screens build information from multiple sources inc. airlines, ground control, and in some cases airport management. They're typically run off-site at a data center since that provides high resiliency and some data is sourced from outside the airport anyway. If the airport has a dedicated fiber channel between their data center and airport itself and it was damaged, this could be the result. The question I'd ask is why they have no fallback strategies (e.g. cellular, VPN over a less secure internet channel, etc). This might be an area where they can explore graceful degradation, meaning that gate information would continue to display (from airport ground control), but some flight times might be missing.

MBCook 8y ago
It seems so strange that they wouldn’t have a redundant link (like the various options you suggested) or at least fall back to easily available public data (flight information sites) that would at least help some.
- gaius 8y ago
  Perhaps they could have gone with a connectivity vendor with an extensive 4G network for backup if the (inexplicably) non-redundant fixed line was cut.
  - Someone1234 8y ago
    They're with Vodafone who has that.
    - gaius 8y ago
      Indeed. How embarrassing!
    - ljm 8y ago
      The problem is that it's the UK and mobile coverage isn't exactly world class here. You can get more consistent and reliable coverage towards the Russian border than you can around London.
  - heartbreak 8y ago
    I used to work for a major digital signage company and we had a huge contract with Verizon for cellular fallback at every site.
- lakisy 8y ago
  Some years ago, we lost access to our data center although it had triple network redundancy. What happened was that three cables were leaving the DC leased by three different companies. But a mile down the road, all three different cable entered the same physical pipe. When the physical pipe was severed, all three network cables went dark. We should wait until they announced details of what happened before we decide that we know what happened and offer solution ("If they only had redundant links")
  - lgeorget 8y ago
    If my memory serves, one requirement of high-tier datacenters is that they have redundant electric lines leaving the building from opposite sides and in opposite directions. I guess it must be the same for optic fibers. I fully agree with you on your last sentence. One should wait for a bit more than a vague declaration reported in a newspaper before giving unsolicited advice.
    - hermitdev 8y ago
      Yes, they leave different ends/sides of the building, but you don't have much control on where they have utility right of way and whether or not at some point all of your lines share the same right of way (which is likely).
    - mmt 8y ago
      > high-tier datacenters How many people (even tech ops profesionals), these days, know how to tell the difference? Of those that do, how many don't just keep their mouth shut because their bosses don't want to hear it? These details are just trusted to "the cloud", instead. Of course, the airport itself isn't a datacenter of any tier, and that "last mile" can be the biggest redundancy challenge, anyway.
  - fulafel 8y ago
    I have heard this story before, with a twist that initially it was checked for, but the telcos changed it back some years down the road. I wonder what's a good way to audit this.
- rleigh 8y ago
  Or even the option of inputting the data manually on a local system, so they could use the big displays rather than some whiteboards!
  - mbreese 8y ago
    As someone above already mentioned — the whiteboards themselves also convey the message of “something is wrong, you may want to verify this info” very well. The medium is the message, etc...
- Moggie100 8y ago
  While I agree that it's odd that there was no fallback, the whiteboards do immediately convey 'stuff is broken, you should check this board' pretty decently, as they would stand out like a sore thumb in the airport normally. If there was a digital fallback, it would absolutely have to have something to tell commuters that the information might be stale (as someone frantically types up flight updates in a back room somewhere)
JumpCrisscross 8y ago
> why they have no fallback strategies They do. It's the whiteboards. Whiteboards are cheap, resilient and robust. Exactly what one wants for a backup.
- tomc1985 8y ago
  Well you also need to have people run around and update them.
  - bhhaskin 8y ago
    Whats wrong with that? It seems to have worked. Sure not ideal, but better than canceling flights and shutting down the airport.
    - antidesitter 8y ago
      Obviously it’s better than shutting down the airport. But is it better than the alternatives Someone1234 proposed? That was the question.
      - williamdclt 8y ago
        I don't have the data of course but it does seem better. Those alternative are pretty expensive to build and maintain, have to be tested regularly and are prone to failure. For a situation that has a very low probability of happening, whiteboards do seem more reasonable, that's a very lean approach!
        Dylan16807 8y ago
        The chance of a single fiber line getting cut is not "very low", it's something that a big entity should expect to happen. Much like a power outage; I'm sure they have very good plans for that case. An extra modem is cheap as dirt.
    - tomc1985 8y ago
ams6110 8y ago
The data on the screens themselves could be represented by nothing more than a simple CSV file. A dial-up modem could handle the bandwidth to keep that up to date.
walshemj 8y ago
Or maybe the backup was in the same conduit and they both suffered backhoe fade - diverse routing is your friend.

7952 8y ago

According to this they have just finished rebuilding their network... https://www.computerweekly.com/news/252441044/Gatwick-Airport-completes-15m-network-upgrade-to-support-expansion-plans https://www.computerweekly.com/news/252441044/Gatwick-Airpor...

isostatic 8y ago
Oh I love it [Gatwick has launched an app] for internal staff at the airport, such as shop assistants, baggage handlers and train station employees. ... He says the need for the app stems from some passengers entering “panic mode” and expecting to get flight information from anyone who appears to work at the airport. They may approach an employee in a non-passenger-facing department who has no more information than they do. ... “Passengers sometimes get more information from Twitter and Facebook feeds than from staff on the ground,” says Chacko. “This is an attempt to bridge that communication gap and make everyone look smarter and more up to date.” I wonder how well it worked today https://www.computerweekly.com/news/4500269843/Gatwicks-Airport-Community-App-using-technology-to-make-life-simpler https://www.computerweekly.com/news/4500269843/Gatwicks-Airp...

duxup 8y ago

As someone who used to work in networking. Single points of failure are a way of life...

londons_explore 8y ago

As is common in cases like this, there probably was a redundant link, but the link and its failover procedure probably wasn't recently tested.

outworlder 8y ago

Instead of redundant links, couldnt they have an option to update data manually?