4 ms·

Seems nuts. Spreading locations would allow more and better monitoring to be sure locations weren't compromised, and the ability to drop a location if it was.

Seems nuts. Spreading locations would allow more and better monitoring to be sure locations weren't compromised, and the ability to drop a location if it was. Extra layers of protection aren't merely "adding to the attack surface" (although that's possible, antivirus products becoming the vector, for example.)

tw04 10y ago
It would also, at minimum, double their costs. They're in a cutthroat industry - doubling costs would likely mean going out of business.
- koluft 10y ago
  Why assume computer costs is substantial to overall costs of maintaining and flying airplanes?
  - ubernostrum 10y ago
    Why do people assume that this stuff works like a tech job? The kind of national and global operations coordination Delta does from their center in Atlanta is not a "oh, network's down, work from the coffee shop" kind of job. Delta + regional affiliates operate nearly 8,000 flights per day on six continents through thirteen hubs, involving around 80,000 employees and ~820 mainline aircraft plus I don't even know how many smaller regional jets, plus codesharing on SkyTeam alliance partners and 13 other non-alliance-partner carriers. Spec out an ops center capable of ingesting and organizing all the data on that, presenting it for human use, and then communicating instructions back out to all those people, planes, airports, maintenance bases, etc. And then ask yourself if that's something you just build a couple extra copies of (along with on-call staff to come in and pick up the whole thing at a moment's notice, since if the main one goes down it's not like you'll be flying the people to the backup locations). The answer, of course, is that it's not something you build extras of. You build it once, and build it as reliable as you can, because building spares just is not feasible; the only people who maintain extras of this kind of infrastructure are governments who worry about getting into nuclear wars.
    - maccam94 10y ago
      Uber has way more QPS and route planning complexity, and they still have multiple datacenters for redundancy.
      - ubernostrum 10y ago
        * Uber's drivers can operate largely independently of each other. A local issue does not cascade to affect larger areas. An airline's sub-components largely cannot operate independently; issues on a local level can easily cascade upward to become regional issues, and then regional issues can cascade upward to become national or even global ones, as passengers and deadheading crew members misconnect, late flights become later en route to following destinations, etc. * Uber does not submit route plans to any authority, or deal with issues of transit across controlled borders. Airlines must submit valid flight plans for approval to aviation authorities in the appropriate countries, and routinely must handle transit of aircraft, people and cargo across controlled international borders, planning all aspects in advance. * Uber simply makes use of any available roads and pickup/drop-off zones. Airlines must pay for and sometimes obtain regulatory approval for takeoff and landing slots, use of gates at airports, use of facilities at airports, etc., keeping track of what resources are available and which will be in use at any given time. * Uber famously tells regulators to, effectively, go fuck themselves. Airlines which attempt this are shut down. Immense effort must be dedicated to compliance with applicable safety and commerce regulations at all times. * Uber does not own or maintain the cars its passengers use. Airlines own or lease their aircraft and are responsible for their maintenance, on legally-mandated schedules. Airliner maintenance is a wonder in and of itself. And on and on and on it goes. Again: this ain't a Silicon Valley tech startup. Thinking in terms of even the largest SV startups is going to lead you badly astray.
      - niftich 10y ago
        Uber didn't exist before 2009. They got a bunch of people in a room and wrote some code, and now they're successful. Greenfield development at it's best. Delta started off flying planes, and has been around longer than computers existed. Like many established businesses, they have a mix of new and legacy technologies, and can't just copy everything up to AWS. They'd have to dedicate years of time and effort over many employees to rewrite some of their systems, while still maintaining their existing ones in parallel until it's safe to migrate off. All the while they have an actual business to run, with real revenues, real expenditures, and IT is just (an important, but costly) piece in the big picture. These companies are neither incompetent, nor malicious. They just have to find the money and time to get done the enhancements they'd like to their systems, and change doesn't happen overnight. Likely, with this awkward generator fire, they'll try to hasten their efforts.
- ams6110 10y ago
  Being unable to fly their airplanes will likely mean going out of business as well.
  - michael_fine 10y ago
    It certainly didn't mean that this time.
fennecfoxen 10y ago
> Spreading locations would allow more and better monitoring to be sure locations weren't compromised, and the ability to drop a location if it was. If Delta is the mainframe-oriented business that I suspect it is, "spreading locations" means turning their software stack from the monolith of monoliths into a distributed system. It's a damn fine idea, and likely to take them decades of effort across dozens of teams with hilarious expensive bugs along the way.

tw04 10y ago

It would also, at minimum, double their costs. They're in a cutthroat industry - doubling costs would likely mean going out of business.

koluft 10y ago
Why assume computer costs is substantial to overall costs of maintaining and flying airplanes?
- ubernostrum 10y ago
  Why do people assume that this stuff works like a tech job? The kind of national and global operations coordination Delta does from their center in Atlanta is not a "oh, network's down, work from the coffee shop" kind of job. Delta + regional affiliates operate nearly 8,000 flights per day on six continents through thirteen hubs, involving around 80,000 employees and ~820 mainline aircraft plus I don't even know how many smaller regional jets, plus codesharing on SkyTeam alliance partners and 13 other non-alliance-partner carriers. Spec out an ops center capable of ingesting and organizing all the data on that, presenting it for human use, and then communicating instructions back out to all those people, planes, airports, maintenance bases, etc. And then ask yourself if that's something you just build a couple extra copies of (along with on-call staff to come in and pick up the whole thing at a moment's notice, since if the main one goes down it's not like you'll be flying the people to the backup locations). The answer, of course, is that it's not something you build extras of. You build it once, and build it as reliable as you can, because building spares just is not feasible; the only people who maintain extras of this kind of infrastructure are governments who worry about getting into nuclear wars.
  - maccam94 10y ago
    Uber has way more QPS and route planning complexity, and they still have multiple datacenters for redundancy.
    - ubernostrum 10y ago
      * Uber's drivers can operate largely independently of each other. A local issue does not cascade to affect larger areas. An airline's sub-components largely cannot operate independently; issues on a local level can easily cascade upward to become regional issues, and then regional issues can cascade upward to become national or even global ones, as passengers and deadheading crew members misconnect, late flights become later en route to following destinations, etc. * Uber does not submit route plans to any authority, or deal with issues of transit across controlled borders. Airlines must submit valid flight plans for approval to aviation authorities in the appropriate countries, and routinely must handle transit of aircraft, people and cargo across controlled international borders, planning all aspects in advance. * Uber simply makes use of any available roads and pickup/drop-off zones. Airlines must pay for and sometimes obtain regulatory approval for takeoff and landing slots, use of gates at airports, use of facilities at airports, etc., keeping track of what resources are available and which will be in use at any given time. * Uber famously tells regulators to, effectively, go fuck themselves. Airlines which attempt this are shut down. Immense effort must be dedicated to compliance with applicable safety and commerce regulations at all times. * Uber does not own or maintain the cars its passengers use. Airlines own or lease their aircraft and are responsible for their maintenance, on legally-mandated schedules. Airliner maintenance is a wonder in and of itself. And on and on and on it goes. Again: this ain't a Silicon Valley tech startup. Thinking in terms of even the largest SV startups is going to lead you badly astray.
    - niftich 10y ago
      Uber didn't exist before 2009. They got a bunch of people in a room and wrote some code, and now they're successful. Greenfield development at it's best. Delta started off flying planes, and has been around longer than computers existed. Like many established businesses, they have a mix of new and legacy technologies, and can't just copy everything up to AWS. They'd have to dedicate years of time and effort over many employees to rewrite some of their systems, while still maintaining their existing ones in parallel until it's safe to migrate off. All the while they have an actual business to run, with real revenues, real expenditures, and IT is just (an important, but costly) piece in the big picture. These companies are neither incompetent, nor malicious. They just have to find the money and time to get done the enhancements they'd like to their systems, and change doesn't happen overnight. Likely, with this awkward generator fire, they'll try to hasten their efforts.
ams6110 10y ago
Being unable to fly their airplanes will likely mean going out of business as well.
- michael_fine 10y ago
  It certainly didn't mean that this time.

fennecfoxen 10y ago

> Spreading locations would allow more and better monitoring to be sure locations weren't compromised, and the ability to drop a location if it was. If Delta is the mainframe-oriented business that I suspect it is, "spreading locations" means turning their software stack from the monolith of monoliths into a distributed system. It's a damn fine idea, and likely to take them decades of effort across dozens of teams with hilarious expensive bugs along the way.