18 ms·

Feds spend billions to run museum-ready computer systems

• paavokoya 10y ago
  To be fair, my last employer (aerospace manuf.) ran an incredibly dated and ancient OS with pretty decent results. It was simple, to the point, ugly as hell but got the job done without needing constant updates etc etc. Also we never had a problem with malware (because who writes malware for a 30 year old OS?) I understand this article mentions many different sectors and functions for antiquated systems but sometimes an update simply isn't needed.
  • themodelplumber 10y ago
    > because who writes malware for a 30 year old OS? Careful. I believe I just read an article (Wired?) about some hackers doing exactly this. Security through Seniority might be a good name for the mindset. :-)
    • nickpsecurity 10y ago
      That's great. I'm going to have to remember that phrase. Another I considered is Security through Obsolesence. Makes it sound more retarded. ;)
    • paavokoya 10y ago
      Is there a term for when you say something knowingly broad but ALSO understand there are specific cases where it doesn't apply? Like a simple term I can paste onto the end of statements like (sic) or similar so that I can avoid the nitpickers? EDIT: honestly not being facetious just curious because this exact thing happens constantly.
      • nickpsecurity 10y ago
        That's how models and rule work in general. There's a rule or pattern that mostly applies but always exceptions. In this case, the old systems were all torn apart by older hackers. There's young ones, just a few, hacking mainframes and stuff now. The ease of finding first flaws showed only reason they weren't found sooner is nobody cared enough to look. Or couldn't afford the relics to play with. So, believing one is safe just because computers are old is akin to Security via Senority. Now, there are older approaches like tagged HW or dedicated lines whose intrinsic properties reduce odds of attacks. Definitely apply any good techniques from past to present situation. Just that it's old... especially since it's older than INFOSEC itself... doesnt make it safer.
      • bglazer 10y ago
        That's an interesting question. I can't think of one off the top of my head. You should try http://english.stackexchange.com/ http://english.stackexchange.com/ Questions like this are pretty common there. As an added bonus, Peter Shor (famous for Shor's algorithm) just might answer your question.
      • jccalhoun 10y ago
        an exaggeration?
      • Snargorf 10y ago
        "(generally)" ? But you're right, it's sad how many people insist on feeling smart by taking a general statement as a universal statement and nitpicking it. It's so tiring.
      • 205guy 10y ago
        The rebuttal was pertinent and polite, and (almost) provided a reference, so I really don't think it can be qualified as nitpicking. Your statement was a bit exaggerated for effect, so you can hardly criticize someone who responds to it, for effect. I suggest you avoid the knowingly broad and include nuance to your arguments. I like the suggestion of adding "generally".
    • devonkim 10y ago
      I remember a story of someone that was decommissioning an HP-UX box at a university after years of service doing... not much it seems. While shutting it down they noticed that the system had been compromised and an investigation began. It turns out the intruder had exploited a remote vulnerability, logged in as a regular user, and had tried to compile a rootkit. After days of attempts to compile the software, the intruder seemed to have found a way to patch the remote vulnerability, and logged out to never return. This is the same thing as security through obscurity or security through lack of usability IMO.
      • vt240 10y ago
        That is pretty funny because decades ago we used to remotely compromise university HP 9000 systems, create new privileged accounts, and apply the latest patch sets to the system to close up any security issues. I would assume that is a pretty common practice, you don't want someone else messing with your system. I would bet the investigation didn't go far enough.
    • antisthenes 10y ago
      Security through unplanned obsolescence.
  • hosh 10y ago
    I remember reading a counterpoint about this a while back -- sometimes for critical systems, the risk of updates is really high. For example, NASA still uses hardened 808x systems. On top of that, for space-based systems in an ionized environment, the risk of having hardware developing hardware faults is non-zero, and the kind of things people do for error correction in that environment is insane. The flip side of this is that, when a technology is widely used, there is a scaling that happens. If you are the sole user of a technology, then part of your cost is maintaining that technology that used to be shared by all the other users of that technology. And then there's the bigger question: how effective is nuclear deterrence? And I don't mean for the United States of America vs. the rest of the world. I mean for the global, human civilization, and homo sapiens as a species.
    • paavokoya 10y ago
      >If you are the sole user of a technology, then part of your cost is maintaining that technology that used to be shared by all the other users of that technology. Yeah the guy who maintained the OS was basically the only guy in the country who could so obviously we had to put up with his bullshit lol
      • cmurf 10y ago
        Space Cowboys (2000) was along these lines. The cranky guy who was the only one who understood the ancient "OS", being its creator, was played by Clint Eastwood.
    • danjayh 10y ago
      W/ regard to space-based and airborne systems (commercial jets also actually operate in a fairly high radiation environment): The risk of faults is not only non-zero, it's expected. Google 'single event upset' and 'nor flash soft error. Short version - (SEU) any bit in your RAM, CPU, peripherals, or databus can (and will, eventually) be randomly flipped in a high radiation environment. (Soft Error) - A random cell in your flash may be pushed into an indeterminate voltage range, and will return a different value every time you read it. So, for instance, you may CRC it, think it's good, copy it to RAM, and then the CRC of the RAM copy will fail because you got a different value the second time you read it. There are ways to deal with this. Google 'lockstep CPU' for information on a common, fairly hardcore approach. Basically you replicate the CPU (and the rest of the hardware) and cross-check every single clock cycle; you essentially have two or more computers in one box doing the work of a single computer. As you can imagine, this hardware is typically entirely custom, tested more thoroughly than anything you've ever imagined (unless you're in a safety critical industry yourself), and very expensive to design & test. Hardware refresh cycles are typically measured in decades, often governed by component obsolescence.
  • benologist 10y ago
    > who writes malware for a 30 year old OS? This may be relevant ... https://en.wikipedia.org/wiki/Stuxnet https://en.wikipedia.org/wiki/Stuxnet
  • hola_hola 10y ago
    >>because who writes malware for a 30 year old OS?) Sounds like a taunt. Almost makes me want to write some code
    • 5ilv3r 10y ago
      I'm interested in your idea for the delivery mechanism.
      • cripblip 10y ago
        Just leave some 8" floppies in the parking lot ;)
      • Sanddancer 10y ago
        Ebay. It'll take a bit of effort to gather up the obsolete materials yourself, but get a few obsolete drives, get a backdoor firmware in there, and wait until some .gov clicks buy it now.
  • nashashmi 10y ago
    An old joke about the pentagon goes along the lines of "we don't need to worry about security because our systems are too old, rare and proprietary to find." As someone else said, security via age and rarity.
    • 5ilv3r 10y ago
      Control of the network helps a lot too.
  • ep103 10y ago
    I worked, once, at a large, national (US) company that will not be named. They had an old database from the early 70s that stored _all_ of their data, everything, contacts, billing, etc. That was accessible through a special proprietary program, that was overseen by one college kid after the rest of his team was let go. That proprietary program was essentially an old DS prompt, which was connected to via a Java applet for an early version of Internet Explorer < 7 that emulated the old dos-like user interface. That Java Applet was connected to via a Java web service, running on machines that had the java applet and ie installed. The Java web service was accessed by tens, possibly a hundred thousand people nationally, through a few different interfaces. The one I was aware of (probably the largest), connected to the Java Web service, through a C# WCF service. The C# WCF service was built as a backend for a new javascript / html4 front end for their company. That new UI, was intended to partially (BUT ONLY PARTIALLY) replace a strange, 100% actionscript web ui made previously. Learning about their system architecture, was quite literally like stepping back through time. I felt like an archaeologist, uncovering layers of an ancient city. It was also amazing how many people were employed supporting each system, each of which could have replaced the lower tiers if they had just upgraded at the time. It was similarly amazing, how the company had laid off everyone at entire layers, when management arbitrarily decided they needed to make cuts, while other layers doing the _exact_same_thing were staffing huge quantities of people, completely unaware that a single bug in the layer beneath them was not being overseen, or supported, and could bring the whole house down at any time.
    • 205guy 10y ago
      These two anecdotes (parent and grand-parent) show that there are two ways to manage legacy systems: a mindful, efficient way and a wasteful, risky way. Come to think of it, those same two ways apply to any project, old or new. So it all really boils down to project management and corporate practices. I agree with all the comments that say that legacy systems, government or corporate are not inherently, necessarily bad, inefficient, insecure, and in need of replacing.
      • ep103 10y ago
        alternatively, the grandparent post's mindset becomes the parent post's when the application needs to be extended, and the grandparent's simply hasn't had that need yet : )
• nickpsecurity 10y ago
  The real problem is there's huge, legacy systems tied to these platforms that they don't understand and are too risky to port/re-engineer. Think of our military systems or payroll going down since software was ported wrong or relied on underdocumented assembler or compiler feature. There's some hope on reducing costs at least. Look up NuVAX for an example of emulators being designed to work exacly like old hardware for fraction of price, space, energy, and so on. I haven't heard attempts but next step might be instrumenting them to trace programs code/data for porting. Or binary translation to modern architectures. I know DEC did latter for VAX-to-Alpha port.
  • VMG 10y ago
    The private sector usually manages to keep it's systems updated. This here seems like the typical incentive and accountability problems of government bureaucracy.
    • devonkim 10y ago
      There are similar analogs in private sector such as the massive prevalence of COBOL and IBM mainframes in even technology-heavy sectors like finance. Factors affecting public sector that private industry typically doesn't face typically, though, is that most federal contracts basically mean that the entire supply chain must be "Made in America." When the federal government buys something, they want to have as much of it made by US citizens on US land and with US investors unless there's no viable alternative that meets criteria (there's hardly any fabrication facilities in the US, for example). Outsourcing labor and not owning as much of your supply chain is profitable. The federal government operates somewhat similar to the Carnegie model of vertical integration in the classic sense. Obviously, this does not hold very well in today's business environment except for a select few companies (Apple being the notable one) because cost models are so different. There are accountability, alignment, and bureaucracy problems across the Fortune 100 aplenty even excluding the parts that sell to the federal government. It's tough to scale organizations as behemoth as large as the Fortune 10, I don't see how the federal government's millions of workers and perhaps even more millions of contractors are any easier.
    • 10y ago
• gherkin0 10y ago
  Honestly, I don't see good solution to this until the rate of technological change really slows down. It seems like your options are to either pay to periodically re-engineer every system or pay to maintain obsolete hardware.
  • x0x0 10y ago
    Maybe pay for vmware or similar to run your old systems on modern hardware?
    • jrmcauliffe 10y ago
      Vmware virtualises 'modern' hardware.
• Animats 10y ago
  On the other hand, they're getting decades out of the software. Use the bleeding-edge stuff, and it's obsolete in two or three years now. Use a "cloud" service, and the service probably goes away within five years. The new stuff has too much churn. Where will Rails be in ten years? Python? Java will still be around; it's the successor to COBOL.
  • orf 10y ago
    Yeah, like all the Python code I've written is just going to expire in 10 years. Popular languages rarely ever die, there is clearly an extraordinarily long tail
    • 5ilv3r 10y ago
      Does your code all work in python 3? I know none of my old ruby 1.8.3 code still works.
      • SwellJoe 10y ago
        I don't think your Ruby 1.8.3 code is supposed to work with Python 3. Seriously, though, as far as I can tell, the transition to Python 3 is extremely low on pain. My Perl 5 to Perl 6 transition will be much more interesting, however.
    • knieveltech 10y ago
      I thought this too back when I coded perl for a living. Still scratching my head over how PHP basically won that fight.
      • SwellJoe 10y ago
        Perl 5 has maintained almost perfect backward compatibility for decades (going back to Perl 4 in some cases...some of our code has some Perl 4-isms, like use of "local", that still work fine). I guess they wanted to wait until they really figured out how to completely break backward compatibility to break it (and they did, of course...nobody's gonna be able to transition to Perl 6 without a rewrite, though it'll be possible to do a transition process with some Perl 5 and some Perl 6, which is cool, I guess).
• Johnny555 10y ago
  Using floppy disks and old hardware and software doesn't sound like a problem if it still runs and does what it's supposed to do. I'm skeptical that building a modern system would really save money since the temptation for feature creep is too great.
  • colinthompson 10y ago
    Agreed. Plus, even if it is decided to modernize any of these systems, we would likely have to run an old system with the new one in parallel for quite some time to ensure a properly smooth transition. Air traffic control is an example of where such practices occur.
  • deleted 10y ago
    [deleted]
  • danjayh 10y ago
    Also, replace mentally replace every instance of "museum ready" in this article with "mission critical" (which most of them probably are), and it'll seem a lot less ridiculous that they maintain them.
  • rst 10y ago
    Even if the software never needs an update (possible for isolated process-control systems, less so for central record-keeping at the IRS), there's hardware to consider -- where, nowadays, do you get a replacement 8-inch floppy drive? Or, say, new read heads for one?
    • ArkyBeagle 10y ago
      This is interesting: http://jimwarholic.com/2009/04/fdd-floppy-disk-drive-emulators.php http://jimwarholic.com/2009/04/fdd-floppy-disk-drive-emulato...
  • beedogs 10y ago
    > Using floppy disks and old hardware and software doesn't sound like a problem if it still runs and does what it's supposed to do. That's great reasoning, until the day it stops working.
• DanielBMarkham 10y ago
  Most federal systems that are anywhere over ten years old (and that's most of them) are complete mysteries to the people who both use and maintain them. A long time ago, I was responsible for such a system. I didn't ask for the job; I simply was the smartest person in the room for too long. I vividly remember one day we had a problem with folks in a remote location entering things and those things getting mangled and/or lost on the way to the system-of-record. For one system, with maybe five thousand users and perhaps a few gigabytes of traffic a month, I was on a call with 30 people spanning most of the Earth. I learned that there were at least a dozen separate systems at that location between the person entering the data and the data being sent to HQ. Each system was old. Each system had a separate vendor which claimed to be the only vendor to understand that system (Sometimes this was true. Many times they were just bluffing.) And -- and this was the kicker -- for each of our dozens of locations, each location manager, because of their friendship with politicians, made their own decisions about how machines were configured and which programs were installed. They were complaining to us because things were bad, but they did not feel like they answered to us. I was responsible for fixing it. At the end of that call, I was reminded of Arthur C. Clarke's quip: Any sufficiently advanced technology is indistinguishable from magic. But I doubt I thought of it in the way he meant it.
  • ramblenode 10y ago
    As a younger developer, I can't emphasize enough how much I enjoy hearing these types of stories. :)
• protomyth 10y ago
  I wonder, what would be your answer is someone came up to you and said "We need a computer that we can maintain and keep in service for 50 years or more. What should we do?".
  • ethbro 10y ago
    Build it with React, Node, and MongoDB on top of AWS, of course... oh, and toss in DSL-of-the-month for fun.
  • jewel 10y ago
    If you know up front that it needs to last for 50 years, and the requirements are unlikely to change, it's easy to justify the expense of getting lots of spare hardware to be able to handle component failure. That's almost certainly going to be cheaper than trying to port the software forward every decade as technology changes. Besides that, use open protocols and open source for the whole stack. Use open source hardware too, but don't depend on being able to order new hardware. Processes could change (we could move away from silicon to something else), or file formats for schematics can change so that you can't find anyone to produce the product. Also helpful would be to target a common VM like the JVM instead of relying on hardware-specific abilities.
    • danjayh 10y ago
      This is exactly what you do -- lots of spares. If you're big enough, and have a big enough need, you can also design you own hardware ... although even that has limits, since foundries do eventually retire process nodes.
  • devonkim 10y ago
    I've read that the answer to this question is why a large part of our military-industrial complex exists - the US government gave Lockheed and others 50+ year long non-cancellable contracts to build equipment for the government back in WW2 because they couldn't justify the business risk (similar to why Fannie Mae and FMAC were created) and it was deemed in the government's "best interest" to keep these companies afloat in some capacity. Aircraft carriers we don't need, guns that are long obsolete, tanks that the Pentagon doesn't want anymore - the list goes on with the excess manufacturing spending the government seems to be roped into buying because of... "reasons." I've never read through those contracts myself but I would hope that they're public record if they're so old and important.
• Shivetya 10y ago
  anecdotal, in the early to mid eighties I was in the US Air Force. The machine I was first assigned to watch over was in the secure comm center. This burroughs machine was the first non-tube computer made by burroughs. it could boot from paper tape or card and was replete with blinking lights. later I moved up in tech to a sperry/unisys system. all our personnel data and such was loaded via cards, physical cards in multiple boxes till near 88. So honestly I don't doubt they still do similar. I was just so glad we got out of boxes of cards because having to fix runs each night got old and all for bent card. it got me into programming, turbo pascal at the time. why, when we moved off physical cards it was then onto 360k floppies. The problem was, the upload/download programs provided could take half an hour or more to transfer to the 1100/70. The turbo pascal program did it in five or less per disk without issue.
• syngrog66 10y ago
  On one hand it seems inefficient and perhaps dangerous to be reliant on such old systems. On the other hand, the idea of a new software project to replace it also sounds at risk of being extremely expensive and overly complicated. Because of all the government contracting anti-patterns. In theory there's a middle ground that avoids both these extremes. In reality, with government software... I'm skeptical it will happen.
• tn13 10y ago
  Despite all the ipads and what note I have found a pen and notebook to be the better note taking equipment than anything else. If it is getting the job done cheaply and efficiently that required and better than the alternatives it is the best technology to use.
• panic 10y ago
  "Feds spend billions to maintain museum-ready buildings" "Feds spend billions to enforce museum-ready laws" These computer systems aren't even that old compared to many things the government spends money on.
• chx 10y ago
  I want to know only one thing: COBOL is named under Social Security so I suspect the "outdated computer language that is difficult to write and maintain" Treasury uses is not COBOL -- but oh god then what it is??
  • fiatmoney 10y ago
    Dating from 1960 ("the systems are about 56 years old"), and especially given that the system in question is likely an IBM mainframe, Fortran would be my guess. Fortran can actually be surprisingly pleasant, at least as pleasant as C, but I'm guessing their particular code is not.
    • bigger_cheese 10y ago
      Could be PL/I
  • riprowan 10y ago
    I'd guess these are coded in assembler, or possibly Fortran.
  • Kadin 10y ago
    One of the many other articles that have been floating around the past few days mentioned that Treasury has a bunch of programs written in an old IBM architecture's version of assembler.
    • chx 10y ago
      Bingo! I just found http://arstechnica.com/information-technology/2016/05/government-agencies-keep-sacrificing-cash-to-zombie-it-systems-gao-finds/ http://arstechnica.com/information-technology/2016/05/govern... mentioning both Individual Master File and Business Master File written in IBM mainframe assembler 56 years ago. Joy. Although I suspect it's a bit of an exaggeration because it seems the IRS is using System/360 which started 1965 https://books.google.ca/books?id=HwniRloeB6cC&pg=PA2&lpg=PA2&dq=ibm+360+irs+%22master+file%22&source=bl&ots=DNqni9DXg1&sig=ag4xDy_ERLUjJeslLy3sIq5rUks&hl=en&sa=X&ved=0ahUKEwix8Krp_PnMAhUGyWMKHfoGA0sQ6AEIKjAB#v=onepage&q=ibm%20360%20irs%20%22master%20file%22&f=false https://books.google.ca/books?id=HwniRloeB6cC&pg=PA2&lpg=PA2...
• JackPoach 10y ago
  I think this deserves its own term - porkware
• jenkstom 10y ago
  Seems like a great opportunity for virtualization...