23 ms·

America’s Cities Are Running on Software from the ’80s

228 points by eplanit 8y ago

mc32 8y ago
It seems they this kind of system would benefit from a more user friendly interface; however, I don’t think all decades old software necessarily needs to be uprooted and replaced because it’s old, so long as it’s functional. BaRT, I wonder if they’ve upgraded their system software to more modern software? They certainly could add more intelligence (system health, wear, etc.) and begin automating more aspects, so it could make sense to upgrade here, but I’m sure there are lots of places where upgrades are not necessary and the old software is good enough, so long as it’s still supported by a vendor.
- blakesterz 8y ago
  >> I don’t think all decades old software necessarily needs to be uprooted and replaced because it’s old, so long as it’s functional. I'd say that's especially true in places like power plants. The old stuff is impossible to hack remotely because it's not on any networks. The rush to put everything on the network has made these types of systems vulnerable and places are only now trying to figure out how to secure them.
  - deleted 8y ago
    [deleted]
- NullPrefix 8y ago
  >add more intelligence (system health, wear, etc.) Of course, telemetry is the first thing that comes to mind about modern software.
- jdavis703 8y ago
  Which old software are we talking about? Is it the Clipper Card system that has tens of thousands of business rules that prevent transit agendcies from rolling out new passes and pricing strategies? Or are we taking about the software that runs the trains and currently causes a bottle neck through the transbay tube? Not everything needs to be rewritten in React using Google’s Material UI design standards, but my god, our public software really sucks. Even as the article talked about the tax assessors office... I know someone who bought a condo, but SF can’t collect property tax because their office is so backlogged. So instead they told the owner to make sure they save the money because in a couple of years they’re going to come asking for it.
  - wbl 8y ago
    The software is not the issue with the tube. It's the fixed block system imposing minimum 1 minute headways.
  - derwiki 8y ago
    > SF can’t collect property tax because their office is so backlogged IME they're able to collect property tax, no problem.
    - gph 8y ago
      Clearly the truth lies somewhere in between both these statements. Something like: "SF can't collect the full and accurate amount of property tax in a timely manner from properties that were recently reassessed because their local assessor office is backlogged due to use of old and inefficient software." Which is basically what this whole article is saying.
  - danepowell 8y ago
    The Clipper Card system _must_ fall into this category. I recently called because I was certain my card was being fraudulently used--the value would dip $5-10 for a single ride, and there were top-up credit card transactions that didn't correspond even remotely to my credit card statement. Turns out SF Muni and Clipper Card (and possibly Clipper Card's billing vendor?) are two/three completely independent organizations. Everything is passed between them in batches with up to a several-month delay, with batches often arriving out of order. This makes auditing nearly impossible and is what made me think something sketchy was happening. It's a miracle it works at all.
  - inferiorhuman 8y ago
    Is it the Clipper Card system that has tens of thousands of business rules that prevent transit agendcies from rolling out new passes and pricing strategies You're putting the cart before the horse there. It's the transit agencies themselves that advocate for the lack of prices. BART, for instance, held out on implementing TransLink/Clipper for ages because they wanted their own BART-only cash purse on the card. Clipper is still fairly new and I don't think the fare rules have changed much, if at all, since its inception.
inetknght 8y ago
> Assessors are prone to make mistakes when using the vintage software because it can’t display all the basic information for a given property on one screen. Well they're in for a surprise if they think modern user experience designers would fix that.
- rietta 8y ago
  Geez. This is the truth. Applying A/B tested mobile design fads from consumer app to data driven for-professional-use software is leading us in a very bad direction.
  - jjoonathan 8y ago
    Replace half of the labeled buttons with inscrutable generic hieroglyphs and hide the other half under a hamburger menu! <Aliens Guy>User Experience</Aliens Guy>
  - gambler 8y ago
    Very true. Modern web UIs have some good ideas and some bad ideas, but fundamentally they are designed for people browsing stuff and optimized for conversion and engagement. At work, people do work and you need to optimize for efficiency and reducing burnout. A lot of people don't get this and blindly copy designs. Yay, your tool now looks like Amazon! ...which people leisurely use two hours a week, while your tool is being used 9 hours a day, in a crunch.
- doodpants 8y ago
  Agreed. Perhaps this software from the '80s should be replaced by software with interface design sensibilities from the '90s.
- smt88 8y ago
  I've found some great UX designers who could fix this. They complain that they're constantly at war with designers, who think things like text labels are "ugly".
  - 8y ago
deanalevitt 8y ago
> Minnesota spent about a decade and $100 million to replace its ancient vehicle-licensing and registration software, but the new version arrived with so many glitches in 2017 that Governor Tim Walz has asked for an additional $16 million to fix it. This raises so many questions.
- icedchai 8y ago
  Insane. That's like 100 people for 10 years (ballpark...)
  - vonmoltke 8y ago
    That's some awful cheap people.
    - icedchai 8y ago
      100K/year average. Consider they are not all highly paid engineers, Minnesota is like 1/3rd the cost of the bay area, a lot of it was outsourced overseas where they're even cheaper.
- kiba 8y ago
  Raise what questions? It seems par the course for many software projects.
  - deanalevitt 8y ago
    Well, my questions are: What was the original budget? Were there cost overruns? Who got the tender, and why? Where was the complexity that made it a $100M, decade long project?
  - imtringued 8y ago
    I don't see how cost overruns by more than an order of magnitude are par for the course.
    - 8y ago
thisisweirdok 8y ago
There are multi-billion dollar companies that still need IE11 because they don't want to build new internal applications. Contrasted to the funding that some cities get, it's amazing they have computers at all. There's this huge fervor over disruption and startup dollars, but just like the rest of the country... municipal software infrastructure is just sitting around full of holes.
chobytes 8y ago
Im not sure that I think software is always bad just because its old. I recall working as a technician years ago we used text user interface ticket system from the early 90s. While it wasnt the prettiest system it did its job and one could navigate it quickly once they has commited the keystrokes to muscle memory. We thought about replacing it with more modern software but after evaluating a few of the options found none of them to be particularly compelling. If anything, we appreciated what our “old” system could do even more.
- brobdingnagians 8y ago
  I'm constantly and pleasantly surprised by how nice and feature-ful text utilities are. First time I used htop, I was much happier than most of the GUI task managers I had used. It seems like having a simple user interface helps the designers focus more on having it actually do more of the useful things than on how many pixels to put on that margin...
  - rayiner 8y ago
    You can see this first hand in almost every single Microsoft program that’s undergone a Win32 to Metro/UWP transition. The new versions can’t do anything.
    - maxxxxx 8y ago
      So true. It’s hard to believe that their new control panels since Win 8 still lack features of the old ones.
    - kalleboo 8y ago
      See also: any mobile anything. Many times a week do I have to switch to the desktop site to get things done since it wouldn't fit into the clean mobile design.
    - AnIdiotOnTheNet 8y ago
      And they do all that nothing so much slower than the old ones. Everyone involved with that project should be ashamed.
redsavagefiero 8y ago
Good. We don't have to worry about node.js, apache whatsit and Google go-away v1.0 being our goto technologies in a critical space. That's progress. Stick a rest/rpc interface on the outputs and ctl and attach a wireless dongle to a middlebox: voila modernized! If you really want to get fancy attach ELK to the 'stack' and put what data you want in it to represent something or another. Mission accomplished!
- derwiki 8y ago
  This is sorta what my university did when they launched online registration. And that was why only 10 students at a time could use the online registration system-- that was the number of virtual terminals the mainframe had. And it was a Java applet :weary:
tomatotomato37 8y ago
And? I'd rather civic data being handled on some ancient console that's relatively bugfree than some modern webapp nonsense that loses data half the time but has pretty hamburger menus.
kilon 8y ago
If the sofware is from the 80s that is very good news, it means it runs on simpler hardware and the code will be simpler and easier to fix. Which is what made it viable for us to code in Assembly back in the 80s and early 90s with the same ease someone codes in c# nowadays. Coding in Assembly now is pure insanity. Sure you could still write super ugly code but super ugly simple code is far better than super ugly complex code. Even if it comes with a forest of gotos.
lqet 8y ago
So? I remember a large bookshop in my hometown in which each section of the store had a terminal with DOS-like interface to their internal database. Employees were able to use it blazingly fast, and they still used it a few years ago. Except for security concerns, why should newer software automatically be better?
- rightbyte 8y ago
  Old crud apps ware probably the prime of usability. These hotel booking systems with green text terminals and hot keys for everything. The video store where I grew up had a similar text based interface to some floppy located database for rentals. Amazingly fast and some override button to force invalid entries when someone else had messed up or something was registered wrong. Binaries run straight from a floppy with no fancy OS to interrupt the very simple yet manually tedious things it does. Many apps today are way over cooked for what they do.
snarfy 8y ago
> Issue: Control systems for most of the city’s 3,000 traffic signals date to the 1960s and make it harder for the department to manage congestion. Cost (per intersection): $175,000 to $735,000 That cost seems outrageous. Are traffic signal control boxes really that expensive?
- clarkmoody 8y ago
  It's a government expenditure, so the market signals of profit and loss cannot work to keep costs under control. Classic monopoly pricing at play. The outrageous prices could also be attributed to good old-fashioned cronyism: some city council member's brother owns a local civil engineering firm that gets a 10-year contract and names its own price.
  - snowwindwaves 8y ago
    Have you ever worked for a government or public institution? Everything takes 5x longer than it should.
- syn0byte 8y ago
  In one case I've heard around the water cooler; A city's intersections were(are?) rocking Sun SPARC servers to do all the control stuff. Yes, each intersection apparently had its own. I don't know if I ever really believed it until this article confirmed that kind of cost.
- mymythisisthis 8y ago
  Upgrading traffic systems will help with self-driving cars. It's one of the missing puzzle pieces. Think of what modern cameras and data storage, would do for innovative change.
- mattkrause 8y ago
  I'm also having a hard time believing that Philly has half a billion dollars worth of traffic signals.... Unit costs can be misleading if they're hiding a lot of non-recurring engineering costs. I can easily believe it would cost $175,000 (or more) to replace a single set of traffic signals, especially if they are built from scratch and need to interface with some ancient system. However, the 2nd, 3rd, .... 3,000th signals ought to be progressively cheaper as things get worked out.
buboard 8y ago
Oh well, good for them, have you seen the software they make nowadays?
oliwarner 8y ago
Things like this obviously attract headlines, but the real problem comes from the millions of other machines, decades newer than the ones in TFA, that are still irredeemably EOL. I contract for a few companies that supply local governments in the UK. I regularly have to fill out stringent checklists about IT security provisions. PCI-DSS and beyond. So I always chuckle when we meet them on-site and they're running an XP install with IE8, plastered with toolbars. These are machines I would sooner throw into the abyss than try to rescue. They'll make me jump through hoops but there they are inputting and manipulating citizen data on machines that are almost certainly part of a botnet. Tablets are also increasingly a problem. As a b2b-service webdeb, I still see a lot of first gen iPad Minis in active service in enterprise. Last patched mid-2016. Oh well. The next big worm attack is going to be devastating to organisations like this.
- at-fates-hands 8y ago
  > Tablets are also increasingly a problem. As a b2b-service webdeb, I still see a lot of first gen iPad Minis in active service in enterprise. Last patched mid-2016. I used to work for a large oil company. Their IT guys were pretty adamant about security. Right before I left the company, they purchased another smaller distributor. All their guys had ipad minis and same thing, were last patched in 2016. The IT guys right away were telling the execs, "Yeah, they're all getting Win10 laptops, fully patched and locked down to use only our software. No way we're going to keep using these."
gdulli 8y ago
Meanwhile, my modern engineering organization has projects less than three years old that are already obsolete or orphaned because they were built on someone's favorite JS framework or CMS that isn't relevant or growing anymore and no one wants to touch them now.
missblit 8y ago
> the shift from old-school servers to rented cloud storage has made it tougher than ever to fund upgrades They do know that it's still possible to run old-school servers, right? There must be some upside or need to switch to cloud computing, either that or people just get blinded by buzzwords. Hard to say from the article. Anyone know why governments may switch to cloud despite the issues in the article?
- gambler 8y ago
  I don't know about governments, by at my last job the transition to cloud was driven mostly by: 1. The desire of some people to have AWS stuff on their resumes. 2. The optimistic belief that the same people who couldn't figure out how to manage normal servers in 10+ years will figure out how to design good serverless apps in a few month.
- coldacid 8y ago
  Because they are very much blinded by buzzwords, and want to look like they're "with it" by choosing whatever seems the most advanced even if it would actually detract from the job at hand.
- linuxftw 8y ago
  Governments are mostly administrative. Running your own servers is 'real work' which mostly gets contracted out. Used to be necessary to have your own servers because that's where the software your administrative people used ran. The government could build it's own power plants, and hire the employees to run them. Probably cheaper/easier to contract that out (eg, buy power from power company).
quotha 8y ago
if it ain't broke..
AdmiralAsshat 8y ago
I work on several pieces of business software at my job. One was built for UNIX mainframes, almost entirely in C, with the end-users still capable of running on dumb terminals (or these days a PuTTY window). The "successor" runs on Windows servers with J2EE, a GUI, and all the modern niceties. The most common complaint I hear from people who've switched from the legacy software to the modern one is that their end-users' productivity is greatly reduced by the point-and-click interface. The older software's TUI had a very steep learning curve, but almost like learning Vim, once they developed a muscle memory for the necessary keystrokes, they could access and update information much more quickly.
- forinti 8y ago
  I once saw an attempt at modernising a mainframe application that had a single input line where the user would type a sequence of numbers. The new GUI had all the fields organised and labeled. The user then requested and got a single input box where he would type in all the numbers just the way he did before.
deleted 8y ago
[deleted]
bigpicture 8y ago
A few years ago I had to update some FORTRAN code written in the 1980s that did bond valuations and some other financial stuff. It hadn't been touched since it was written and was now running out of space because the number of bonds keeps increasing year after year. I'd never done anything with FORTRAN before, so I spent a couple days reading about it and looking into how it works, fearful that I'd muck everything up trying to fix it. I open the code and find that it was the most elegantly written software you could wish for, with fantastic comments and structure. I changed two numbers and was done, buying us another 30 years or so of rock-solid service.
- drharby 8y ago
  I hope to god you over sold yourself
- systemtest 8y ago
  Did you leave a comment for the guy who has to fix it in 30 years?
  - pinko 8y ago
    Hopefully there's even odds it will be a woman by then.
    - systemtest 8y ago
      Perhaps we go full circle and have predominately female programmers by then.
      - edoceo 8y ago
        A full circle puts us back where we started?
        mrguyorama 8y ago
        Programming used to be a "woman's job" as secretaries.
JoeAltmaier 8y ago
America's industry is running on software from the '80s. You can't imagine the horrors on the shop floor. Never mind XP or Windows ME - DOS with green-screen terminals; hand-coded apps booting on home-brew hardware; IBM dinosaurs that should be extinct but lumber on in the back office. And there's nobody that can help them fix anything. So upgrade you say? Then they'd have to start from scratch. And they long ago laid off the accounting experts, tax lawyers and business planners. They have button-pushers and data entry folks. Nobody left who knows were the data comes from, where it goes, or who all uses it.
- bluedino 8y ago
  We recently bought a $4 million dollar steel fabrication system, automated, top of the line, runs Windows XP. Rumor is the company has been trying to re-write the control software to run on Windows 7 for the last 10 years (name rhymes with spaghetti) We interact with it using little data packets (ASCII, sent from a VB app) of coordinates to tell it what to do. Insane amounts of money would be lost if that machine goes offline for a day. We can't just replace it unless it's 100% compatible. Some of our bigger competitors have 10 of these machines. There's no real benefit of upgrading them, though. We'd still run the whole damn machine behind it's own ASA and on it's own network. The users don't care. And our programmers wouldn't be happy to change it.
  - southern_cross 8y ago
    I regularly see stories of software that is at least 20 years old now, running multimillion dollar industrial systems that are expected to last for at least another 20 years before being retired (if even then). There will probably be no changes to that software in the interim, and it's possible that it can't be changed even if you wanted to, for various reasons. As long as the computer is well-isolated, the biggest ongoing risk here is replacing old and failing PC components when necessary. But the same might be also be true of the industrial systems themselves.
- reaperducer 8y ago
  You can't imagine the horrors on the shop floor OK... DOS with green-screen terminals Not a horror. If it works, why mess with it? hand-coded apps booting on home-brew hardware Also not a horror. Bespoke hardware is everywhere, as are hand-coded apps. AI does not yet build software. IBM dinosaurs Still not a horror. Perhaps you just don't have a lot of experience with legacy systems.
blunte 8y ago
This is like everything else in life: there is the initial cost, there are maintenance and support costs, and there are replacement costs. With roads, it's fairly obvious when maintenance or replacement is required, so people responsible for financial expenditures cannot ignore them. But with software, the "invisible" nature of it seems to allow those responsible to ignore reality and kick the problem down the road until hopefully it is someone else's problem. (Incidentally, this is how a lot of modern public companies with with respect to 5-year CEOs and quarterly earnings and actual responsibility to shareholders, employees, and public). Anyone who has technical experience knows that it's a tough sell to convince upper management to spend money on problems that they can't even see. The big question is, when is the technical debt (or outright catastrophic failure of a decayed system) worth more loss in revenue than its replacement would cost?
mdesq 8y ago
A few years ago, I met guy in my travels who seemed quite carefree and just enjoying frequent international travel with his spouse, funding various art initiatives, etc. Turns out he wrote some utility-related software for a large American city back in the 80s and has nice a recurring revenue stream from that. Other than some now very occasional support requirements, it's essentially just a source of passive income. He said switching costs would be too great, and there's no need to change something that's working.
WhompingWindows 8y ago
I work in the federal government and the performance of the tech is abysmal. If all of the ridiculously talented developers in SV who are using algorithms to capture our attention via psychological hooks, instead used their talents to improve our government tech infrastructure, the world would be a far better place.
- magduf 8y ago
  The problem there is that the government doesn't want to hire those ridiculously talented developers in SV to do that improvement. The government process for procuring software is totally broken and hopelessly inefficient, and when the government does try to do it on its own, they can't hire decent developers because they absolutely refuse to pay industry-scale wages. Somehow they think that the "benefits" of a Federal job are competitive with what industry is paying, but I guess they haven't been watching the news where Federal workers have to constantly worry about getting furloughed due to government shutdowns.
  - aaya 8y ago
    Agreed. It seems that these institutions are unwilling or unable to pay for talent and that they don't have the capacity--in terms of processes and culture--to leverage that talent even if they were able to acquire it. Here's a relevant comment thread from a post a few weeks ago about Wells Fargo going offline: https://news.ycombinator.com/item?id=19110672 https://news.ycombinator.com/item?id=19110672
sixdimensional 8y ago
I have one phrase to add to this discussion - "sustainability"[1]. Systems have to be engineered differently with that in mind. I have real concerns that most of what we are developing in high level languages/systems (with dependency management dependent on external sources) will not last. [1] https://simple.m.wikipedia.org/wiki/Sustainability https://simple.m.wikipedia.org/wiki/Sustainability
tsmarsh 8y ago
Says the company that runs on the Bloomberg Terminal.
- smhenderson 8y ago
  I've never seen one but I've read that they're actually quite user friendly and provide a wealth of information. Was what I read over rated? Any examples of why they're not so great?
  - richjdsmith 8y ago
    I don't know what they're going on about. BT is fantastic.
  - mruts 8y ago
    The bloomberg terminal is pretty awesome. It's written in several different languages though, so maybe that's the GP's problem?
  - tsmarsh 8y ago
    They are, but they're basically 80s/90s technology. I work in emacs with lisps (mostly clojure) on x86, usually via a terminal emulator. Old tech is stable, fast and cheap. I know that, you know that, Bloomberg make their billions off of it... but they're calling out Cities for it. Just weird.
dev_dull 8y ago
> Minnesota spent about a decade and $100 million to replace its ancient vehicle-licensing and registration software, but the new version arrived with so many glitches in 2017 that Governor Tim Walz has asked for an additional $16 million to fix it. This is exactly why cities don’t replace these old pieces of software. It’s not that a intern couldn’t write something better. It’s that the physical act of “replacing” the old with the new is often very expensive. This is a story that’s unfortunately all too common in government. Other people’s money on other people and whatnot.
ArtWomb 8y ago
Just a heads up. Sat Mar 2, there is a kickoff event for open data week and civic tech in NYC https://beta.nyc/ https://beta.nyc/
wolfspider 8y ago
This is essentially what I have been doing the past few years among other tasks. The difference is we do a lot of things in-house and even receive security audits from companies like IBM and insurance-based development process audits as well (do you have backups of source code, CI, integration testing, etc..). Getting off an AS/400 is not easy as it has a completely different architecture and OS. When it comes to data like this legal assurances have to be in place and voted on before real change happens. We have a financial package which replaces this system and some of it written by our team while the backend and most of the migration handled by an outsourced company. The challenges we faced were handling lots of data going back decades to migrate and doing audits for different departments as the migration was ongoing (parallels). All staff have to be trained up in the new software. All of our new stuff has to be PCI compliant. Many processes based on paperwork have to be turned into an e-filing system- this means heavy OCR workloads to get it into a newer system. A lot of financials end up becoming transparent and visible to the public as a result. If the government were to adopt better practices where technology is concerned it will take interest from the citizens themselves and many additional high level positions focused solely on technology. In this article when it says it took them 4 years to just acquire the funds to start on the project I believe it.
griffinkelly 8y ago
Reminds me of a great story on 60 minutes about the MTA and how they're using signaling systems built in the early 1900s: https://www.cbsnews.com/news/mta-why-has-the-nyc-subway-gone-off-the-rails-60-minutes/ https://www.cbsnews.com/news/mta-why-has-the-nyc-subway-gone...
peatmoss 8y ago
Thought exercise: if you wanted to write code to support a business or nonprofit, and you knew that the software would be used for many years, and probably never see an update, and wouldn’t have anyone to tend network services, how would you build it? Me, I'd probably go with a static executable with a terminal UI and a flat file sqlite database that could be copied by mortals, and hope that whatever I set up as a backup regime would survive future migrations to new hardware.
- smacktoward 8y ago
  A fascinating idea -- kind of a software analogue to the 10,000-year clock (https://en.wikipedia.org/wiki/Clock_of_the_Long_Now https://en.wikipedia.org/wiki/Clock_of_the_Long_Now).
evadne 8y ago
I would encourage thinking about this through the lens of survivor bias. If said pieces of software were so bad, they would have been replaced. So, the business utility delivered by these pieces of software surely exceed the cost of maintenance and upkeep.
aeturnum 8y ago
sigh The buildings were built in the '80s too (sometimes even earlier!) and it's fine. Software is no longer new. If there are problems, they don't come from the age of the constructions.
- zamalek 8y ago
  Upgrading it would also likely lead to extravagant costs to the taxpayer, as these government software projects tend to do.
- sambf 8y ago
  The vast majority of Paris buildings were build between 1851 and 1914 [1]. I myself live in a 1903 building. [1] https://www.comeetie.fr/galerie/BatiParis/#12/48.8589/2.3491 https://www.comeetie.fr/galerie/BatiParis/#12/48.8589/2.3491
clavalle 8y ago
Legacy code is everywhere. What slows down adoption of new software in these areas isn't generally the system being replaced itself, but the requirement that the new software interface with some other 'critical' legacy system(s). It is an ossified web of business processes and tools and their dependencies.
- nobleach 8y ago
  The biggest problem I ran into (while supporting these old systems based on DOS and Netware 3.1) was keeping them inline with newer tech. I had a client (an accountant) with several old DOS based apps. Getting their network of printers to work reliably was a crapshoot. I'd get calls all the time that some program just stopped printing. At least Netware stayed reliable.
llacb47 8y ago
I'm glad that my county has modernized.
failrate 8y ago
Let's be open: software hasn't become fundamentally better even with better tools and languages. Quality software is more a product of the person or team that created it than the available bells and whistles.
- frosted-flakes 8y ago
  Maybe not higher quality, but certainly more capabilities.
  - failrate 8y ago
    I yield that my potential high point in terms of scale and complexity are objectively higher, but I feel like we are solving the same problems over and again with various permutations of bandaids and baling wire.
gtycomb 8y ago
You are looking at one of the great lost opportunites for hacker enterprenuers. Cities/state agencies depend on cio-purchasing managers to help maintain their IT, and this ends up in third-party handshakes i.e jobs run by very specialised outsourcing outfits who know how to market here. It takes a different type of management mindset to turn this around but things are entrenched firmly for change. The agencies will keep getting the '80s technology with no end in sight.
discreditable 8y ago
This article makes me wonder what nightmares await those who have to maintain today's software at 30 years old.
- AnIdiotOnTheNet 8y ago
  It'd have to be valuable enough to still be used in 30 years.
moorhosj 8y ago
Entire country running on Constitution from 1700s.
- dragonwriter 8y ago
  Actually, the last update was in 1992, which is pretty far from the 1700s. (Admittedly, the update that was merged in 1992 was submitted in 1789.)
  - Aloha 8y ago
    Talk about protracted development ;-)
djohnston 8y ago
that probably means it's well written code.
ahallock 8y ago
Is software from the 80s necessarily bad? I think it's a fallacy to assume age equates to poor quality, but you see this scare tactic used in media quite a bit.
- coldacid 8y ago
  It's not bad, but requirements and usage methods have changed, and the software hasn't. It's fair to update or replace software in such a situation, so long as the new/updated system meets the actual needs of its users as they are today.
taf2 8y ago
AM I the only one that thinks this is totally awesome. If the code I write today is still in use in 40 years that would feel good.
- flatfilefan 8y ago
  Especially when it is so good that it looks like a black box to the word: correct input -> correct output; and nothing more.
AngeloAnolin 8y ago
I think this stems from the fact that IT in government is still treated as a cost rather than an integral part of the business. Generally, software projects will require hundred of hours of vetting, subject matter expert analysis, political will and pressure before they even get past the approval stage. And by the time the project is approved, the process which the project was intended to modernize may have already adopted a different set of methods thereby setting the project for failure (by not meeting what was needed).
Johnny555 8y ago
I think this is part of the problem: "We’re dealing with an irrational public who wants greater and greater service delivery at the same time they want their taxes to be lower," There's noting irrational about wanting government to take advantage of the same technological advancements that businesses use to improve service while cutting costs. When I order from Amazon, I see real-time inventory, get immediate order confirmation, near real-time tracking as the package is handled (and actual real-time tracking for some orders were I can see where the delivery vehicle is), and delivery to my door step is 2 days (or sometimes the same day). In contrast, when I ordered a dog license from my county, I had to mail (FAX was also accepted) a paper form, and then 6 weeks later I received a metal plate in the mail (just a numeric ID on it, no personalization). The only way I knew they even received my order is because my check cleared the bank 2 weeks after I mailed it (no credit cards allowed). I don't expect my county animal services department to build an order an fulfillment system that rivals Amazon's, but there are thousands of this offices across the country, they could all cooperate on one system and save money dealing wit these paper forms.
- Aloha 8y ago
  If the paper form costs them less to process than whatever automated system that would provide whatever quality of service you deem acceptable would, why would they switch. Newer and more automated doesnt always save money. I work for a medium size company, our expense reporting system is 100% paper or excel driven - we could go to Concur, but it would probably cost more than our existing system does, and may not pay for itself within the lifetime of the product.
- dajohnson89 8y ago
  If only the solution were higher taxes. Local & federal government alike are comically wasteful of taxpayer money. I have no problem with paying higher taxes per se, but the return on our investment right now is abysmal. Something has to be done to make governments more efficient. I don't know what that is.
- flatfilefan 8y ago
  "There's noting irrational about wanting government to take advantage of the same technological advancements that businesses use to improve service while cutting costs." - the irrational thing is to expect this in absence of competition. How about having two or more local governments acting at the same time to enable the true competition?
ganonm 8y ago
Maths is running on 'software' from 500BC. Don't see anyone complaining.
boznz 8y ago
Always look at your dependencies and in that I include the programming language, the os and the hardware. I have seen good systems fail for supporting the wrong label printer.
tomc1985 8y ago
Dear people, can we please stop freaking out about old software?
intrasight 8y ago
My first job in the late 80s was programming nuclear power plants. It was a good learning experience. I bet that software is still in place too ;)
- southern_cross 8y ago
  IIRC, the original software pretty much has to run as-is until the day the nuclear power plant gets retired - typically 40 years, at least.
  - flatfilefan 8y ago
    What about the bit rot, decay, half life and all that?
    - southern_cross 8y ago
      As I understand it, the software itself never really gets changed, because if it did then the whole plant might have to undergo testing and regulatory approval again. But the reactors get shut down periodically (every three years, maybe) for refueling, so the control hardware undergoes routine maintenance at that time. Or something like that.
gambler 8y ago
Almost every week, I wish the framework I am working right now was written in Smalltalk 80 rather than Java.
rc_hadoken 8y ago
Things that last a long time tend to keep lasting a long time. I'd bet the code is stellar. New things have a higher risk of what I call 'non-survival'. A concept I've learned from Taleb.
gambler 8y ago
It's a mistake to think that just because something is written in 2019, it automatically benefits from the extra 40 years of collective experience. The collective has Korsakoff syndrome and aversion to history. To be fair, there are two general kinds of legacy systems. Ones that work so well it makes no economic sense to replace them and ones that are such a giant, horrible clusterfuck that they warped the entire organization around their own idiosyncrasies. In the second case, replacing the system requires replacing most business processes as well, so it's very hard.
tyre 8y ago
This is not a software problem. It's a purchasing problem. We built better software for cities and ran into this over and over and over again. They're not incentivized to change. Citizens don't care or know enough to raise hell about city operations. The employees who do the work can't advocate for themselves and/or feel no reason to risk their jobs to make their own lives better (there's no reason to be more efficient.) Leadership cares mostly about press releases—blockchain or AI or whatever buzzword sounds better to them than the incremental changes that will actually contribute to fixing cities. It's frustrating, bleak, and the inverse of inspiring. I tried for 2.5 years before losing hope. My co-founder carried on another year before running into the same. The best way to get people what they need is to bribe them—e.g. work through an intermediary who takes a cut and delivers the rest to a political campaign—which we refused to do. Local governments affect most Americans' day-to-day far more than the national political shitshow and this only becomes more true as populations further concentrate in cities. And yet, they are utterly soul-sucking morasses of internal and external politics, misaligned incentives, and bureaucracy. They drive out the people who can and genuinely want to effect change. It's a damn shame.
- pergadad 8y ago
  For public sector it's inherently risky to buy software and even devices. They often lack the capacity to properly outline what they need, to understand what's possible, to know the real cost/price they'd need to offer to get good results for tenders, and most importantly lack the ability to write for technical tenders to get what they need. The effect of that if something is running well often the most rational solution is to leave it running. Yes it might not be the most efficient solution, but it might still be more cost effective than losing 3 months of work due to conversion to a new system, in addition to all the extra cost, training, downtime, ... In the current ecosystem more and more also seems to be license based. Do you really want to change a decently running system with just occasional maintenance cost for an untested system that will change all existing processes and in addition ups your annual cost by quite a bit? Any time you change a system you probably have to change some additional systems too as suddenly things don't work together anymore.... Cities are often stretched for resources ranging from (good) personnel to financial to physical space to ... It's probably in many cases the rational decision to stick with how it is if the system mostly works.
stcredzero 8y ago
For local officials throughout the country, the shift from old-school servers to rented cloud storage has made it tougher than ever to fund upgrades. They can budget physical equipment as capital expenses, meaning they could issue bonds to pay for them. But cloud computing is a service, as the people selling it love to say, which means officials have to pay for it with operating funds Could a company structure access to cloud computing over a fixed period of time as a capital expense? What if there was some sort of on-site access device involved?
StreamBright 8y ago
Ain't broken, don't fix it.
rmason 8y ago
I've publicly advocated a new development model to both the state and cities here in Michigan. Share the cost of development and make the software open source. Unique differences such as laws and tax rates should be in proprietary plugins. Makes zero sense to me that say fifty states pay a vendor to create software for a new federal law. The state's pay for one off software but the vendor writes a single program that's either database driven or uses custom functions to reflect stuff unique to the state. It's a colossal waste of money. I can't get the government people to tell me why I am wrong. They just seemed locked in to doing things the way they've always done them. Hey, where's that 8 inch floppy again, I've got work to do.
- robaato 8y ago
  Entrenched business models/relationships/incentives no doubt at work... Rather like ancient guilds and raising the bar of entry to newcomers who might otherwise undercut them.
bungie4 8y ago
Nothing is easy anymore. The stacks are so deep with millions of lines of code. When it works, its amazing. But it seems not much just 'works' anymore. Compared to the 80-90's. You were very close to the metal. Comparatively speaking.
robaato 8y ago
Slightly off topic, my first significant job in '80s was programming a Stratus in their PL/1 implementation. As I recall it was really quite a nice mix of some COBOL record structures with more modern language options. Shame it didn't last! The Stratus itself was well engineered but missed their opportunity in the market subsequently. Moved on to writing C for an RS/6000 and doing graphics work (few low level libraries around then). Interesting times...
fmela 8y ago
Oh, that's not so bad. American nuclear missiles are running software from the 70s, loaded via 8-inch floppy disks. https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nuclear/index.html https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nucl...
andrevergamito 8y ago
I work for a billion-dollar-company, in the semiconductor industry and yet we use a 1986 asset tracking software we access through telnet and despite the 7M USD effort in three year for the development of newer software we ended up reverting to the old software. Non to mention we still use Lotus Notes for email and windows NT on some production machines. The same machines that validates a good 30percent of electrical components on electric devices and cars...
turc1656 8y ago
This is the problem right here - "It took her office almost four years to secure $36 million for updated assessors’ hardware and software that can, among other things, give priority to cases in which delays may prove costly." Why the hell does it cost $36 million for that? It's typical bloat/waste/padding as far as I'm concerned. Just another municipality that pisses taxpayer money away. I guarantee you that a private business could have made that for a fraction of the price.
microcolonel 8y ago
If it ain't broke, don't fix it. If it is broke, do fix it.

228 points by eplanit 8y ago

mc32 8y ago

It seems they this kind of system would benefit from a more user friendly interface; however, I don’t think all decades old software necessarily needs to be uprooted and replaced because it’s old, so long as it’s functional. BaRT, I wonder if they’ve upgraded their system software to more modern software? They certainly could add more intelligence (system health, wear, etc.) and begin automating more aspects, so it could make sense to upgrade here, but I’m sure there are lots of places where upgrades are not necessary and the old software is good enough, so long as it’s still supported by a vendor.

blakesterz 8y ago
>> I don’t think all decades old software necessarily needs to be uprooted and replaced because it’s old, so long as it’s functional. I'd say that's especially true in places like power plants. The old stuff is impossible to hack remotely because it's not on any networks. The rush to put everything on the network has made these types of systems vulnerable and places are only now trying to figure out how to secure them.
- deleted 8y ago
  [deleted]
NullPrefix 8y ago
>add more intelligence (system health, wear, etc.) Of course, telemetry is the first thing that comes to mind about modern software.
jdavis703 8y ago
Which old software are we talking about? Is it the Clipper Card system that has tens of thousands of business rules that prevent transit agendcies from rolling out new passes and pricing strategies? Or are we taking about the software that runs the trains and currently causes a bottle neck through the transbay tube? Not everything needs to be rewritten in React using Google’s Material UI design standards, but my god, our public software really sucks. Even as the article talked about the tax assessors office... I know someone who bought a condo, but SF can’t collect property tax because their office is so backlogged. So instead they told the owner to make sure they save the money because in a couple of years they’re going to come asking for it.
- wbl 8y ago
  The software is not the issue with the tube. It's the fixed block system imposing minimum 1 minute headways.
- derwiki 8y ago
  > SF can’t collect property tax because their office is so backlogged IME they're able to collect property tax, no problem.
  - gph 8y ago
    Clearly the truth lies somewhere in between both these statements. Something like: "SF can't collect the full and accurate amount of property tax in a timely manner from properties that were recently reassessed because their local assessor office is backlogged due to use of old and inefficient software." Which is basically what this whole article is saying.
- danepowell 8y ago
  The Clipper Card system _must_ fall into this category. I recently called because I was certain my card was being fraudulently used--the value would dip $5-10 for a single ride, and there were top-up credit card transactions that didn't correspond even remotely to my credit card statement. Turns out SF Muni and Clipper Card (and possibly Clipper Card's billing vendor?) are two/three completely independent organizations. Everything is passed between them in batches with up to a several-month delay, with batches often arriving out of order. This makes auditing nearly impossible and is what made me think something sketchy was happening. It's a miracle it works at all.
- inferiorhuman 8y ago
  Is it the Clipper Card system that has tens of thousands of business rules that prevent transit agendcies from rolling out new passes and pricing strategies You're putting the cart before the horse there. It's the transit agencies themselves that advocate for the lack of prices. BART, for instance, held out on implementing TransLink/Clipper for ages because they wanted their own BART-only cash purse on the card. Clipper is still fairly new and I don't think the fare rules have changed much, if at all, since its inception.

inetknght 8y ago

> Assessors are prone to make mistakes when using the vintage software because it can’t display all the basic information for a given property on one screen. Well they're in for a surprise if they think modern user experience designers would fix that.

rietta 8y ago
Geez. This is the truth. Applying A/B tested mobile design fads from consumer app to data driven for-professional-use software is leading us in a very bad direction.
- jjoonathan 8y ago
  Replace half of the labeled buttons with inscrutable generic hieroglyphs and hide the other half under a hamburger menu! <Aliens Guy>User Experience</Aliens Guy>
- gambler 8y ago
  Very true. Modern web UIs have some good ideas and some bad ideas, but fundamentally they are designed for people browsing stuff and optimized for conversion and engagement. At work, people do work and you need to optimize for efficiency and reducing burnout. A lot of people don't get this and blindly copy designs. Yay, your tool now looks like Amazon! ...which people leisurely use two hours a week, while your tool is being used 9 hours a day, in a crunch.
doodpants 8y ago
Agreed. Perhaps this software from the '80s should be replaced by software with interface design sensibilities from the '90s.
smt88 8y ago
I've found some great UX designers who could fix this. They complain that they're constantly at war with designers, who think things like text labels are "ugly".
- 8y ago

deanalevitt 8y ago

icedchai 8y ago
Insane. That's like 100 people for 10 years (ballpark...)
- vonmoltke 8y ago
  That's some awful cheap people.
  - icedchai 8y ago
    100K/year average. Consider they are not all highly paid engineers, Minnesota is like 1/3rd the cost of the bay area, a lot of it was outsourced overseas where they're even cheaper.
kiba 8y ago
Raise what questions? It seems par the course for many software projects.
- deanalevitt 8y ago
  Well, my questions are: What was the original budget? Were there cost overruns? Who got the tender, and why? Where was the complexity that made it a $100M, decade long project?
- imtringued 8y ago
  I don't see how cost overruns by more than an order of magnitude are par for the course.
  - 8y ago

thisisweirdok 8y ago

There are multi-billion dollar companies that still need IE11 because they don't want to build new internal applications. Contrasted to the funding that some cities get, it's amazing they have computers at all. There's this huge fervor over disruption and startup dollars, but just like the rest of the country... municipal software infrastructure is just sitting around full of holes.

chobytes 8y ago

Im not sure that I think software is always bad just because its old. I recall working as a technician years ago we used text user interface ticket system from the early 90s. While it wasnt the prettiest system it did its job and one could navigate it quickly once they has commited the keystrokes to muscle memory. We thought about replacing it with more modern software but after evaluating a few of the options found none of them to be particularly compelling. If anything, we appreciated what our “old” system could do even more.

brobdingnagians 8y ago
I'm constantly and pleasantly surprised by how nice and feature-ful text utilities are. First time I used htop, I was much happier than most of the GUI task managers I had used. It seems like having a simple user interface helps the designers focus more on having it actually do more of the useful things than on how many pixels to put on that margin...
- rayiner 8y ago
  You can see this first hand in almost every single Microsoft program that’s undergone a Win32 to Metro/UWP transition. The new versions can’t do anything.
  - maxxxxx 8y ago
    So true. It’s hard to believe that their new control panels since Win 8 still lack features of the old ones.
  - kalleboo 8y ago
    See also: any mobile anything. Many times a week do I have to switch to the desktop site to get things done since it wouldn't fit into the clean mobile design.
  - AnIdiotOnTheNet 8y ago
    And they do all that nothing so much slower than the old ones. Everyone involved with that project should be ashamed.

redsavagefiero 8y ago

Good. We don't have to worry about node.js, apache whatsit and Google go-away v1.0 being our goto technologies in a critical space. That's progress. Stick a rest/rpc interface on the outputs and ctl and attach a wireless dongle to a middlebox: voila modernized! If you really want to get fancy attach ELK to the 'stack' and put what data you want in it to represent something or another. Mission accomplished!

derwiki 8y ago
This is sorta what my university did when they launched online registration. And that was why only 10 students at a time could use the online registration system-- that was the number of virtual terminals the mainframe had. And it was a Java applet :weary:

tomatotomato37 8y ago

And? I'd rather civic data being handled on some ancient console that's relatively bugfree than some modern webapp nonsense that loses data half the time but has pretty hamburger menus.

kilon 8y ago

If the sofware is from the 80s that is very good news, it means it runs on simpler hardware and the code will be simpler and easier to fix. Which is what made it viable for us to code in Assembly back in the 80s and early 90s with the same ease someone codes in c# nowadays. Coding in Assembly now is pure insanity. Sure you could still write super ugly code but super ugly simple code is far better than super ugly complex code. Even if it comes with a forest of gotos.

lqet 8y ago

So? I remember a large bookshop in my hometown in which each section of the store had a terminal with DOS-like interface to their internal database. Employees were able to use it blazingly fast, and they still used it a few years ago. Except for security concerns, why should newer software automatically be better?

rightbyte 8y ago
Old crud apps ware probably the prime of usability. These hotel booking systems with green text terminals and hot keys for everything. The video store where I grew up had a similar text based interface to some floppy located database for rentals. Amazingly fast and some override button to force invalid entries when someone else had messed up or something was registered wrong. Binaries run straight from a floppy with no fancy OS to interrupt the very simple yet manually tedious things it does. Many apps today are way over cooked for what they do.

snarfy 8y ago

> Issue: Control systems for most of the city’s 3,000 traffic signals date to the 1960s and make it harder for the department to manage congestion. Cost (per intersection): $175,000 to $735,000 That cost seems outrageous. Are traffic signal control boxes really that expensive?

clarkmoody 8y ago
It's a government expenditure, so the market signals of profit and loss cannot work to keep costs under control. Classic monopoly pricing at play. The outrageous prices could also be attributed to good old-fashioned cronyism: some city council member's brother owns a local civil engineering firm that gets a 10-year contract and names its own price.
- snowwindwaves 8y ago
  Have you ever worked for a government or public institution? Everything takes 5x longer than it should.
syn0byte 8y ago
In one case I've heard around the water cooler; A city's intersections were(are?) rocking Sun SPARC servers to do all the control stuff. Yes, each intersection apparently had its own. I don't know if I ever really believed it until this article confirmed that kind of cost.
mymythisisthis 8y ago
Upgrading traffic systems will help with self-driving cars. It's one of the missing puzzle pieces. Think of what modern cameras and data storage, would do for innovative change.
mattkrause 8y ago
I'm also having a hard time believing that Philly has half a billion dollars worth of traffic signals.... Unit costs can be misleading if they're hiding a lot of non-recurring engineering costs. I can easily believe it would cost $175,000 (or more) to replace a single set of traffic signals, especially if they are built from scratch and need to interface with some ancient system. However, the 2nd, 3rd, .... 3,000th signals ought to be progressively cheaper as things get worked out.

buboard 8y ago

Oh well, good for them, have you seen the software they make nowadays?

oliwarner 8y ago

Things like this obviously attract headlines, but the real problem comes from the millions of other machines, decades newer than the ones in TFA, that are still irredeemably EOL. I contract for a few companies that supply local governments in the UK. I regularly have to fill out stringent checklists about IT security provisions. PCI-DSS and beyond. So I always chuckle when we meet them on-site and they're running an XP install with IE8, plastered with toolbars. These are machines I would sooner throw into the abyss than try to rescue. They'll make me jump through hoops but there they are inputting and manipulating citizen data on machines that are almost certainly part of a botnet. Tablets are also increasingly a problem. As a b2b-service webdeb, I still see a lot of first gen iPad Minis in active service in enterprise. Last patched mid-2016. Oh well. The next big worm attack is going to be devastating to organisations like this.

at-fates-hands 8y ago
> Tablets are also increasingly a problem. As a b2b-service webdeb, I still see a lot of first gen iPad Minis in active service in enterprise. Last patched mid-2016. I used to work for a large oil company. Their IT guys were pretty adamant about security. Right before I left the company, they purchased another smaller distributor. All their guys had ipad minis and same thing, were last patched in 2016. The IT guys right away were telling the execs, "Yeah, they're all getting Win10 laptops, fully patched and locked down to use only our software. No way we're going to keep using these."

gdulli 8y ago

Meanwhile, my modern engineering organization has projects less than three years old that are already obsolete or orphaned because they were built on someone's favorite JS framework or CMS that isn't relevant or growing anymore and no one wants to touch them now.

missblit 8y ago

> the shift from old-school servers to rented cloud storage has made it tougher than ever to fund upgrades They do know that it's still possible to run old-school servers, right? There must be some upside or need to switch to cloud computing, either that or people just get blinded by buzzwords. Hard to say from the article. Anyone know why governments may switch to cloud despite the issues in the article?

gambler 8y ago
I don't know about governments, by at my last job the transition to cloud was driven mostly by: 1. The desire of some people to have AWS stuff on their resumes. 2. The optimistic belief that the same people who couldn't figure out how to manage normal servers in 10+ years will figure out how to design good serverless apps in a few month.
coldacid 8y ago
Because they are very much blinded by buzzwords, and want to look like they're "with it" by choosing whatever seems the most advanced even if it would actually detract from the job at hand.
linuxftw 8y ago
Governments are mostly administrative. Running your own servers is 'real work' which mostly gets contracted out. Used to be necessary to have your own servers because that's where the software your administrative people used ran. The government could build it's own power plants, and hire the employees to run them. Probably cheaper/easier to contract that out (eg, buy power from power company).

quotha 8y ago

if it ain't broke..

AdmiralAsshat 8y ago

I work on several pieces of business software at my job. One was built for UNIX mainframes, almost entirely in C, with the end-users still capable of running on dumb terminals (or these days a PuTTY window). The "successor" runs on Windows servers with J2EE, a GUI, and all the modern niceties. The most common complaint I hear from people who've switched from the legacy software to the modern one is that their end-users' productivity is greatly reduced by the point-and-click interface. The older software's TUI had a very steep learning curve, but almost like learning Vim, once they developed a muscle memory for the necessary keystrokes, they could access and update information much more quickly.

forinti 8y ago
I once saw an attempt at modernising a mainframe application that had a single input line where the user would type a sequence of numbers. The new GUI had all the fields organised and labeled. The user then requested and got a single input box where he would type in all the numbers just the way he did before.

deleted 8y ago

[deleted]

bigpicture 8y ago

A few years ago I had to update some FORTRAN code written in the 1980s that did bond valuations and some other financial stuff. It hadn't been touched since it was written and was now running out of space because the number of bonds keeps increasing year after year. I'd never done anything with FORTRAN before, so I spent a couple days reading about it and looking into how it works, fearful that I'd muck everything up trying to fix it. I open the code and find that it was the most elegantly written software you could wish for, with fantastic comments and structure. I changed two numbers and was done, buying us another 30 years or so of rock-solid service.

drharby 8y ago
I hope to god you over sold yourself
systemtest 8y ago
Did you leave a comment for the guy who has to fix it in 30 years?
- pinko 8y ago
  Hopefully there's even odds it will be a woman by then.
  - systemtest 8y ago
    Perhaps we go full circle and have predominately female programmers by then.
    - edoceo 8y ago
      A full circle puts us back where we started?
      - mrguyorama 8y ago
        Programming used to be a "woman's job" as secretaries.

JoeAltmaier 8y ago

America's industry is running on software from the '80s. You can't imagine the horrors on the shop floor. Never mind XP or Windows ME - DOS with green-screen terminals; hand-coded apps booting on home-brew hardware; IBM dinosaurs that should be extinct but lumber on in the back office. And there's nobody that can help them fix anything. So upgrade you say? Then they'd have to start from scratch. And they long ago laid off the accounting experts, tax lawyers and business planners. They have button-pushers and data entry folks. Nobody left who knows were the data comes from, where it goes, or who all uses it.

bluedino 8y ago
We recently bought a $4 million dollar steel fabrication system, automated, top of the line, runs Windows XP. Rumor is the company has been trying to re-write the control software to run on Windows 7 for the last 10 years (name rhymes with spaghetti) We interact with it using little data packets (ASCII, sent from a VB app) of coordinates to tell it what to do. Insane amounts of money would be lost if that machine goes offline for a day. We can't just replace it unless it's 100% compatible. Some of our bigger competitors have 10 of these machines. There's no real benefit of upgrading them, though. We'd still run the whole damn machine behind it's own ASA and on it's own network. The users don't care. And our programmers wouldn't be happy to change it.
- southern_cross 8y ago
  I regularly see stories of software that is at least 20 years old now, running multimillion dollar industrial systems that are expected to last for at least another 20 years before being retired (if even then). There will probably be no changes to that software in the interim, and it's possible that it can't be changed even if you wanted to, for various reasons. As long as the computer is well-isolated, the biggest ongoing risk here is replacing old and failing PC components when necessary. But the same might be also be true of the industrial systems themselves.
reaperducer 8y ago
You can't imagine the horrors on the shop floor OK... DOS with green-screen terminals Not a horror. If it works, why mess with it? hand-coded apps booting on home-brew hardware Also not a horror. Bespoke hardware is everywhere, as are hand-coded apps. AI does not yet build software. IBM dinosaurs Still not a horror. Perhaps you just don't have a lot of experience with legacy systems.

blunte 8y ago

This is like everything else in life: there is the initial cost, there are maintenance and support costs, and there are replacement costs. With roads, it's fairly obvious when maintenance or replacement is required, so people responsible for financial expenditures cannot ignore them. But with software, the "invisible" nature of it seems to allow those responsible to ignore reality and kick the problem down the road until hopefully it is someone else's problem. (Incidentally, this is how a lot of modern public companies with with respect to 5-year CEOs and quarterly earnings and actual responsibility to shareholders, employees, and public). Anyone who has technical experience knows that it's a tough sell to convince upper management to spend money on problems that they can't even see. The big question is, when is the technical debt (or outright catastrophic failure of a decayed system) worth more loss in revenue than its replacement would cost?

mdesq 8y ago

A few years ago, I met guy in my travels who seemed quite carefree and just enjoying frequent international travel with his spouse, funding various art initiatives, etc. Turns out he wrote some utility-related software for a large American city back in the 80s and has nice a recurring revenue stream from that. Other than some now very occasional support requirements, it's essentially just a source of passive income. He said switching costs would be too great, and there's no need to change something that's working.

WhompingWindows 8y ago

I work in the federal government and the performance of the tech is abysmal. If all of the ridiculously talented developers in SV who are using algorithms to capture our attention via psychological hooks, instead used their talents to improve our government tech infrastructure, the world would be a far better place.

magduf 8y ago
The problem there is that the government doesn't want to hire those ridiculously talented developers in SV to do that improvement. The government process for procuring software is totally broken and hopelessly inefficient, and when the government does try to do it on its own, they can't hire decent developers because they absolutely refuse to pay industry-scale wages. Somehow they think that the "benefits" of a Federal job are competitive with what industry is paying, but I guess they haven't been watching the news where Federal workers have to constantly worry about getting furloughed due to government shutdowns.
- aaya 8y ago
  Agreed. It seems that these institutions are unwilling or unable to pay for talent and that they don't have the capacity--in terms of processes and culture--to leverage that talent even if they were able to acquire it. Here's a relevant comment thread from a post a few weeks ago about Wells Fargo going offline: https://news.ycombinator.com/item?id=19110672 https://news.ycombinator.com/item?id=19110672

sixdimensional 8y ago

I have one phrase to add to this discussion - "sustainability"[1]. Systems have to be engineered differently with that in mind. I have real concerns that most of what we are developing in high level languages/systems (with dependency management dependent on external sources) will not last. [1] https://simple.m.wikipedia.org/wiki/Sustainability https://simple.m.wikipedia.org/wiki/Sustainability

tsmarsh 8y ago

Says the company that runs on the Bloomberg Terminal.

smhenderson 8y ago
I've never seen one but I've read that they're actually quite user friendly and provide a wealth of information. Was what I read over rated? Any examples of why they're not so great?
- richjdsmith 8y ago
  I don't know what they're going on about. BT is fantastic.
- mruts 8y ago
  The bloomberg terminal is pretty awesome. It's written in several different languages though, so maybe that's the GP's problem?
- tsmarsh 8y ago
  They are, but they're basically 80s/90s technology. I work in emacs with lisps (mostly clojure) on x86, usually via a terminal emulator. Old tech is stable, fast and cheap. I know that, you know that, Bloomberg make their billions off of it... but they're calling out Cities for it. Just weird.

dev_dull 8y ago

> Minnesota spent about a decade and $100 million to replace its ancient vehicle-licensing and registration software, but the new version arrived with so many glitches in 2017 that Governor Tim Walz has asked for an additional $16 million to fix it. This is exactly why cities don’t replace these old pieces of software. It’s not that a intern couldn’t write something better. It’s that the physical act of “replacing” the old with the new is often very expensive. This is a story that’s unfortunately all too common in government. Other people’s money on other people and whatnot.

ArtWomb 8y ago

Just a heads up. Sat Mar 2, there is a kickoff event for open data week and civic tech in NYC https://beta.nyc/ https://beta.nyc/

wolfspider 8y ago

This is essentially what I have been doing the past few years among other tasks. The difference is we do a lot of things in-house and even receive security audits from companies like IBM and insurance-based development process audits as well (do you have backups of source code, CI, integration testing, etc..). Getting off an AS/400 is not easy as it has a completely different architecture and OS. When it comes to data like this legal assurances have to be in place and voted on before real change happens. We have a financial package which replaces this system and some of it written by our team while the backend and most of the migration handled by an outsourced company. The challenges we faced were handling lots of data going back decades to migrate and doing audits for different departments as the migration was ongoing (parallels). All staff have to be trained up in the new software. All of our new stuff has to be PCI compliant. Many processes based on paperwork have to be turned into an e-filing system- this means heavy OCR workloads to get it into a newer system. A lot of financials end up becoming transparent and visible to the public as a result. If the government were to adopt better practices where technology is concerned it will take interest from the citizens themselves and many additional high level positions focused solely on technology. In this article when it says it took them 4 years to just acquire the funds to start on the project I believe it.

griffinkelly 8y ago

Reminds me of a great story on 60 minutes about the MTA and how they're using signaling systems built in the early 1900s: https://www.cbsnews.com/news/mta-why-has-the-nyc-subway-gone-off-the-rails-60-minutes/ https://www.cbsnews.com/news/mta-why-has-the-nyc-subway-gone...

peatmoss 8y ago

Thought exercise: if you wanted to write code to support a business or nonprofit, and you knew that the software would be used for many years, and probably never see an update, and wouldn’t have anyone to tend network services, how would you build it? Me, I'd probably go with a static executable with a terminal UI and a flat file sqlite database that could be copied by mortals, and hope that whatever I set up as a backup regime would survive future migrations to new hardware.

smacktoward 8y ago
A fascinating idea -- kind of a software analogue to the 10,000-year clock (https://en.wikipedia.org/wiki/Clock_of_the_Long_Now https://en.wikipedia.org/wiki/Clock_of_the_Long_Now).

evadne 8y ago

I would encourage thinking about this through the lens of survivor bias. If said pieces of software were so bad, they would have been replaced. So, the business utility delivered by these pieces of software surely exceed the cost of maintenance and upkeep.

aeturnum 8y ago

sigh The buildings were built in the '80s too (sometimes even earlier!) and it's fine. Software is no longer new. If there are problems, they don't come from the age of the constructions.

zamalek 8y ago
Upgrading it would also likely lead to extravagant costs to the taxpayer, as these government software projects tend to do.
sambf 8y ago
The vast majority of Paris buildings were build between 1851 and 1914 [1]. I myself live in a 1903 building. [1] https://www.comeetie.fr/galerie/BatiParis/#12/48.8589/2.3491 https://www.comeetie.fr/galerie/BatiParis/#12/48.8589/2.3491

clavalle 8y ago

Legacy code is everywhere. What slows down adoption of new software in these areas isn't generally the system being replaced itself, but the requirement that the new software interface with some other 'critical' legacy system(s). It is an ossified web of business processes and tools and their dependencies.

nobleach 8y ago
The biggest problem I ran into (while supporting these old systems based on DOS and Netware 3.1) was keeping them inline with newer tech. I had a client (an accountant) with several old DOS based apps. Getting their network of printers to work reliably was a crapshoot. I'd get calls all the time that some program just stopped printing. At least Netware stayed reliable.

llacb47 8y ago

I'm glad that my county has modernized.

failrate 8y ago

Let's be open: software hasn't become fundamentally better even with better tools and languages. Quality software is more a product of the person or team that created it than the available bells and whistles.

frosted-flakes 8y ago
Maybe not higher quality, but certainly more capabilities.
- failrate 8y ago
  I yield that my potential high point in terms of scale and complexity are objectively higher, but I feel like we are solving the same problems over and again with various permutations of bandaids and baling wire.

gtycomb 8y ago

You are looking at one of the great lost opportunites for hacker enterprenuers. Cities/state agencies depend on cio-purchasing managers to help maintain their IT, and this ends up in third-party handshakes i.e jobs run by very specialised outsourcing outfits who know how to market here. It takes a different type of management mindset to turn this around but things are entrenched firmly for change. The agencies will keep getting the '80s technology with no end in sight.

discreditable 8y ago

This article makes me wonder what nightmares await those who have to maintain today's software at 30 years old.

AnIdiotOnTheNet 8y ago
It'd have to be valuable enough to still be used in 30 years.

moorhosj 8y ago

Entire country running on Constitution from 1700s.

dragonwriter 8y ago
Actually, the last update was in 1992, which is pretty far from the 1700s. (Admittedly, the update that was merged in 1992 was submitted in 1789.)
- Aloha 8y ago
  Talk about protracted development ;-)

djohnston 8y ago

that probably means it's well written code.

ahallock 8y ago

Is software from the 80s necessarily bad? I think it's a fallacy to assume age equates to poor quality, but you see this scare tactic used in media quite a bit.

coldacid 8y ago
It's not bad, but requirements and usage methods have changed, and the software hasn't. It's fair to update or replace software in such a situation, so long as the new/updated system meets the actual needs of its users as they are today.

taf2 8y ago

AM I the only one that thinks this is totally awesome. If the code I write today is still in use in 40 years that would feel good.

flatfilefan 8y ago
Especially when it is so good that it looks like a black box to the word: correct input -> correct output; and nothing more.

AngeloAnolin 8y ago

I think this stems from the fact that IT in government is still treated as a cost rather than an integral part of the business. Generally, software projects will require hundred of hours of vetting, subject matter expert analysis, political will and pressure before they even get past the approval stage. And by the time the project is approved, the process which the project was intended to modernize may have already adopted a different set of methods thereby setting the project for failure (by not meeting what was needed).

Johnny555 8y ago

I think this is part of the problem: "We’re dealing with an irrational public who wants greater and greater service delivery at the same time they want their taxes to be lower," There's noting irrational about wanting government to take advantage of the same technological advancements that businesses use to improve service while cutting costs. When I order from Amazon, I see real-time inventory, get immediate order confirmation, near real-time tracking as the package is handled (and actual real-time tracking for some orders were I can see where the delivery vehicle is), and delivery to my door step is 2 days (or sometimes the same day). In contrast, when I ordered a dog license from my county, I had to mail (FAX was also accepted) a paper form, and then 6 weeks later I received a metal plate in the mail (just a numeric ID on it, no personalization). The only way I knew they even received my order is because my check cleared the bank 2 weeks after I mailed it (no credit cards allowed). I don't expect my county animal services department to build an order an fulfillment system that rivals Amazon's, but there are thousands of this offices across the country, they could all cooperate on one system and save money dealing wit these paper forms.

Aloha 8y ago
If the paper form costs them less to process than whatever automated system that would provide whatever quality of service you deem acceptable would, why would they switch. Newer and more automated doesnt always save money. I work for a medium size company, our expense reporting system is 100% paper or excel driven - we could go to Concur, but it would probably cost more than our existing system does, and may not pay for itself within the lifetime of the product.
dajohnson89 8y ago
If only the solution were higher taxes. Local & federal government alike are comically wasteful of taxpayer money. I have no problem with paying higher taxes per se, but the return on our investment right now is abysmal. Something has to be done to make governments more efficient. I don't know what that is.
flatfilefan 8y ago
"There's noting irrational about wanting government to take advantage of the same technological advancements that businesses use to improve service while cutting costs." - the irrational thing is to expect this in absence of competition. How about having two or more local governments acting at the same time to enable the true competition?

ganonm 8y ago

Maths is running on 'software' from 500BC. Don't see anyone complaining.

boznz 8y ago

Always look at your dependencies and in that I include the programming language, the os and the hardware. I have seen good systems fail for supporting the wrong label printer.

tomc1985 8y ago

Dear people, can we please stop freaking out about old software?

intrasight 8y ago

My first job in the late 80s was programming nuclear power plants. It was a good learning experience. I bet that software is still in place too ;)

southern_cross 8y ago
IIRC, the original software pretty much has to run as-is until the day the nuclear power plant gets retired - typically 40 years, at least.
- flatfilefan 8y ago
  What about the bit rot, decay, half life and all that?
  - southern_cross 8y ago
    As I understand it, the software itself never really gets changed, because if it did then the whole plant might have to undergo testing and regulatory approval again. But the reactors get shut down periodically (every three years, maybe) for refueling, so the control hardware undergoes routine maintenance at that time. Or something like that.

gambler 8y ago

Almost every week, I wish the framework I am working right now was written in Smalltalk 80 rather than Java.

rc_hadoken 8y ago

Things that last a long time tend to keep lasting a long time. I'd bet the code is stellar. New things have a higher risk of what I call 'non-survival'. A concept I've learned from Taleb.

gambler 8y ago

It's a mistake to think that just because something is written in 2019, it automatically benefits from the extra 40 years of collective experience. The collective has Korsakoff syndrome and aversion to history. To be fair, there are two general kinds of legacy systems. Ones that work so well it makes no economic sense to replace them and ones that are such a giant, horrible clusterfuck that they warped the entire organization around their own idiosyncrasies. In the second case, replacing the system requires replacing most business processes as well, so it's very hard.

tyre 8y ago

This is not a software problem. It's a purchasing problem. We built better software for cities and ran into this over and over and over again. They're not incentivized to change. Citizens don't care or know enough to raise hell about city operations. The employees who do the work can't advocate for themselves and/or feel no reason to risk their jobs to make their own lives better (there's no reason to be more efficient.) Leadership cares mostly about press releases—blockchain or AI or whatever buzzword sounds better to them than the incremental changes that will actually contribute to fixing cities. It's frustrating, bleak, and the inverse of inspiring. I tried for 2.5 years before losing hope. My co-founder carried on another year before running into the same. The best way to get people what they need is to bribe them—e.g. work through an intermediary who takes a cut and delivers the rest to a political campaign—which we refused to do. Local governments affect most Americans' day-to-day far more than the national political shitshow and this only becomes more true as populations further concentrate in cities. And yet, they are utterly soul-sucking morasses of internal and external politics, misaligned incentives, and bureaucracy. They drive out the people who can and genuinely want to effect change. It's a damn shame.

pergadad 8y ago
For public sector it's inherently risky to buy software and even devices. They often lack the capacity to properly outline what they need, to understand what's possible, to know the real cost/price they'd need to offer to get good results for tenders, and most importantly lack the ability to write for technical tenders to get what they need. The effect of that if something is running well often the most rational solution is to leave it running. Yes it might not be the most efficient solution, but it might still be more cost effective than losing 3 months of work due to conversion to a new system, in addition to all the extra cost, training, downtime, ... In the current ecosystem more and more also seems to be license based. Do you really want to change a decently running system with just occasional maintenance cost for an untested system that will change all existing processes and in addition ups your annual cost by quite a bit? Any time you change a system you probably have to change some additional systems too as suddenly things don't work together anymore.... Cities are often stretched for resources ranging from (good) personnel to financial to physical space to ... It's probably in many cases the rational decision to stick with how it is if the system mostly works.

stcredzero 8y ago

For local officials throughout the country, the shift from old-school servers to rented cloud storage has made it tougher than ever to fund upgrades. They can budget physical equipment as capital expenses, meaning they could issue bonds to pay for them. But cloud computing is a service, as the people selling it love to say, which means officials have to pay for it with operating funds Could a company structure access to cloud computing over a fixed period of time as a capital expense? What if there was some sort of on-site access device involved?

StreamBright 8y ago

Ain't broken, don't fix it.

rmason 8y ago

I've publicly advocated a new development model to both the state and cities here in Michigan. Share the cost of development and make the software open source. Unique differences such as laws and tax rates should be in proprietary plugins. Makes zero sense to me that say fifty states pay a vendor to create software for a new federal law. The state's pay for one off software but the vendor writes a single program that's either database driven or uses custom functions to reflect stuff unique to the state. It's a colossal waste of money. I can't get the government people to tell me why I am wrong. They just seemed locked in to doing things the way they've always done them. Hey, where's that 8 inch floppy again, I've got work to do.

robaato 8y ago
Entrenched business models/relationships/incentives no doubt at work... Rather like ancient guilds and raising the bar of entry to newcomers who might otherwise undercut them.

bungie4 8y ago

Nothing is easy anymore. The stacks are so deep with millions of lines of code. When it works, its amazing. But it seems not much just 'works' anymore. Compared to the 80-90's. You were very close to the metal. Comparatively speaking.

robaato 8y ago

Slightly off topic, my first significant job in '80s was programming a Stratus in their PL/1 implementation. As I recall it was really quite a nice mix of some COBOL record structures with more modern language options. Shame it didn't last! The Stratus itself was well engineered but missed their opportunity in the market subsequently. Moved on to writing C for an RS/6000 and doing graphics work (few low level libraries around then). Interesting times...

fmela 8y ago

Oh, that's not so bad. American nuclear missiles are running software from the 70s, loaded via 8-inch floppy disks. https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nuclear/index.html https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nucl...

andrevergamito 8y ago

I work for a billion-dollar-company, in the semiconductor industry and yet we use a 1986 asset tracking software we access through telnet and despite the 7M USD effort in three year for the development of newer software we ended up reverting to the old software. Non to mention we still use Lotus Notes for email and windows NT on some production machines. The same machines that validates a good 30percent of electrical components on electric devices and cars...

turc1656 8y ago

This is the problem right here - "It took her office almost four years to secure $36 million for updated assessors’ hardware and software that can, among other things, give priority to cases in which delays may prove costly." Why the hell does it cost $36 million for that? It's typical bloat/waste/padding as far as I'm concerned. Just another municipality that pisses taxpayer money away. I guarantee you that a private business could have made that for a fraction of the price.

microcolonel 8y ago

If it ain't broke, don't fix it. If it is broke, do fix it.