16 ms·

IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

156 points by artsandsci 8y ago

olivermarks 8y ago
'The Individual Master File contains data from 1 billion taxpayer accounts dating back several decades and is the chief IRS application responsible for receiving 100 million Americans’ individual taxpayer data and dispensing refunds. ' I hope they bought a new hard drive to back it up...
- mark212 8y ago
  don't know about you, but I'm hoping they didn't.
ams6110 8y ago
> Since Republicans gained control of the House of Representatives in 2010, their partisan attacks have left the IRS with nearly 10,000 fewer customer service representatives to assist taxpayers and a patchwork of IT systems, some dating back to the Kennedy Administration, which is ultimately harming all taxpayers,” Rep. Gerry Connolly, D-Va., told Nextgov. So the patchwork 1960s-era systems are the fault of Republicans in the current decade? Maybe "starve the beast" really is a better approach. Nothing else has worked.
- TallGuyShort 8y ago
  Their customer service reps sucked long before 2010 and have never remotely "assisted" me.
  - hcknwscommenter 8y ago
    My experience differs from yours. I have used IRS customer service five different times over the last two decades and each time they have been courteous, efficient, and helpful.
- wvenable 8y ago
  https://federalnewsradio.com/tom-temin-commentary/2018/01/irs-clutches-its-modernization-holy-grail/ https://federalnewsradio.com/tom-temin-commentary/2018/01/ir...
- SiempreViernes 8y ago
  If you look again, you will see the charge is that there are now 10k fewer first line support workers who previously could help citizens and (for some reason) the legacy hardware.
- briandear 8y ago
  The real question is why we have a tax system that would need 10,000 additional customer support reps. The problem isn’t a lack of reps, it’s a system that requires that many reps. The flat tax, file-on-a-postcard system proposed by 2016 presidential candidate Ted Cruz was mocked by the establishment because it seemed tha politicians on both sides have a vested interest in a complex tax code — it is the means by which politicians derive their power. See Milton Freidman on the subject: https://youtu.be/TruCIPy79w8 https://youtu.be/TruCIPy79w8
  - hcknwscommenter 8y ago
    File on a postcard. Sure. This year, I could have used the 1040EZ (the entirety of which could fit on a post card if you removed the embedded instructions), and left some money on the table, so I chose the more complicated route. Last year, I just used the 1040EZ. Flat tax is a completely different issue.
  - SiempreViernes 8y ago
    It is weird to see the argument that making tax filingd hard builds power, for it to work you have to dismiss the inuitive idead that making tax filing simpler will make citizens happy and less likely to complain since they have beens spared from some hassle.
  - monocasa 8y ago
    You can stay away from regressive schemes like a flat tax, and have a tax system with a rubberstamp level of UX. That's how it works in most of Europe and could work here. The IRS already has all of the information needed outside of gambling and stocks, and could simply send you forms already filled out for you to sign. H&R Block and TurboTax simply heavily lobby to keep this from happening to keep their business model around.
    - conanbatt 8y ago
      Whats interesting about the flat taxes is that it makes it hard to avoid at the highest levels, where complex tax structures end up having a very variant level of taxation. Much like corporate taxation is 35%, but google or facebook pay only 11%. Milton Friedman used to say that a flat tax of 15% would be enough to replace the entire scheme of income tax. There is also one major boon: lots of capital is spent on taxes, from accountants to lawyers to gubernamental organizations, etc etc. An effective 15% rate without exemptions would fly very well with the right (simple flat taxes) an the left (no exemptions) but as MF said, the political stances on this topic dont trust each other and they are both right: even if we moved to such a system, the right would still advocate for exemptions and the left for increases in taxation. Its a noticeable deadlock. I also want to mention that flat taxes are not regressive by definition. The progressiveness or regressiveness in that case requires to see where the taxes are spent.
- hcknwscommenter 8y ago
  >So the patchwork 1960s-era systems You have (purposely?) mis-stated the previous statement quoted above. A patchwork of systems that includes some systems dating back to the Kennedy Administration is not the same as a patchwork of 1960'era systems. Could it be that the "starve the best" mentality hindered or blocked a number of planned migrations/upgrades?
- hindsightbias 8y ago
  I worked next to a team in the late 80's where one service center prototyped a wage reporting program that brought in >$150m the first year. They asked for $2M to expand it nationwide. Refused until Clinton came in. It's easy to throw stones but I doubt anything built today by anyone on HN will rival problems as complex as DEWS, ATC, IMF/BMF or Sabre and be around in ten years, let alone 60.
49bc 8y ago
When they can write AI to understand and implement the entire US tax system, then I’ll start worrying about the incoming AI apocalypse they’re always taking about.
- SketchySeaBeast 8y ago
  That's assuming the poor thing doesn't choose to immediately self terminate.
- noetic_techy 8y ago
  More than likely, it will shoot back a question: "Why the hell did you make this so complex?"
  - colemannugent 8y ago
    "A strange game. The only winning move is not to file"
    - deleted 8y ago
      [deleted]
    - Nelson69 8y ago
      If only it were that simple... Not filing or, worse, making an error in your filing makes it a really really nasty game. No recommended. I'll even break my own rule: "Not filing considered harmful"
  - thescriptkiddie 8y ago
    So that every congressman can give their favorite d̶o̶n̶o̶r̶s̶ constituents their very own tax loophole.
  - DrScump 8y ago
    Answer: "42."
favorited 8y ago
Fun (scary) fact – significant portions of the IRS tax system is still written in assembly. It's basically the original codebase plus 40 years of additions and patches as the tax code changed. They were working on projects to "elevate" it to Java, because it's pretty difficult to maintain it as-is.
- deleted 8y ago
  [deleted]
- LinuxBender 8y ago
  How are they retaining staff with the skill-set to maintain that code?
  - technologia 8y ago
    Its tough to recruit for, so you really end up with folks who have job security for the forseeable future and others who become retired-in-place because of the rarity of cleared folks eligible to take on the post.
  - bsder 8y ago
    They don't retain staff. But the ones who retire or leave can generally get a very large contract when things like this pop up. Until, of course, they are all gone. At which point the system will need to be rewritten by IBM, Oracle or some other useless, gigantic consulting firm.
- gascan 8y ago
  I mean, maintainability aside they are a giant calculator right? You don't need to be running the latest, hippest web platform to do a good job at doing a lot of math. And, will any of the current en-vogue tech still be working in 50 years? If you had to pick, Java does make a lot of sense. Java & C are safe bets for "languages that will stick around for decades".
  - inlined 8y ago
    I imagine processing the data, deciding whom to audit, and tracking the process is more complicated. Heck; they're probably using ML for audit red flags
otoburb 8y ago
>>Still, the crash forced the IRS to extend the tax filing deadline one day, delaying some 14 million submissions. For a system that is supposedly so brittle, they were able to get enough of it back up and running to at least accept submissions within one day? That's pretty amazing for 60 year old technology.
- c22 8y ago
  You just have to get the stuck card unjammed, then repunch it and recollate...
  - thaumaturgy 8y ago
    ...and it was the re-collation step that took a dedicated team of professionals a full day.
    - varjag 8y ago
      To be fair, 140 million taxpayers account for a lot of punchcards.
      - Elv13 8y ago
        (assuming it is not /sarcasm) The data is not stored in punch cards, the programs are/were. The data is/was stored in tapes or in tape emulators. Storing data on cards would make random access wayyyy too slow (the operator would have to retrieve and load the card and keep them sorted manually, which is impossible to do at scale) On tape, however, you can predict the distance you have to rewind by computing the physical length each records take if they have auto incremented IDs. A tape can contain a lot of records. Especially if it's a modern one or an array backed emulated one. At least that's what some people I went to school with told me it's how they did it.
        rbanffy 8y ago
        I did a bit of stuff with data coming in on punchcards. You get a batch of punchcards, run it through a program (which can load data from tape or other punchcards) and a new batch of new punchcards with updated data come out on the other side, along with an impressive pile of printed invoices. Wide adoption of tape was a lifesaver.
eecsninja 8y ago
Be thankful we're not getting all the government we're paying for. --Will Rogers Source: https://www.brainyquote.com/quotes/will_rogers_128073 https://www.brainyquote.com/quotes/will_rogers_128073
johnklos 8y ago
Something that the kids of IT today will never learn: Never replace anything unless the thing doing the replacing is better than the thing being replaced. The IRS' IT "upgrade" is just a huge money grab. Nobody has the slightest interest or desire to do anything properly - they simply want in on that lucrative IRS money. It's like the saying: "Anyone who is capable of getting themselves made President should on no account be allowed to do the job." Likewise, the kind of people who are bidding on this project are certainly not the kind of people that should ever be allowed anywhere near the IRS' computer systems.
- s2g 8y ago
  > Something that the kids of IT today will never learn Why, because you're busy being condescending to teach? Software should be considered a system that needs ongoing care and maintenance, just like anything else.
- zitterbewegung 8y ago
  All programs by the government are ways to employ people. They can be ineffective or effective but it still makes whoever employed happy to vote for whoever made the job up.
- stefan_ 8y ago
  Having this stuff digitized is actually pretty nice. The German ELSTER is basically PDF forms on steroids; same tax forms you presumably used to fill out by hand but with input formatting, validation, basic sanity checks and it estimates what you'll pay / get back at the end of it all. Then you can even submit it online if you have registered for a private key. Of course the tax forms themselves are still insane.
- mbell 8y ago
  Or you can continuously keep your code / systems up to date and never get wedged into a nearly impossible situation of having to rewrite 60 year old code (which likely has nothing resembling a test suite nor even a spare piece of compatible hardware to test on).
  - 8y ago
sunstone 8y ago
Next time they do this they might want to trying testing out the new system before putting it into production.
raverbashing 8y ago
The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago. How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?
- dman 8y ago
  How do you create incentive structures for new employees who have to spend months (years?) coming upto speed with giant state machines before they feel remotely confident of the the effect of their changes? For better or for worse, currently the glory is in writing new systems rather than improving existing ones.
  - DrScump 8y ago
    Unfortunately, it's not just about "glory". If you work in "obsolete" languages/environments, you likely will be perceived as less valuable to your next employer.
    - JustSomeNobody 8y ago
      This right here is a huge problem with our industry. If you're not jumping from employer to employer every 1.7 years, you're labeled as out of date. It takes some really talented developers to truly understand some of these old systems. They should be given the proper respect they deserve.
      - s2g 8y ago
        No, you can stick around. I wouldn't consider someone who spent 5 years on AWS, especially if they moved internally, out of date. Whereas I've done nothing remotely close to where I want to be in about 3 years. I feel very scared about my skills, especially since I'm not even 10 years into my career.
        8y ago
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
cptskippy 8y ago
I feel like the authors of this article are being very misleading by using broader terms where they could be more specific, like using system where they could instead say software. The IRS isn't running 60 year old hardware, and the failure was in hardware that was just 18 months old. But because the IRS is running software that is 60 years old, the authors can claim the "system" dates back to the 1960s. It's also clear the authors don't entirely understand what they're talking about. In the second article linked to, which is written by an author of this article, he states "[t]he Individual Master File, a massive application written in the antiquated and low-level Assembly programming language" as if to imply there is but one assembly language. A GAO report I found chasing the last link in the article indicates that the two oldest systems in the Government are the IMF and Business Master File (BMF) which it says both run on IBM Hardware. My guess would be that the IRS is running modern z/Architecture mainframes in System 360 Compatibility mode. The System 360 dates back to the early 60s and IBM has maintained backward compatibility with it all the way up through z/Architecture systems. This would also explain why they're so far over budget and behind schedule.
- walrus01 8y ago
  the z/whatever to host legacy mainframe applications is basically a bare metal platform, custom hypervisor OS, running system 360 VMs, right?
  - cptskippy 8y ago
    I'm not familiar with all of IBM's z offerings but it was my understanding that older systems were completely emulated and that there is no hardware compatibility. The emulation is completely independent of the underlying hardware platform and it allows them to retain ridiculous uptimes because of all of the hotswappable redundant systems that are able to run the systems concurrently.
- reaperducer 8y ago
  It's general reporting by general reporters using general language for the general public. It's not a tech publication. Joe Lunchbucket doesn't care if it's hardware that failed or software. The system failed. That's what his tax dollars pay for.
js8 8y ago
What kind of hardware was it running on? The article is not very specific about that. (At first I thought that they upgraded IBM mainframe to z14, and that caused the crash, but it's not yet 60 years old.)
- cptskippy 8y ago
  They use very broad terms to imply it was 60 year old hardware that failed but early on in the article state it was an 18 month old piece of hardware in a "system" that dates back to the 60s. If you dig into some of the other links and a GAO report you learn that there's two programs, the IMF and BMF, that are partially written in assembly that dates back to the 60s. They're most likely running some z/Architecture in System 360 virtualization mode.
- cmiles74 8y ago
  While the code is 60 years old, the hardware is modern. According to this article, the component that failed is about 18 months old. When they talk about the software and hardware "no longer being supported", I think they really mean that the original target platforms are only present in the form of emulated software. IBM still supports execution of System/360 software on the System z. [1] "Overall IRS maintains over 20 million lines of assembly code. These millions of lines of archaic software, and hardware, that is no longer supported become more difficult and costly to maintain each year, and poses significant cybersecurity risks. To IRS’ credit it keeps these old systems running during filing season, but relying on these antiquated systems for our nation’s primary source of revenue is highly risky, meaning that the chance of having a failure during the filing season is continually increasing." - David Powner, director of IT management issues for the Government Accountability Office [0] "[IRS Chief Information Officer Gina Garza] explained the code for the IMF was written in the 1960s, but the hardware it runs on is modern." [0] [0] https://federalnewsradio.com/technology-main/2017/10/a-sense-of-urgency-as-irs-legacy-it-systems-grow-increasingly-older/ https://federalnewsradio.com/technology-main/2017/10/a-sense... [1] https://arstechnica.com/information-technology/2014/04/50-years-ago-ibm-created-mainframe-that-helped-bring-men-to-the-moon/ https://arstechnica.com/information-technology/2014/04/50-ye...
deleted 8y ago
[deleted]
jgalt212 8y ago
Can the USDS assist here?
- normaljoe 8y ago
  Last time I talked to somebody at USDS two years ago maybe, they where already helping. I think the biggest task was trying to figure out getting off tape and moving normal hard drive storage. Apparently this is hard because the really old programs are the OS and file system. Without the details it sounds to me this is what broke or something around that.
tomohawk 8y ago
It's probably easier to just scrap the income tax than to fix the system. Maybe go to a land tax instead.
sehugg 8y ago
A delivery date for CADE 2, the IRS’ subsequent modernization effort, has slipped several years even as contractors working on the project have earned as much as $290 million. They call them "Beltway Bandits" for a reason.
- empath75 8y ago
  As someone working on a modernization contract, the problems are more complicated than just contractors standing on shovels. Building an app to a complex and ever shifting spec is a much different process than the way that startups work. You can’t just release a half done app to friends and family. You need to do the whole thing at once, with 100% if the functionality and it has to be ready for everyone in the country to use it in day 1. And the federal government has fairly onerous requirements for getting stuff through security— you can’t just google for random shit from GitHub and expect to put it straight into production. I can only speak to my particular project, but everyone on it is working hard at getting it done on time and on budget, using modern deployment and development practices, but there’s limits to how much you’re allowed to just move fast and break things.
- dx034 8y ago
  I'm always amazed that so many are outraged by Investment Bankers and Lawyers making millions when IT consultants get so little attention. Maybe it's because few people understand what IT consultants actually do and have no alternatives but hiring them but cost overruns due to increased IT consultancy appear much more prevalent than escalating legal costs.
PinkMilkshake 8y ago
So they need something proven, scalable, highly concurrent, fault tolerant, and runs in a VM so it's not tied to hardware. Sounds familiar.
interfixus 8y ago
>[...] date back to 1960, when John F. Kennedy was president No he wasn't, so my confidence starts flagging.
BrandoElFollito 8y ago
What part is complicated when transitioning such a system? Legacy data which must be transformed? Old business logic which must be maintained? I do not know if there is a need to replay old rules vs. just kerping a snapshot Continuity? Maybe there are so many transactions happening, linked to existing data that an interruption is complicated? Risk of accessing old data which is better left undisturbed?

156 points by artsandsci 8y ago

'The Individual Master File contains data from 1 billion taxpayer accounts dating back several decades and is the chief IRS application responsible for receiving 100 million Americans’ individual taxpayer data and dispensing refunds. ' I hope they bought a new hard drive to back it up...

mark212 8y ago
don't know about you, but I'm hoping they didn't.

ams6110 8y ago

> Since Republicans gained control of the House of Representatives in 2010, their partisan attacks have left the IRS with nearly 10,000 fewer customer service representatives to assist taxpayers and a patchwork of IT systems, some dating back to the Kennedy Administration, which is ultimately harming all taxpayers,” Rep. Gerry Connolly, D-Va., told Nextgov. So the patchwork 1960s-era systems are the fault of Republicans in the current decade? Maybe "starve the beast" really is a better approach. Nothing else has worked.

TallGuyShort 8y ago
Their customer service reps sucked long before 2010 and have never remotely "assisted" me.
- hcknwscommenter 8y ago
  My experience differs from yours. I have used IRS customer service five different times over the last two decades and each time they have been courteous, efficient, and helpful.
wvenable 8y ago
https://federalnewsradio.com/tom-temin-commentary/2018/01/irs-clutches-its-modernization-holy-grail/ https://federalnewsradio.com/tom-temin-commentary/2018/01/ir...
SiempreViernes 8y ago
If you look again, you will see the charge is that there are now 10k fewer first line support workers who previously could help citizens and (for some reason) the legacy hardware.
briandear 8y ago
The real question is why we have a tax system that would need 10,000 additional customer support reps. The problem isn’t a lack of reps, it’s a system that requires that many reps. The flat tax, file-on-a-postcard system proposed by 2016 presidential candidate Ted Cruz was mocked by the establishment because it seemed tha politicians on both sides have a vested interest in a complex tax code — it is the means by which politicians derive their power. See Milton Freidman on the subject: https://youtu.be/TruCIPy79w8 https://youtu.be/TruCIPy79w8
- hcknwscommenter 8y ago
  File on a postcard. Sure. This year, I could have used the 1040EZ (the entirety of which could fit on a post card if you removed the embedded instructions), and left some money on the table, so I chose the more complicated route. Last year, I just used the 1040EZ. Flat tax is a completely different issue.
- SiempreViernes 8y ago
  It is weird to see the argument that making tax filingd hard builds power, for it to work you have to dismiss the inuitive idead that making tax filing simpler will make citizens happy and less likely to complain since they have beens spared from some hassle.
- monocasa 8y ago
  You can stay away from regressive schemes like a flat tax, and have a tax system with a rubberstamp level of UX. That's how it works in most of Europe and could work here. The IRS already has all of the information needed outside of gambling and stocks, and could simply send you forms already filled out for you to sign. H&R Block and TurboTax simply heavily lobby to keep this from happening to keep their business model around.
  - conanbatt 8y ago
    Whats interesting about the flat taxes is that it makes it hard to avoid at the highest levels, where complex tax structures end up having a very variant level of taxation. Much like corporate taxation is 35%, but google or facebook pay only 11%. Milton Friedman used to say that a flat tax of 15% would be enough to replace the entire scheme of income tax. There is also one major boon: lots of capital is spent on taxes, from accountants to lawyers to gubernamental organizations, etc etc. An effective 15% rate without exemptions would fly very well with the right (simple flat taxes) an the left (no exemptions) but as MF said, the political stances on this topic dont trust each other and they are both right: even if we moved to such a system, the right would still advocate for exemptions and the left for increases in taxation. Its a noticeable deadlock. I also want to mention that flat taxes are not regressive by definition. The progressiveness or regressiveness in that case requires to see where the taxes are spent.
hcknwscommenter 8y ago
>So the patchwork 1960s-era systems You have (purposely?) mis-stated the previous statement quoted above. A patchwork of systems that includes some systems dating back to the Kennedy Administration is not the same as a patchwork of 1960'era systems. Could it be that the "starve the best" mentality hindered or blocked a number of planned migrations/upgrades?
hindsightbias 8y ago
I worked next to a team in the late 80's where one service center prototyped a wage reporting program that brought in >$150m the first year. They asked for $2M to expand it nationwide. Refused until Clinton came in. It's easy to throw stones but I doubt anything built today by anyone on HN will rival problems as complex as DEWS, ATC, IMF/BMF or Sabre and be around in ten years, let alone 60.

49bc 8y ago

When they can write AI to understand and implement the entire US tax system, then I’ll start worrying about the incoming AI apocalypse they’re always taking about.

SketchySeaBeast 8y ago
That's assuming the poor thing doesn't choose to immediately self terminate.
noetic_techy 8y ago
More than likely, it will shoot back a question: "Why the hell did you make this so complex?"
- colemannugent 8y ago
  "A strange game. The only winning move is not to file"
  - deleted 8y ago
    [deleted]
  - Nelson69 8y ago
    If only it were that simple... Not filing or, worse, making an error in your filing makes it a really really nasty game. No recommended. I'll even break my own rule: "Not filing considered harmful"
- thescriptkiddie 8y ago
  So that every congressman can give their favorite d̶o̶n̶o̶r̶s̶ constituents their very own tax loophole.
- DrScump 8y ago
  Answer: "42."

favorited 8y ago

Fun (scary) fact – significant portions of the IRS tax system is still written in assembly. It's basically the original codebase plus 40 years of additions and patches as the tax code changed. They were working on projects to "elevate" it to Java, because it's pretty difficult to maintain it as-is.

deleted 8y ago
[deleted]
LinuxBender 8y ago
How are they retaining staff with the skill-set to maintain that code?
- technologia 8y ago
  Its tough to recruit for, so you really end up with folks who have job security for the forseeable future and others who become retired-in-place because of the rarity of cleared folks eligible to take on the post.
- bsder 8y ago
  They don't retain staff. But the ones who retire or leave can generally get a very large contract when things like this pop up. Until, of course, they are all gone. At which point the system will need to be rewritten by IBM, Oracle or some other useless, gigantic consulting firm.
gascan 8y ago
I mean, maintainability aside they are a giant calculator right? You don't need to be running the latest, hippest web platform to do a good job at doing a lot of math. And, will any of the current en-vogue tech still be working in 50 years? If you had to pick, Java does make a lot of sense. Java & C are safe bets for "languages that will stick around for decades".
- inlined 8y ago
  I imagine processing the data, deciding whom to audit, and tracking the process is more complicated. Heck; they're probably using ML for audit red flags

otoburb 8y ago

>>Still, the crash forced the IRS to extend the tax filing deadline one day, delaying some 14 million submissions. For a system that is supposedly so brittle, they were able to get enough of it back up and running to at least accept submissions within one day? That's pretty amazing for 60 year old technology.

c22 8y ago
You just have to get the stuck card unjammed, then repunch it and recollate...
- thaumaturgy 8y ago
  ...and it was the re-collation step that took a dedicated team of professionals a full day.
  - varjag 8y ago
    To be fair, 140 million taxpayers account for a lot of punchcards.
    - Elv13 8y ago
      (assuming it is not /sarcasm) The data is not stored in punch cards, the programs are/were. The data is/was stored in tapes or in tape emulators. Storing data on cards would make random access wayyyy too slow (the operator would have to retrieve and load the card and keep them sorted manually, which is impossible to do at scale) On tape, however, you can predict the distance you have to rewind by computing the physical length each records take if they have auto incremented IDs. A tape can contain a lot of records. Especially if it's a modern one or an array backed emulated one. At least that's what some people I went to school with told me it's how they did it.
      - rbanffy 8y ago
        I did a bit of stuff with data coming in on punchcards. You get a batch of punchcards, run it through a program (which can load data from tape or other punchcards) and a new batch of new punchcards with updated data come out on the other side, along with an impressive pile of printed invoices. Wide adoption of tape was a lifesaver.

eecsninja 8y ago

Be thankful we're not getting all the government we're paying for. --Will Rogers Source: https://www.brainyquote.com/quotes/will_rogers_128073 https://www.brainyquote.com/quotes/will_rogers_128073

johnklos 8y ago

Something that the kids of IT today will never learn: Never replace anything unless the thing doing the replacing is better than the thing being replaced. The IRS' IT "upgrade" is just a huge money grab. Nobody has the slightest interest or desire to do anything properly - they simply want in on that lucrative IRS money. It's like the saying: "Anyone who is capable of getting themselves made President should on no account be allowed to do the job." Likewise, the kind of people who are bidding on this project are certainly not the kind of people that should ever be allowed anywhere near the IRS' computer systems.

s2g 8y ago
> Something that the kids of IT today will never learn Why, because you're busy being condescending to teach? Software should be considered a system that needs ongoing care and maintenance, just like anything else.
zitterbewegung 8y ago
All programs by the government are ways to employ people. They can be ineffective or effective but it still makes whoever employed happy to vote for whoever made the job up.
stefan_ 8y ago
Having this stuff digitized is actually pretty nice. The German ELSTER is basically PDF forms on steroids; same tax forms you presumably used to fill out by hand but with input formatting, validation, basic sanity checks and it estimates what you'll pay / get back at the end of it all. Then you can even submit it online if you have registered for a private key. Of course the tax forms themselves are still insane.
mbell 8y ago
Or you can continuously keep your code / systems up to date and never get wedged into a nearly impossible situation of having to rewrite 60 year old code (which likely has nothing resembling a test suite nor even a spare piece of compatible hardware to test on).
- 8y ago

sunstone 8y ago

Next time they do this they might want to trying testing out the new system before putting it into production.

raverbashing 8y ago

The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago. How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

dman 8y ago
How do you create incentive structures for new employees who have to spend months (years?) coming upto speed with giant state machines before they feel remotely confident of the the effect of their changes? For better or for worse, currently the glory is in writing new systems rather than improving existing ones.
- DrScump 8y ago
  Unfortunately, it's not just about "glory". If you work in "obsolete" languages/environments, you likely will be perceived as less valuable to your next employer.
  - JustSomeNobody 8y ago
    This right here is a huge problem with our industry. If you're not jumping from employer to employer every 1.7 years, you're labeled as out of date. It takes some really talented developers to truly understand some of these old systems. They should be given the proper respect they deserve.
    - s2g 8y ago
      No, you can stick around. I wouldn't consider someone who spent 5 years on AWS, especially if they moved internally, out of date. Whereas I've done nothing remotely close to where I want to be in about 3 years. I feel very scared about my skills, especially since I'm not even 10 years into my career.
      - 8y ago

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

cptskippy 8y ago

I feel like the authors of this article are being very misleading by using broader terms where they could be more specific, like using system where they could instead say software. The IRS isn't running 60 year old hardware, and the failure was in hardware that was just 18 months old. But because the IRS is running software that is 60 years old, the authors can claim the "system" dates back to the 1960s. It's also clear the authors don't entirely understand what they're talking about. In the second article linked to, which is written by an author of this article, he states "[t]he Individual Master File, a massive application written in the antiquated and low-level Assembly programming language" as if to imply there is but one assembly language. A GAO report I found chasing the last link in the article indicates that the two oldest systems in the Government are the IMF and Business Master File (BMF) which it says both run on IBM Hardware. My guess would be that the IRS is running modern z/Architecture mainframes in System 360 Compatibility mode. The System 360 dates back to the early 60s and IBM has maintained backward compatibility with it all the way up through z/Architecture systems. This would also explain why they're so far over budget and behind schedule.

walrus01 8y ago
the z/whatever to host legacy mainframe applications is basically a bare metal platform, custom hypervisor OS, running system 360 VMs, right?
- cptskippy 8y ago
  I'm not familiar with all of IBM's z offerings but it was my understanding that older systems were completely emulated and that there is no hardware compatibility. The emulation is completely independent of the underlying hardware platform and it allows them to retain ridiculous uptimes because of all of the hotswappable redundant systems that are able to run the systems concurrently.
reaperducer 8y ago
It's general reporting by general reporters using general language for the general public. It's not a tech publication. Joe Lunchbucket doesn't care if it's hardware that failed or software. The system failed. That's what his tax dollars pay for.

js8 8y ago

What kind of hardware was it running on? The article is not very specific about that. (At first I thought that they upgraded IBM mainframe to z14, and that caused the crash, but it's not yet 60 years old.)

cptskippy 8y ago
They use very broad terms to imply it was 60 year old hardware that failed but early on in the article state it was an 18 month old piece of hardware in a "system" that dates back to the 60s. If you dig into some of the other links and a GAO report you learn that there's two programs, the IMF and BMF, that are partially written in assembly that dates back to the 60s. They're most likely running some z/Architecture in System 360 virtualization mode.
cmiles74 8y ago
While the code is 60 years old, the hardware is modern. According to this article, the component that failed is about 18 months old. When they talk about the software and hardware "no longer being supported", I think they really mean that the original target platforms are only present in the form of emulated software. IBM still supports execution of System/360 software on the System z. [1] "Overall IRS maintains over 20 million lines of assembly code. These millions of lines of archaic software, and hardware, that is no longer supported become more difficult and costly to maintain each year, and poses significant cybersecurity risks. To IRS’ credit it keeps these old systems running during filing season, but relying on these antiquated systems for our nation’s primary source of revenue is highly risky, meaning that the chance of having a failure during the filing season is continually increasing." - David Powner, director of IT management issues for the Government Accountability Office [0] "[IRS Chief Information Officer Gina Garza] explained the code for the IMF was written in the 1960s, but the hardware it runs on is modern." [0] [0] https://federalnewsradio.com/technology-main/2017/10/a-sense-of-urgency-as-irs-legacy-it-systems-grow-increasingly-older/ https://federalnewsradio.com/technology-main/2017/10/a-sense... [1] https://arstechnica.com/information-technology/2014/04/50-years-ago-ibm-created-mainframe-that-helped-bring-men-to-the-moon/ https://arstechnica.com/information-technology/2014/04/50-ye...

deleted 8y ago

[deleted]

jgalt212 8y ago

Can the USDS assist here?

normaljoe 8y ago
Last time I talked to somebody at USDS two years ago maybe, they where already helping. I think the biggest task was trying to figure out getting off tape and moving normal hard drive storage. Apparently this is hard because the really old programs are the OS and file system. Without the details it sounds to me this is what broke or something around that.

tomohawk 8y ago

It's probably easier to just scrap the income tax than to fix the system. Maybe go to a land tax instead.

sehugg 8y ago

A delivery date for CADE 2, the IRS’ subsequent modernization effort, has slipped several years even as contractors working on the project have earned as much as $290 million. They call them "Beltway Bandits" for a reason.

empath75 8y ago
As someone working on a modernization contract, the problems are more complicated than just contractors standing on shovels. Building an app to a complex and ever shifting spec is a much different process than the way that startups work. You can’t just release a half done app to friends and family. You need to do the whole thing at once, with 100% if the functionality and it has to be ready for everyone in the country to use it in day 1. And the federal government has fairly onerous requirements for getting stuff through security— you can’t just google for random shit from GitHub and expect to put it straight into production. I can only speak to my particular project, but everyone on it is working hard at getting it done on time and on budget, using modern deployment and development practices, but there’s limits to how much you’re allowed to just move fast and break things.
dx034 8y ago
I'm always amazed that so many are outraged by Investment Bankers and Lawyers making millions when IT consultants get so little attention. Maybe it's because few people understand what IT consultants actually do and have no alternatives but hiring them but cost overruns due to increased IT consultancy appear much more prevalent than escalating legal costs.

PinkMilkshake 8y ago

So they need something proven, scalable, highly concurrent, fault tolerant, and runs in a VM so it's not tied to hardware. Sounds familiar.

interfixus 8y ago

>[...] date back to 1960, when John F. Kennedy was president No he wasn't, so my confidence starts flagging.

BrandoElFollito 8y ago

What part is complicated when transitioning such a system? Legacy data which must be transformed? Old business logic which must be maintained? I do not know if there is a need to replay old rules vs. just kerping a snapshot Continuity? Maybe there are so many transactions happening, linked to existing data that an interruption is complicated? Risk of accessing old data which is better left undisturbed?