16 ms·
COBOL underpins the entire financial system
- libraryofbabel 6y agoRather than reading yet another article about how there’s all that COBOL still out there powering critical systems, which should comes as no surprise whatsoever to regular readers of Hacker News —— why not read a book about the deeper reasons how and why old technologies persist, and the ethic and duty of maintaining them. Specifically: David Edgerton, The Shock of the Old: Technology and Global History Since 1900 (2006). Andrew L. Russell and Lee Vinsel, The Innovation Delusion: How Our Obsession with the New Has Disrupted the Work That Matters Most (2020). You won’t agree with everything in those books, but they will make you think deeply about old technologies and the stories we tell ourselves about innovation.
- herodoturtle 6y agoThese books sound great. Would you recommend I read them in that order too?
- libraryofbabel 6y agoYes! The second book is heavily influenced by the first. The first one (Edgerton) is history, with the argument being roughly that we should frame the history of technology not just in terms of inventions/innovations, but also in terms of what was actually in use at a particular moment. (So the early 21st century is about self-driving cars and the iPhone, but also the Haber-Bosch Process and COBOL.) The second book is more about the present, and is concerned with how we tend to talk too much about innovation and not enough about maintenance. (So they love Right to Repair and Open Source maintainers, among other things.)
- herodoturtle 6y agoThank you :)
- throwaway201103 6y agoChicago FD dispatched fire engines by telegraph until the 1990s. http://chicagoareafire.com/blog/2015/02/chicago-fd-history-the-joker-stand/ http://chicagoareafire.com/blog/2015/02/chicago-fd-history-t...
- totemandtoken 6y agoSomewhat related question you might be able to answer - what are good examples of really resilient code bases a junior developer can look at to learn? Old code still in production just sounds antifragile to me and that's the sort of code I'd like to write
- xiphias2 6y agoIt's just the opposite. Old code is so fragile because you don't know what's important in it. An example is Bitcoin's concensus critical code: anything that goes in there probably has to be maintained forever (or for the life of Bitcoin)
- segfaultbuserr 6y ago> Old code is so fragile because you don't know what's important in it. An example is Bitcoin's concensus critical code I don't think Bitcoin is a perfect example here, the Bitcoin consensus rule is fragile because it's Bitcoin, not because it's old. The unchangeable consensus code is a result due to the fundamental property of Bitcoin itself (every client must run in lockstep, even the slightest deviance cannot be tolerated, which means there should be one and only one Bitcoin client whose behavior shall remain consistent for as long as Bitcoin exists), this outcome is independent from its code quality, age, and other factors. ...Nevertheless, on second thought, the Bitcoin analogy isn't too far from the truth. If an aged legacy system is the dependency of many other systems and serves a critical role, it effectively becomes Bitcoin-like.
- xiphias2 6y agoOf course, regulations help old code to remain as well, which is probably the case in finance. Just think about Boeing's case where the company is trying to fit new code into the memory of a 8086 chip.
- segfaultbuserr 6y agoI thought it was the 80286, not the original 8086. Don't underestimate the computing power of an aircraft too much...
- deleted 6y ago[deleted]
- 0x445442 6y agoI'd throw Postman's work in there as well... https://en.wikipedia.org/wiki/Technopoly https://en.wikipedia.org/wiki/Technopoly
- tuatoru 6y agoThanks for these! I have a copy of The Shock of The Old and I did find it thought-provoking. I had recently become aware of the ideas/discipline of discourse analysis[1] so it was more enlightening than it might have been to someone who wasn't really aware that our narratives are framed (like younger me). I'll be checking out the Russell and Vinsel. 1. https://en.wikipedia.org/wiki/Discourse_analysis https://en.wikipedia.org/wiki/Discourse_analysis
- noodlesUK 6y agoI have no problem with older systems being used for financial stuff like this. Sometimes it’s easier to just keep using what you already have. What I do have a problem with is when the antiquated computer systems make an entire country’s banking sector move glacially. ACH is so slow, and other countries (EU especially) have payment systems like SEPA and Faster Payments which are universally used, and clear pretty much instantly. I don’t know why the US doesn’t have something like this (I believe the fed is working on it), but it’s definitely created the room for the myriad of p2p payment apps like cash app and Venmo in the US, which really need not exist in a more developed banking system.
- toomuchtodo 6y agoThe Federal Reserve is rolling out an instant payment system called FedNow in 2023. It’s design and implementation languished until Congressional pressure with concerns about Zelle becoming a dominant payment platform as a private entity (owned by Early Warning Systems). A separate issue is the Fed exploring creation of “digital currency”, essentially deposit accounts at the central bank. That was encouraged by the threat of Facebook’s Libra and China’s “digital yuan”. https://corpgov.law.harvard.edu/2020/08/31/fednow-the-federal-reserves-planned-instant-payments-service/ https://corpgov.law.harvard.edu/2020/08/31/fednow-the-federa... https://www.reuters.com/article/us-china-currency-digital-explainer-idUSKBN27411T https://www.reuters.com/article/us-china-currency-digital-ex...
- fennecfoxen 6y agoThis is not without controversy. It has placed the Fed in the awkward position of competing with some of the firms it regulates, who had already begun building their own systems.
- deleted 6y ago[deleted]
- grenoire 6y agoIn all fairness, same shit in Europe with PSD-2 (Payment Services Directive). Imagine an auditorium with representatives from each bank in Europe, no matter how big or small; each shouting at each other about how their systems are they way they are and will not be changed! It makes agreeing upon standards awfully hard. SEPA standards are pretty slick though, built a couple implementations myself for transfer documents (pain.00X) and It Just Works™.
- jackfoxy 6y agoCoding in COBOL was a real downer after 2 years of assembly language coding. One register pointed to the top of all your main memory in assembler. Solving COBOL dumps required resolving pointers to pointers, and I had not kept good notes in COBOL class. Fortunately soon after I returned to COBOL programming debuggers came on line that solved the COBOL misdirection for you. Misdirection is the kindest word I can think of to describe COBOL dumps. I've really never solved a dump since COBOL, which I have not worked in since 1996. Debugging has come a long way.
- herodoturtle 6y agoThis was such an incredibly insightful historical perspective. Thanks for sharing.
- unnouinceput 6y agoAny COBOL contractors that can tell what the pay/hour is? First hand experience please, rumors I've read them all already.
- boredatworkme 6y agoMy firm pays a contractor for supporting our mainframe systems. His company charges us ~$900 a day for his services.
- bingojess 6y agoYour firm pays his company $216k a year. He probably sees 1/3 to 1/2 of that, maybe less. I'm not in software but that seems low doesn't it
- blackrock 6y ago$900/day is $225,000/year. Or about $112.50/hour. But this is what your company pays as Corp-to-Corp, to his company. His gross pay is significantly less. Maybe at $65/hour. So his annual pay might only be $130,000/year. And while this is decent pay, for middle class wages, it is not really very impressive.
- rantwasp 6y agoalso, how does one learn COBOL nowaday? it’s mostly a curiosity buy would be interesting to see what people that use it recommend
- lstamour 6y agohttps://www.coursera.org/learn/cobol-programming-vscode https://www.coursera.org/learn/cobol-programming-vscode is one place to start. :) If you're done that, I presume there's also https://www.coursera.org/professional-certificates/ibm-z-mainframe https://www.coursera.org/professional-certificates/ibm-z-mai... though it's starting to feel even more like marketing after seeing all that.
- beny23 6y ago
- megraf 6y agoI have some experience in Fintech. I wrote the translation layer between financial cores and a RESTful API (essentially wrapper logic). D+H, Fiserv, BottomLine, all provided us integration documentation which allowed us to preform actions (transfer funds between accounts, get balances, etc.) and it wasn't _too_ bad. I do remember one particular core that was surprising difficult. Everything was encoded in EBCDIC. You had to rotate telnet (!) connections depending on the day of the week. There was one person (a man named Earl) who could help you if you had a problem with the continuous stationary documentation (luckily it was scanned). Oh the memories...
- vmception 6y agorotating telnet connections lol
- skunkworker 6y agoI was amazed to learn how much the US depends on SFTP for ACH. Though if a more instant based system becomes available through the Federal Reserve we could finally get to quicker payments like the rest of the world.
- patrickk 6y agoIn the EU, SEPA was forced through via legislation[1]. The financial industry had to be dragged kicking and screaming along. [1] https://en.m.wikipedia.org/wiki/Single_Euro_Payments_Area https://en.m.wikipedia.org/wiki/Single_Euro_Payments_Area
- valenterry 6y agoIt's funny: the more readable and maintainable and testable and modular some code written in a certain programming language is, the easier it is to understand and rewrite it step by step. Hence, code written in good programming languages might have a shorter lifetime compared to say... COBOL or PHP code. A bit paradox.
- tannhaeuser 6y agoI think the elitist attitude towards COBOL really should stop. COBOL is a domain-specific language for implementing commercial/financial batch processing jobs, no more, no less. With adequate control over decimal arithmetic and established declarative practices for structured file exchange, subroutine reuse, and job control. Based on a language spec with reasonable portability. It's not about good or bad but fitness for a purpose. COBOL programs can be very clear and to the point compared to eg Java programs replacing them with lots of misguided OOP patterns and metaprogramming.
- valenterry 6y agoCOBOL just didn't improve as a language (yeah, I know that there is also OOP COBOL) and hence it's not productive to use it. Even Java is improving more than COBOL and improving fast is certainly not Java's strength...
- glangdale 6y agoFor those who haven't seen this old joke, Bruce Clement suggested in SIGPLAN 1992 that OOP COBOL should be called "ADD 1 TO COBOL GIVING COBOL".
- cogman10 6y agoI think it is fair to say that Java up till about Java 8 had a problem with not evolving fast enough. However, I'd say now Java is really one of the quicker evolving languages out there, which is really impressive. They went from a go like evolution pace to being closer to a Rust or Javascript evolution. Turns out, fast release cycles do a lot of good for a language.
- bikamonki 6y agoSo if I learn Cobol, I'll land a job in a bank?
- spzb 6y agoYou might also need to build a time machine to go back to the 1960s
- commandlinefan 6y agoYes, but it will pay 1/5 of what you can make expending the same effort learning python and doing “machine learning” for a web startup.
- Veen 6y agoOn the other hand, your COBOL job will last a long time. And you won't be talked into taking a shitty salary with the promise of big payout that doesn't happen.
- lanewinfield 6y agoThis is like the code you write on the stuff you blast off into space—sometimes the most important things need the simplest, most trustworthy, and, yes, oldest code.
- Threeve303 6y agoOur wonderful reserve currency rests atop a fairly well tested and robust system called SWIFT which at its core is just ftp servers with some message queues. If it ain’t broke?
- throwawaybutwhy 6y agoHave some doubts about its robustness and security [0]. To be honest, though, once you are against a dedicated nation-state team all bets are off. [0] https://en.wikipedia.org/wiki/2015%E2%80%932016_SWIFT_banking_hack https://en.wikipedia.org/wiki/2015%E2%80%932016_SWIFT_bankin...
- chx 6y agoblink I've never heard of SWIFTNet using FTP at its core, do you have some links I could read? Wikipedia, for example , while not a fully trusted source, has > Alliance Access (SAA) and Alliance Messaging Hub (AMH) are the main messaging software applications by SWIFT, which allow message creation for FIN messages, routing and monitoring for FIN and MX messages. The main interfaces are FTA (files transfer automated, not FTP) https://en.wikipedia.org/wiki/Society_for_Worldwide_Interbank_Financial_Telecommunication https://en.wikipedia.org/wiki/Society_for_Worldwide_Interban...
- henning 6y agoI don't like articles about COBOL, because most of them talk about how young people don't want to learn it. I'm fine with learning shitty technology. The real issue is the money. Look at the salaries for COBOL jobs. They suck shit. If banks need to update their software, hire people and train them. It will take time, and time is money. That money could be invested into something else (there is an opportunity cost). If that something else is a higher priority to you, then maintaining the integrity of your critical software is not actually as important to you as you say it is. I will not take a pay cut to work on COBOL for a bank.
- kody 6y agoRight. I grew up hearing that COBOL and Fortran devs would become extinct and extremely valuable in my lifetime.
- patrickk 6y agoThis mention this in the article. They used to grab people off the street and teach them COBOL, paid for by the banks. What happened between the 1960s and today that this type of practice stopped?
- refactor_master 6y agoThey realized they could wait around for the government to bail them out on this expense too? Either it’s a really easy language to learn or the banks used to invest much more in teaching it than they do now (= zero). If COBOL is so domain-specific, why not teach it as such on the job? Why is it the public’s fault for not investing in it?
- rbanffy 6y ago> Either it’s a really easy language to learn or the banks used to invest much more in teaching it than they do now (= zero). Very few companies invest in training these days. They expect schools and colleges to teach some professional skills (this is not what they should be doing) and invest as little as possible in new hires, specially for technical jobs.
- fctorial 6y agoThe header (as high as the screen) doesn't work without js. Scroll to read the actual article (readable without js).
- kome 6y agoWhy new languages can't be easy for newbies as BASIC and COBOL were? Why technology gets harder instead of getting easier?
- Heliosmaster 6y agoBecause we keep building on top of things, and trying to make stuff simpler to use, rather than simpler to change. Hence why I'm curious to see what programming could be in 10-15 years time, when programmers had an iPhone (or equivalent) they _whole_ life.
- kome 6y ago"Because we keep building on top of things, and trying to make stuff simpler to use, rather than simpler to change" wow, very good point
- BlueTemplar 6y agoIt's been a long time since I have used BASIC, but is Python really harder?
- rbanffy 6y agoIt introduces some concepts that are absent from BASIC. If you think about it, BASIC, with its limited conditional branching, no concept of functions - it was set variables, then use GOSUB, with all variables being globally visible, is very close to the machine language of the host computer with some added types. If you wanted to do recursion, you'd need to invent a stack.
- stjohnswarts 6y agoPython is much easier than BASIC for anything other than the simplest of tasks. Plus there is memory safety and much more dynamism. It also encourages you to write in a more explicit fashion than things like LISP or Ruby or Perl (ug!) that I prefer.
- 6y ago
- beagle3 6y agoCOBOL has a SEP field on it (and for good reasons - it is someone else's problem), but when discussing APL, I've often reached for it as an example that everything is just preference. When I show an APL or K gem such as "|/0(0|+)\" (which is a complete and rather efficient implementation of the maximum-subarray-sum problem), people usually complain that "it's unreadable". But then "ADD ONE TO X GIVING X" compared to "++x" or "x=x+1" shows that explicit is only better than implicit when you're not familiar with the notation. Readability is all about your expectation and familiarity.
- nextaccountic 6y ago> COBOL has a SEP field on it (and for good reasons - it is someone else's problem), What do you mean by SEP field? (I can't find it in the article either). Is this a programming construct like a field of a record?
- QuesnayJr 6y agoSomebody Else's Problem field. It's a joke from Hitchhiker's Guide to the Galaxy. Instead of an invisibility field, where you can't see something, you use an SEP field, which makes you look at something, subconsciously think "that's somebody else's problem", and then you look away.
- teataster 6y agoI found out most people confound familiarity and "intuitiveness" when trying to teach Emacs. They would say it was not intuitive because common key shortcuts were not default. I would try to explain, to no avail I must say, that it is rather intuitive since, once you understand the basics, everything follows quite logically and key bindings are customizable.
- BlueTemplar 6y agoOne of rules of design is that there is no such thing as intuitive, only familiar.
- 6y ago
- 4778468d 6y agoIndeed.com lists 690 cobol job ads for the whole of USA. 36,600 java ads.
- IshKebab 6y agoIs this still really true? Most trading shops use C++ and a ton of banks use Java. This article provides no evidence that COBOL is still the dominant language.
- bitcharmer 6y agoExactly my experience. Been working in fintech for 15+ years. Went through 4 tier 1 banks and a few smaller ones. Most popular languages are cpp, java and sometimes even c# or ocaml. Never did I hear cobol even being mentioned. Just wondering how true those statements about cobol still being so popular are...
- beny23 6y agoHow many retail banks, insurances and government departments have still got a mainframe sitting at the very back though, based on a good few engagements in banks and gov in the UK, I haven’t yet come across where ultimately some mainframe batch job is hidden behind a fair few layers of middleware and MQ. And my guess is that most of that is on COBOL.
- nulbyte 6y agoYes, it dominates. We don't talk about it a lot, because talking about things working isn't all that interesting. We don't talk about it when things don't work, because the failure isn't the language but the design. We don't talk about COBOL so much as we talk with COBOL. I have email after email with COBOL snippets discussing various projects (and I'm an analyst here, not a programmer). No one mentions COBOL by name, but almost anyone, even nonprogrammers, understand it to a degree that they can reason about what is happening and point out errors in the assumptions made by the programmer. There are IVRs that talk to mainframes via tn3270 to execute CICS transactions. Most CICS programs are written in COBOL. There are external websites, internal web applications, even VBA macros that rely on tn3270 and CICS, thus COBOL. While other languages may dominate the business of languages, COBOL dominates the business of business.
- pachico 6y agoI work in a OTA and you would be surprised to know how much of the accommodation industry still relies on good old fax machines. I'm not aware of any other industry using fax as much as this (which I'm sure there is).
- Dowwie 6y agoDon't ever work on a COBOL modernization project. It is a career killer with little to no upside. COBOL systems and the people directly using them are Chernobyl. Both government regulators and operations personnel have made careers out of trusting black boxes that no one fully understands anymore. Regulatory specifications are ambiguous and disorganized. Imagine working with someone who knows that you are trying to shed light on the technology that they were responsible for, yet do not fully understand. How cooperative do you think these people will be with your efforts? How do you think the managers responsible for this work survive the politics of the project's inevitable failure? Blame rolls downhill. Modernization projects require strong leadership and collaboration across many teams. That's a huge problem because such an environment very likely does not exist in the world today. No one understands how the algorithms work or even how they should work! Forget how challenging it is to work with COBOL, which someone could eventually figure out. Whatever you've written has to be tested, and you don't have a trusted source of business logic from which to verify how it ought to work. You only have the black box. If you've been given this task, start interviewing at other companies. You were given a suicide mission by people who are well aware of that. My experience with this was with regulatory margin trading systems supporting a multi-billion dollar market.
- qz2 6y agoThere’s a lot of that out there. I have avoided COBOL but I have seen stuff written in VB6/COM which was also in the same state. Eventually the one guy who knows anything about it leaves or dies. In the name of “progress” someone wraps it in something else and sticks a web API around it but everyone is afraid to touch that bit of VB code. Now a third party calls this and wonders why it takes 10 seconds for an HTTP API call to run and they can’t be issued in parallel or it throws a 500 error. Then the third party asks to send 20,000 of these a day and then finger pointing and stuff occurs at management level between both companies. I have been on both sides of this and it sucks. I came up with Rug Driven Development as a title. All the risks and future problems are swept under the rug quickly while maintaining a happy smile and pretending everything is just fine and dandy.
- sdoering 6y ago
- beny23 6y agoI have an intuition that COBOL will become very valuable in the years to come as the majority of greybeards that keep it running will stop work - and then the projects to rewrite core business functions in banks, insurances will come think and fast and will provide a never ending gravy train for contractors that can help with migrations or keeping the lights on. I for one have COBOL on the list of languages which I want to know enough to understand.
- rzz3 6y agoI realize this is a naive perspective, but rather than try to “translate millions of lines of COBOL to Java”, why not just build a new system from the ground up, based on requirements and first principals? The function of these systems seems fairly simple from the outside. What am I missing?
- Guthur 6y agoAs someone else pointed out, there are no requirements the code is the only source. A long time ago I work on the core processing system for a large US insurer, the business would continually call up for confirmation on the business rules from the code. Not a basis for success on complex business laden with tonnes of regulatory oversight.
- rzz3 6y agoAbsolutely, but for systems like this, there should be clean separation between business rules and core application logic. Filters, hooks, rules engines, etc. I very much realize “easier said than done” but the complacency with regard to keeping these systems the way they are is truly alarming.
- commandlinefan 6y agoThat would take a long time and can’t be realistically predicted (even if it could, the prediction would be months). Management will never ok a development effort whose schedule isn’t predicted as more than a few weeks (ESPECIALLY in fin tech).
- bingojess 6y agoFor companies like banks, the transition is the riskiest part. For management, there is no upside in such projects since the system already works, but downsides are project failure and your banking systems ceasing to work. Not to mention cost.
- stjohnswarts 6y agothe only thing driving change on such a system is when it can no longer to scale to handle the "traffic" or legislation that forces it.
- deleted 6y ago[deleted]
- kanox 6y agoctrl-f blockchain: nothing? This technology is clearly crap and ripe for disruption. Sure: there are some good reasons for things to be the way that they are but decades of compatibility work won't save banks against a competent competitor.
- GnarfGnarf 6y agoThe problem is that if you convert your COBOL code to the language du jour (LDJ), you've only kicked the can down the road. You will always have to keep some COBOL experts on staff (the conversion is never finished), plus the new LDJ experts. In ten or twenty years, LDJ will be obsolete, now you have to convert to LDJ2. Now you have COBOL programmers on staff, plus LDJ programmers, plus LDJ2 programmers. No one wants to be on the obsolete teams, your salary budget bloats, and you have more Babel. If there's always going to be something better, and you're always going to be fragmented, why not stay with just the one single obsolete technology?
- tuatoru 6y agoTo me the appropriate language for a re-write of (software written in) COBOL is SQL. Edit: clarification in parentheses.
- cogman10 6y agoThis is one of my great fears as a software maintainer. It seems almost impossible for any product to ever fully shed itself of a technology. Primarily because while the initial work may yield benefits that management loves, they never prioritize finishing the job as some feature always takes priority. And now you have 2 problems. You have half your product in one language/tech, and another dark corner in another tech. Your build is more complex and your product more fragile.
- gregors 6y agoI can't confirm this is fact, but remember when unemployments systems in NJ couldn't keep up and they were asking for volunteers to help when the pandemic started? The people who worked on such systems were complaining that they were previously laid off to be replaced by contractors. https://www.reddit.com/r/recruitinghell/comments/fz6b74/pitch_call_for_coders_to_serve_in_the_covid19/ https://www.reddit.com/r/recruitinghell/comments/fz6b74/pitc... If I were going to work for a bank, I'd look at Nubank. * They have made 2 tech aquihires - Cognitect and Plataformatec * Clojure * Engineering culture I can't imagine how bad it would be to build up 10 years in a COBOL factory just to be let go. Pretty easy to get a new job I suppose. Most big business places I've dealt with don't embrace remote either.
- mkoubaa 6y agoHere's a place where interop can do magic. There's a .NET implementation of COBOL out there for instance. First thing I'd do if I inherited a project like this is change the runtime and write new code in C#, then slowly migrate away the internals ( and for places where the claim that COBOL is great holds merit, wrap those up in assemblies implemented in COBOL)