21 ms·

The most obsolete infrastructure money could buy – my worst job ever

464 points by kryptiskt 11y ago

vonmoltke 11y ago
Not nearly as bad, but this reminds me of some of the ridiculous requirements at the last program I worked at my previous employer. One of those was that our code had to compile on a 32-bit SPARC pizza box running Solaris 2.6. This was in 2009. It was all about some stupid metric: our code portability was measured by how many different platforms we could compile and run on. It didn't matter how obsolete or inappropriate those platforms were; if we could compile and run it counted. Having that number go down was politically unacceptable.
- Vexs 11y ago
  Reminds me a bit of VIM. I love vim as much as the next guy, but I can't wait for neovim to get stable.
  - afarrell 11y ago
    I was running neovim without issue on OSX until about a month ago when I switched to nuclide
- nailer 11y ago
  There's something strange about Solaris. Around the same time, an investment bank (where the Solaris team constantly justified their existence with 'dtrace!' 'zfs!' 'zones!') was paying 50K a year per server multiplied by hundreds of servers to run Solaris 8 extended support - which didn't have any of those things, which is good, because the staff didn't know how to use them other than say they made Solaris great.
  - trhway 11y ago
    >where the Solaris team constantly justified their existence with 'dtrace!' 'zfs!' 'zones!' they did that inside Sun as well. Most noise was about DTrace, the most useless thing for the Sun/Solaris at that moment.
    - mattzito 11y ago
      Dtrace was, and still is, awesome. Linux has come a long way in this area in the last few years, but DTrace was one of the hottest systems management capabilities I'd seen in forever, particularly on the very large SMP systems that had become Solaris's sweet spot in those waning years.
      - gaius 11y ago
        Quite. Systemtap is a joke compared to dtrace.
        nailer 11y ago
        There are reasonable criticisms of Systemtap, but a lot of Sun marketing was just bizarre. They set it up on Ubuntu (which doesn't include it, and which isn't their competitor: Red Hat is) as an example of what the setup is like. They kept complaining it was unsafe, when it was only unsafe if you wanted to use raw C for tapsets. Sun had some great tech, but they were also a sales and marketing machine that had no problems spouting garbage to make a sale.
    - DonHopkins 11y ago
      I miss fastboot.
- rjsw 11y ago
  TBF, testing on SPARC is a good way of finding endian and alignment bugs.
  - vonmoltke 11y ago
    It would be, if our two target platforms hadn't already been x86-64 and POWER.
- kevin_thibedeau 11y ago
  I had to port a product to Solaris 8 on similarly crusty hardware in 2011. Wasn't so bad other than having to work around an excruciatingly slow build process.
api 11y ago
More like Not Invented Here syndrome to the level of fatal pathology.
- Sanddancer 11y ago
  It's its sibling, Already Invented Here. Every kludge makes sense in the context of the system state when it was implemented, but that leads to a system where /everything/ ends up a kludge. Management didn't want to invest any time in fixing a system that already "worked", so 30 years of temporary fixes turned it into an absolute monstrosity.
  - ethbro 11y ago
    Both of these are important observations. And, as far as I can tell, both of them exist largely because of corporate politics. If you could design a corporate culture that minimized the incentives towards both, I think you'd be a long way towards having a more sustainable company viz technology.
    - webjprgm 11y ago
      There was one time (2006 at a company of less than 50 people) where I absolutely refused to build the new pricing features in a billing system (which would have been a big hack on a long series of smaller hacks) unless they let me refactor it (meaning rebuild it from scratch). After waiting for a few months without their new pricing features they finally gave in. That's at a young, nimble company where the co-CEO had a programming background. I can only imagine how hard a fight that would be at a larger, slower company.
      - api 11y ago
        I've worked for a government data center where they ran instances of Ultrix in a commercial VM to run old data processing software that ingested and barfed out CSVs that contained UUencoded blobs of image data. All this was written in a mixture of very very old Perl, shell scripts in some gawd-awful Ultrix shell CSH script, and C code full of weird -isms that made it difficult to port to anything made after the rise of mammals. This was of course mission critical. At least they virtualized it instead of continuing to run physical Ultrix boxes, which was apparently a huge fight with management. A side dish of LULZ was that they were experimenting with moving the VMs to EC2, so there is probably Ultrix in EC2 now. This stuff is really, really common in large "mature" industries and government.
PaulHoule 11y ago
The only way to stop this madness is to go on strike against it, as you did. I work very hard to never mention online that I've had anything to do with SharePoint or ColdFusion because it will bring recruiters with a job that will kill you.
- foobarbecue 11y ago
  Not hard enough, apparently.
- mring33621 11y ago
  Dear %prospect_name%; I was in our recruiting DB the other day, and came across an old version of your résumé. I am very impressed with your background and think you might be a perfect fit for an opening that we have for a Senior ColdFusion Developer. Please get back to me at your earliest convenience regarding this exciting opportunity. Thank you. Sincerely, %recruiter_sig%
  - Kalium 11y ago
    You're going to give me nightmares. I still get emails like this from a six-year-old version of my resume.
    - ErikRogneby 11y ago
      Where is the national do-not-call list for clueless tech recruiters?
      - Kalium 11y ago
        I don't know. And half of appear to be operating out of India anyway, judging by the noise in the background on the lines.
      - rogerbinns 11y ago
        I maintain one at http://www.rogerbinns.com/blocked_recruiters.txt http://www.rogerbinns.com/blocked_recruiters.txt My resume has a "note to recruiters" attached. It is very clear if a recruiter has made any effort to read any of that. The clueless ones end up in the list above. http://www.rogerbinns.com/recruiters.html http://www.rogerbinns.com/recruiters.html
nasalgoat 11y ago
Although nothing compared to what was in this story, you should see the level of obsolete crap in place at adult website providers. Systems that haven't changed since the 90's that still generate revenue, so they'll never be shut off until there's a flood or a 40MB hard drive finally dies.
- kenko 11y ago
  Have to admit, that approach makes a lot of sense to me. It's still chugging along and bringing in $, so why not just have a policy of benevolent neglect?
  - nasalgoat 11y ago
    That Pentium-90 is probably sucking a ton of power at this point, when it could be virtualized and you could run 50 of those boxes on an $800 Dell pizza box. I mean, they are literally running the same hardware from the 90s.
    - wtallis 11y ago
      That Pentium 90 is a 9W processor. Even after accounting for the inefficiency of a 90's era PSU, there's not much power to be saved.
      - nasalgoat 11y ago
        Plus the hard drive(s), the space it's taking up in the DC, the cooling overhead, etc.
        wtallis 11y ago
        There's not really any significant room for improvement on any of those factors. You might be able to save a rack unit or two, and several watts by replacing hard drives with a SSD, but if you don't have 50 boxes to consolidate, you're still facing the baseline cost of having a box taking up some amount of physical space and requiring several watts of power. Unless you make the much bigger up-front investment of migrating to a shared hosting platform.
russnewcomer 11y ago
I think the most interesting lessons of this story have to do with the following expectations. First, the expectation of support timelines. It seems in tech that support is promised for products far beyond when the vendor should reasonably be expected to support them. Companies make large investments in technology that ultimately aren't long-term enough. Lesson: Any company doing a major project where there is significant cost (as a percentage of the overall cost) in hardware or software, and that selected hardware or software is not commodity level or has low market penetration in their industry should seriously reconsider the projections of TCO, as long-term support will get increasingly expensive over the life of the project (Contrived example, supporting Windows XP via virtualization vs OS/2 Warp) A second expectation is the timeline of viability for technologies. Non-technical business people of the past generally seemed to have an expectation of viability that was around 15-20 years. More recently, stated expectations seem shorter, but are actually not. Lesson: Businesses that make serious investments in technology need to consider length of that investment, and technical people should probably take those estimates and double or triple them, because that's probably how long hardware/software will be in use. Companies providing technology or support need to have realistic timeframes of support. As a corollary to this lesson, I have spent some of my career thus far doing IT support-type roles for small organizations, and while I haven't encountered quite the same level of obsolescence as the author, I have seen people using NT 4 for critical server infrastructure in 2012, I've set up DOSBOX[1] in Windows 7 for clients to continue to use that version of Lotus 1-2-3 they paid for and have been using since 1987 , and I've tried to figure out, in 2011, why Netscape Navigator on, I think System 7, ( using dial-up! ) wouldn't properly render a webpage. People keep tech a long time, and people in the tech world need to expect that, and plan accordingly. The final expectation is one that technology companies have, which is that tech should be replaced. While there are many use cases where tech gets old, breaks, has to be replaced, there are lots of cases where it can be functional for almost indefinite periods of time, and industries that will expect such. Lesson: Tech companies in should not neglect the future, but should also strive to design objects of technology that can last a very, very long time. Companies that do so will find a niche and make money because there will always[2] be enough customers to support them. [1]It was going to get some use on an iPad via LogMeIn and there was some reason I've forgotten why DOSBOX was a better choice than the command prompt [2] Subject to prevailing economic winds, of course. You can still buy buggies and buggy whips, but it's not a growth business.
JohnTHaller 11y ago
I was completely expecting 8 inch floppies or punch cards to show up once I was half way through reading.
- sthorn 11y ago
  LOL, we debugged our PL/M with an Intel ICE ("blue box") that used 8" floppies. Good times.
  - jsnell 11y ago
    One of these? http://www.computinghistory.org.uk/det/731/Intel-MDS-80-Microcomputer-Development-System/ http://www.computinghistory.org.uk/det/731/Intel-MDS-80-Micr... Wow, that looks just amazing.
ddoscampaign 11y ago
Reminds me of a contract gig I passed on to support Microsoft's creaky inheritance of Danger's Hiptop/Sidekick stuck in cheapest maintenance mode. Everyone associated were trying to rotate out and it seemed like morale was less than zero. Also never work for slave-driving shops like Taos (aka Tause Mountain) unless you really need the money because it will crush your soul. (eg master cool techs and be awesomely personable to get cooler gigs/referrals.)
- rietta 11y ago
  That transition was a real mess. We had some games in the T-Mobile Sidekick store and getting paid the royalties by Microsoft was way more difficult than it should have been after the transition. It took months to get added as a vendor and I had to even send them a "final invoice" using the numbers that they had provided. The next collection step was going to be at least small claims court. The Danger team did built a great system for its time and by the time Microsoft took over, it went downhill fast. And then Microsoft's Kin flopped too.
  - hga 11y ago
    The Danger team did built a great system for its time and by the time Microsoft took over, it went downhill fast. Microsoft managed one of the biggest cloud computing screw ups in history to date: https://en.wikipedia.org/wiki/2009_Sidekick_data_loss https://en.wikipedia.org/wiki/2009_Sidekick_data_loss The incident caused a public loss of confidence in the concept of cloud computing, which had been plagued by a series of outages and data losses in 2009. It also was problematic for Microsoft, which at the time was trying to convince corporate clients to use its cloud computing services, such as Azure and My Phone. I've heard good things about Oracle's RAC, but it's understandably intolerant of your screwing up its disks (SAN mis/re-configuring) when you aren't properly maintaining backups. I also heard the consultants you have to hire after you manage such a feat are expensive.
    - bro-stick 11y ago
      Yup. I managed some dataguard (not RAC) instances on AWS for Palm pre HP. Thankfully we had DR plans and snapshots to cover our asses. Edit: Fixed HA techs.
Theodores 11y ago
I supported booking systems that could have bookings stretching far into the future, this made switching systems tricky as a lot of quite complex bookings would need recreating with each taking proper time to do. The scope for automation was there but that wasn't possible. With the software we supported there was no expertise in the company as the original developers left a long time ago. Therefore there was no way to understand why something broke or to write a patch. Instead everything was fixed by rote, if this went wrong then you did this. If that went wrong then you did whatever was needed for that. All of these fixes were handed down in the oral tradition and you took your own notes - there was no manual. We further out sourced to India so there was no trace of the people responsible for the code, just us clueless idiots. Some of these systems were MS DOS based and this was about a decade ago. What I did manage though was a means of getting on to any computer anywhere in the world even if it was totally air gapped and owned by someone that did not speak English. Even better, in most cases this would take seconds from a standing start. If I needed to access a computer that was on a remote network but turned off, I could usually Magic Packet it on and be fixing it as soon as it booted. This was a golden era for my remote access skills, nowadays I have difficulties getting my own computer online! There is another way of looking at this, from the perspective of what will be out there for you to do in 25 years time, I am optimistic that there will be legacy work for us if we cannot keep up with the new stuff.
- rwallace 11y ago
  > What I did manage though was a means of getting on to any computer anywhere in the world even if it was totally air gapped and owned by someone that did not speak English. Even better, in most cases this would take seconds from a standing start. I'm curious - how did you do this?
myth_buster 11y ago
I'm wondering whether the use of ingenious (adj) (of a person) clever, original, and inventive. in place of indigenous (adj) originating or occurring naturally in a particular place; native. is a Freudian slip or sarcasm. Edit: Never mind; they used it with in-house
- kelvin0 11y ago
  Sarcasm would be my bet ...
russell 11y ago
As soon as I saw the title, I thought of the furor when Gingrich and Clinton discovered that the FAA was using vacuum tube computers and radar in the air traffic control system. http://govinfo.library.unt.edu/npr/library/clinton.htm http://govinfo.library.unt.edu/npr/library/clinton.htm The foreign (sole) vacuum tube supplier that Clinton mentioned was Russia. Oh, the embarrassment! I knew that there were some delays in rolling out the the replacement system, so I did some googling to see when it actually occurred. My god, it's now. We are in the middle of it. The upgrade was responsible for the recent air traffic meltdown. Please somebody, point out my flawed research skills. I cant believe this.
- steve-howard 11y ago
  I found this article from Wired earlier this year: http://www.wired.com/2015/02/air-traffic-control/ http://www.wired.com/2015/02/air-traffic-control/ It doesn't specificaly mention vacuum tubes, but it does explain why it's entirely plausible that they're still around: >This technology is complicated and novel, but that isn't the problem. The problem is that NextGen is a project of the FAA. The agency is primarily a regulatory body, responsible for keeping the national airspace safe, and yet it is also in charge of operating air traffic control, an inherent conflict that causes big issues when it comes to upgrades. Modernization, a struggle for any federal agency, is practically antithetical to the FAA's operational culture, which is risk-averse, methodical, and bureaucratic. Paired with this is the lack of anything approximating market pressure. The FAA is the sole consumer of the product; it's a closed loop.
  - _delirium 11y ago
    I'm skeptical that the culprit is mainly to be found in specific features of the FAA, and the article doesn't seem to provide much causal evidence of that (though I recognize that it's hard to prove that kind of causality). Major upgrades/replacements/migrations either failing or coming in a decade late and billions over budget is the norm in Fortune 500 companies, too, which leads to a hypothesis that it's a more general problem with big rewrites of safety/business-critical systems in large organizations.
jackgavigan 11y ago
In a previous life, I was a systems engineer at a company[0] that had several client with what would be charitably termed "legacy" systems, including SCO UnixWare and "Olivetti Unix". 0: https://news.ycombinator.com/item?id=10048801 https://news.ycombinator.com/item?id=10048801
- kjs3 11y ago
  Is that Olivetti X/OS or their SVR4 port. X/OS is really legacy.
  - jackgavigan 11y ago
    It was the late '90s so it was probably the SVR4 port (I actually think a lot of those "ports" were little more than rebranding).
    - kjs3 11y ago
      Yeah...pretty much everyone with a vaguely ISA/EISA/MCA box in those days just shipped a 99% unchanged reference port of i386 SVR4. It was interesting that one of the exceptions was, of all folks, Dell. They had a really nicely done SVR4 distribution with a hugely improved installer, a lot of open source stuff prebuilt and lots of bug fixes. Ran it for a while as my primary desktop and liked it alot. Shame Dell didn't stick with it.
hahamrfunnyguy 11y ago
The first part sounded actually kind of interesting, being stuck in this role for eternity would suck though.
notacoward 11y ago
How ancient this all seems . . . until you realize people are doing the exact same thing by shipping "golden" VM/container images around instead of doing proper package and configuration management.
- 1971genocide 11y ago
  As a engineer I do not have time to get into this new VM/container thing. I fail to see how it leads to anything productive ( no offence ). Call me a simpleton but I am happy with better package management and config systems.
  - DannoHung 11y ago
    Containers are pretty flexible (unfortunately?). They can be used as a stop-gap to everyone jumping ship to Nix (we can only pray to god) somewhere down the line in that they let you get some decent isolation properties out of existing package management. Since you can pretty much use any config management system you like with containers, they also play nice with that too. There is one other ancillary benefit in that they enforce a little bit tighter process isolation than you might get if you just installed a bunch of stuff on a given server. So that lets you do things like overcommit available resources (put more on the server than it can run at one time, but stochastically, it can probably handle) and it helps ensure you have "livestock" and not "pets" (servers configured automatically, repeatably, and with no concern with their demise rather than servers you need to baby). They're pretty much just a (mostly) handy abstraction for a bunch of things that you probably want to be doing but is exceptionally frustrating to set up piece by piece.
  - icedchai 11y ago
    "New" VM thing? It's only been around for eons now.
  - GFK_of_xmaspast 11y ago
    "I do not have time to get into this new .... thing" is pretty much the kind of thing that leads to problems like the original article.
mekael 11y ago
Ah legacy enterprise software/hardware, it keeps me employed and insane. Where i currently work we have a system which was originally designed in '69(coding started in 75) running on an ibm iSeries. It's old old tech and most of the system is cludged together so poorly that instead of being able to write patches we just write wrappers around the applications(micro services really) that modify the program as its running. Yeahp. We have a system written in something called "Workstation Basic" which was designed/written in '79( a decade before I was born). The original author no longer works for us and can only be contacted certain times of the year because he's a snowbird. Millions of dollars of business is done in access databases and custom made excel workbooks. It's a fun time.
- siavosh 11y ago
  It's an interesting time when one can build a living/breathing piece of engineering that can outlive us indefinitely.
  - smeyer 11y ago
    This isn't specific to computer or software engineering, of course. There are still Roman aqueducts in use that were engineered thousands of years ago.
    - mekael 11y ago
      I'm fascinated by things like the aqueducts. The skill to make them is something i can never comprehend.
      - grayclhn 11y ago
        Overengineeing doesn't take nearly as much skill as you'd think. It would be pretty easy for any of us to design a wall that, if left alone, would last for thousands of years.
        kragen 11y ago
SeanLuke 11y ago
Mine. Pick Basic, with only an ed-like line editor, in all caps, on Pick OS running inside an emulator called VMark UniVerse on HP/UX. I lasted a year.
- csixty4 11y ago
  Glad you got out of it so quickly. I spent five years in Pick at the turn of the century and it set my career back quite a bit. Just out of curiosity, how did you get out and what are you working on these days?
  - SeanLuke 11y ago
    > Just out of curiosity, how did you get out and what are you working on these days? I quit and went to grad school. I'm now a computer science professor.
    - csixty4 11y ago
      Awesome.
unoti 11y ago
It's the people that make a job good or bad; the technology plays a comparatively small role. I've worked with people I'd do anything for, including spending a couple years wading through old technology. You do it for the team, because you love your people. The same is true using new technology: it's easiest when you're doing it for your team.
- navanit 11y ago
  Or your end users. Everything becomes easier when you really care about the demographic you're serving.
hinkley 11y ago
Jesus, man. Company X? Company Y? Product Z? Variable naming is a very important skill.
- kevin_thibedeau 11y ago
  On legacy systems you may only have six significant characters for identifiers. Gotta scrounge for every last character.
  - me2i81 11y ago
    Six characters! Luxury. I worked on a system where we only had five.
    - rwallace 11y ago
      Five characters! Luxury. I learned to program on a system where we only had two. (Not joking: on the Commodore eight bit machines, only the first two characters of variable names were significant.)
      - cecilpl 11y ago
        Applesoft BASIC (for the Apple ][e) was like this too.
        StillBored 11y ago
        Yes, because both apple and commodore license microsoft basic as the core of their basic implementations.
      - mark-r 11y ago
        I learned to program on a BASIC system that had only one. Although strings were suffixed with $, so you could have both A and A$. Did I also mention that only upper case was available?
      - nocman 11y ago
peeters 11y ago
> That idea was nixed since the machine was old and fragile, we couldn't risk poking around in the inside. Well that pretty much sums up everything doesn't it. I don't see how a rational brain could produce that thought.
- egwor 11y ago
  I once worked somewhere where we never switched an old machine off. It was 15 years old and would cost £15K to replace. They were recommended to never switch it off - on the basis that it was most likely to fail on power up.
userbinator 11y ago
Aside from the bureaucracy, that actually sounds like a fun job to me, solving obscure problems and working with things almost no one has heard of, instead of churning out and maintaining boring "cookie-cutter" applications. Then again, I do like retrocomputing, reverse-engineering, and the demoscene... I imagined myself in a job interview in 2010, trying to explain how useful my extensive knowledge of Xenix, PL/M build systems, and VMS would be to my prospective new employer. You can phrase it as having gained that knowledge through your problem-solving skills. Working on obscure things, that you can't just Google the answer to when you get stuck, really helps build those skills.
- russell 11y ago
  I first read the term as "necrocomputing" instead of "retrocomputing". I think I have a preference for "necrocomputing", but then I may have been reading too much fantasy lately.
- stdbrouw 11y ago
  Wouldn't the sheer pointlessness of it all bother you after a while? If someone wanted to pay you to dig a ditch and fill it up again, for the right price, would you do it? I mean, I'm not trying to claim that working for that hot new startup which is going to revolutionize ___ or endlessly tweaking some megacorp's ERP system is necessarily such a boon to humanity, but still.
  - saalweachter 11y ago
    Most things we do are a little pointless in the end. In the long run, the universe will cool to uniformly distributed inert matter, and all of our efforts will scarcely affect the distribution. I currently work at big company X doing cutting edge work on the latest rewrite of big system Y, used by millions of people each day. In five years, the next rewrite will begin, and probably within ten years every last line of code I've written will be deprecated and deleted.
- Joeri 11y ago
  I expect the job of programmer archaeologist will become a proper category sometime in the next 50 years.
crystalmeph 11y ago
Not for technical reasons, but my worst job in software was also my first, a part time gig in college back in the late 90s. It was a software company owned by two people that didn't write code or do anything on a computer other than send email. They had somehow managed to hire somebody to make a moderately successful version 1.0 of their product, but now none of the original programmers were there and they were trying to come up with a brand new release. They mentioned during the interview that they had been "burned" by every programmer they'd ever hired. In retrospect, this was a red flag that the business owners were the only common denominator there, but again, this was my first ever software job, I had no idea what to expect. They stopped paying me after 1 month, and kept promising that I'd get paid as soon as the big contract they were pursuing landed. I quit shortly after I took a phone call from one of their creditors demanding that they fax over a check.
- mey 11y ago
  If this was in the US, you should seek back wages through your state.
  - crystalmeph 11y ago
    I got another job quickly, and it didn't hurt me too much financially (I was still living at home with my parents at the time). But you're right, I should have, if only to force them them out of business and out of the next guy's misery.
    - danielweber 11y ago
      It still may not be too late. Plus other employees who were screwed but unaware of their rights may automatically get checks.
      - vacri 11y ago
        You're suggesting legally chasing a few weeks salary after nearly 20 years? That's quite a grudge you're encouraging, and likely to cost much more than it would gain, even if it could be proven.
zem 11y ago
wow, my worst job infrastructure experience ever was when we were all forced to run a windows virtual machine to run outlook as our mail client, even though we were developing on linux and there was no real reason to prefer outlook over a linux mail client. (our company was making software for the windows vm market, and the ceo got the bright idea that everyone should run a windows vm to dogfood our product.) it was a pretty miserable experience, but this kind of puts it into perspective :)
bbcbasic 11y ago
Excellent! It feels like I am on the Daily WTF. A severe WTF followed by a thread of everyone's battle stories. I've worked on .NET most of my career. Biggest old-tech WTF is JS in IE5, or hours of fun getting those DCOM balls spinning again.
helmut_hed 11y ago
I do love war stories, but I think the OP has missed something: After the meeting one of the managers told me that it was really our job to come up with projects that the customer wanted to buy, not the other way around. And it usually couldn't just be a general project for minor improvements, it'd need clear and ideally measurable goals I think the OP has misunderstood the nature of consulting. This is exactly what needs to be done to convince X to spend some of their cash on having Y do work for them. You need to make a business case, with concrete duration, cost, goals, and benefits. That this system is a mess is really not in dispute by anyone... but no one will pay to improve it "just because".
markbnj 11y ago
I won't say it was my worst job ever, at all, but I can empathize. I joined a consulting company a few years back and ended up doing performance optimization on a legacy point of sale system written twenty years before, in C, for a large home improvement supplies company. Some of it actually proved interesting, but in terms of career moves it wasn't the best.
knodi123 11y ago
My first job in software was also my worst. I was learning a lot by building and enhancing a SaaS applicant tracking system for a small firm in Denton. But it was run by really shady people; our CEO would hire grossly unqualified people that he was friends with back in his home country... Our director of QA had previously been a high school history teacher, and had never used a computer before... it was nuts. We were also actively screwing one of our business partners. They sued us, and my boss considered me a very dangerous witness, so he sent me on a paid vacation so I couldn't be subpoenaed. Crazy place.
dan_m2k 11y ago
Not as vintage as this, but back in '10, I, along with a couple of other freelancers that I'd worked with in the past got asked to work in a <legacy clothing manufacturer> in a big, old mill in Yorkshire. We were immediately isolated from the modern-ish IT department and placed in a room at the end of the mill that contained a couple of dozen vacuum drive tape machines. So loud that they silenced the room when they were warming up. Our task was to "check for security problems" in their ecommerce platform because there was some serious financial motivation from the men upstairs. Their 'platform' was a decade-ish old install of osCommerce that had been hacked to death to make it support multi-side and multi language, together with some customisations to it's templating engine to provide asset reuse across sites. It was immediately apparent that all the SQL injection and most other vulnerabilities present years ago in that old version of OSc were still there. They were alongside the many, many vulns that were introduced with the hacks. Oh, and most of the hacks were done by a mixture of programmers on the continent, so variable names and comments were easy to understand. They didn't want to even consider upgrading, just patch it up and get outta there. Out of professionalism it was necessary to document the many times management passed on our recommendations to fix issues we found. Looking through that file, most of those issues seem to be present even today.
veddox 11y ago
I love it :-) What a time travel! I've been reading a bit about computing history of late, and that article sounds like it's straight from the pages of some old Jargon File entry...
sytringy05 11y ago
Great story! I worked at a large aerospace manufacturer who had their most important production systems (just for a single factory) running on an ancient VAX. It was written in the 60's and the original author's son, who was the sole person on the planet who knew anything detailed about it - he was the architect, the dev, the BA, the sysadmin and so on - was due to retire after 35 years with the company. The DR plan was similar to this - they had found a model of the same computer in a museum somewhere that still worked and they had dibs on it. In the end they shut down the factory, one of the major factors in that decision was this system. Just too hard to replace it
bigger_cheese 11y ago
My work is pretty bad. We have a box running windows 3.1 hooked up to an instrument on our plant. The answer I got when I asked about it was it has a proprietary ISA DAQ card in it with 'flaky' drivers' that prevents us upgrading it. It's a non critical instrument but is providing 'useful' data I think its some type of microwave radar measurement instrument - very specialised hardware. It's been working well since the 90's so not worth spending money until it fails. "Proprietary hardware cards" are notorious in industrial control system world they inevitably lock you into some ancient infrastruture when modern hardware and O/S's stop supporting ISA slots and things like that.
- cnvogel 11y ago
  As late as 2008 or so, when leaving university, there was still a PC running DOS for it's pulse height analyzer card for spectroscopy. Same thing: still works, and the pain to use it is not bad enough to warrant the $10k or whatever it takes to replace it with a completely new measurement system.... Which often won't even have better specs regarding the actual measurement.
- cptnbob 11y ago
  One of our clients still has a DOS box connected to a 14.4k modem that we have to drag EDI interchanges off. You wouldn't believe how much it cost to get a modem and POTS line installed in our DC so we could talk to it. There's a windows service written in .net that talks to it via a serial port and POSTs it to a REST endpoint.
  - cnvogel 11y ago
    Of course you could put a small internet connected embedded box to your clients place... (maybe even a commercial terminal server, if you trust the security of it) and directly emulate the whole modem+POTS line.
    - cptnbob 11y ago
      They paid for our modem setup so meh ;-)
andreapaiola 11y ago
In 2007 for a big bank I fixed an intranet site for a version of Mozilla browser (pre-Firefox) that run on an old version of Debian that were used in their branches.
totallystupid 11y ago
This is so funny, thank you.
DonHopkins 11y ago
Would the author mind sharing that tape dump from an old ITS system please?
- hga 11y ago
  See if you can get Devon, plus the people worried about privacy etc., to get their act together.
  - DonHopkins 11y ago
    I remember running across a Turing machine emulator implemented in TECO in Minsky's home directory that I'd REALLY love to get ahold of.
    - hga 11y ago
      Ask and yea shall receive: MSG: APL 1 DISTRIB: *BBOARD EXPIRES: 03/17/81 23:08:54 MINSKY@MIT-MC 03/11/81 23:08:54 Re: too-short programs APL is compact, I suppose. So is TECO. When I wrote the following Universal Turing Machine, which works, I actually understood it. [ I've interpolated the non-printing characters as displayed by (Gnu) EMACS, escape is ^], ^^ is one character, as is \356: ] i1Aul qq+^^0:iqm^[29iiq\356y0L1 00L1 11L2 A1L1 y0L1 0yR2 1AR2 AyR6 yyL3 00L0 1AL3 A1L4 yyL4 0yR5 11L7 A1L4 yyR5 0yL3 1AR5 A1R5 yyR6 0AL3 1AR6 A1R6 y0R7 0yR6 11R7 A0R2 ^[j<sR^[;-d-2ciql-^^^[ci"ed^^^[cii^[ciuq'^[> j<sL^[;-d-2ciql-^^^[ci"ed^^^[cii-2c^[ciuq'^[>jxblx1lx2lx3lx4lx5lx6lx7hk iyyAyyAyy^[32<i0^[>ji110101110000010011011^[ 1uq<htmbqq=> I do not advise attempting to understand this code, which is almost as bad as that for the Universal Turing machine. ADDED: or http://ancell-ent.com/share/minsky-TECO-turing-machine.txt http://ancell-ent.com/share/minsky-TECO-turing-machine.txt Please ack receipt of this and/or send me email (in my HN info); for others, note this is ITS TECO, which I was told was by far the most powerful version of it (fortunately, by the time I showed up learning it was no longer really necessary).
      - DonHopkins 11y ago
        OOP ACK! It was a shot in the dark, but I am SO GLAD I asked!!! Thank you Harold! It looks just like I remember. ;)
amyjess 11y ago
I went to college at the University of Texas at Dallas. Our online Student Information System (SIS) was one of the very rare online services with operating hours. It wasn't available at night (and possibly on weekends too, but I graduated eight years ago, so my memory is fuzzy). At one point, I started asking around, and I found out that it was because SIS ran on an old mainframe that couldn't read and write the disk at the same time. During operating hours, it reads the disk and makes all changes in memory. When operating hours are over, it writes all its changes to disk. This was in 2003-2007 and they still had that mainframe controlling all their class schedules and student data. UTD was considered the biggest tech-oriented university in the southwest, too.
hadeharian 11y ago
What people don't always understand is outside of the tech world, there are people who still think computer upgrades are unnecessary expenditures. It's true that the theoretical lifetime of a computer is probably something like 50 years, minus the hard drive replacement and the power supply which will generally go bad much quicker. The more difficult thing is in a field like aeronautics any minor changes have the possibility of generating catastrophic events. In these situations, the cost of replacement, and testing, and phasing stuff in is enormously high and the risk is obscene when change needs to be made.
Neil_in_Chicago 11y ago
Ten or fifteen years ago, I spent a few years maintaining a "system" in FoxPro for unix on bastard hardware . . . I once had to maintain a system using "Westi" 1.1, Westighouse's knockoff of CICS . . . The day I completed my first serious project and linked it, I crashed the company because the operating system was named "test", so I'd just overwritten it. About the time I was fired, I figured out that the entire system was bootleg. IF NO ONE WILL ADMIT EVEN KNOWING WHO BOUGHT THE IDIOTIC PILE OF MISMATCHED PARTS, WALK OUT IMMEDIATELY.

464 points by kryptiskt 11y ago

Not nearly as bad, but this reminds me of some of the ridiculous requirements at the last program I worked at my previous employer. One of those was that our code had to compile on a 32-bit SPARC pizza box running Solaris 2.6. This was in 2009. It was all about some stupid metric: our code portability was measured by how many different platforms we could compile and run on. It didn't matter how obsolete or inappropriate those platforms were; if we could compile and run it counted. Having that number go down was politically unacceptable.

Vexs 11y ago
Reminds me a bit of VIM. I love vim as much as the next guy, but I can't wait for neovim to get stable.
- afarrell 11y ago
  I was running neovim without issue on OSX until about a month ago when I switched to nuclide
nailer 11y ago
There's something strange about Solaris. Around the same time, an investment bank (where the Solaris team constantly justified their existence with 'dtrace!' 'zfs!' 'zones!') was paying 50K a year per server multiplied by hundreds of servers to run Solaris 8 extended support - which didn't have any of those things, which is good, because the staff didn't know how to use them other than say they made Solaris great.
- trhway 11y ago
  >where the Solaris team constantly justified their existence with 'dtrace!' 'zfs!' 'zones!' they did that inside Sun as well. Most noise was about DTrace, the most useless thing for the Sun/Solaris at that moment.
  - mattzito 11y ago
    Dtrace was, and still is, awesome. Linux has come a long way in this area in the last few years, but DTrace was one of the hottest systems management capabilities I'd seen in forever, particularly on the very large SMP systems that had become Solaris's sweet spot in those waning years.
    - gaius 11y ago
      Quite. Systemtap is a joke compared to dtrace.
      - nailer 11y ago
        There are reasonable criticisms of Systemtap, but a lot of Sun marketing was just bizarre. They set it up on Ubuntu (which doesn't include it, and which isn't their competitor: Red Hat is) as an example of what the setup is like. They kept complaining it was unsafe, when it was only unsafe if you wanted to use raw C for tapsets. Sun had some great tech, but they were also a sales and marketing machine that had no problems spouting garbage to make a sale.
  - DonHopkins 11y ago
    I miss fastboot.
rjsw 11y ago
TBF, testing on SPARC is a good way of finding endian and alignment bugs.
- vonmoltke 11y ago
  It would be, if our two target platforms hadn't already been x86-64 and POWER.
kevin_thibedeau 11y ago
I had to port a product to Solaris 8 on similarly crusty hardware in 2011. Wasn't so bad other than having to work around an excruciatingly slow build process.

api 11y ago

More like Not Invented Here syndrome to the level of fatal pathology.

Sanddancer 11y ago
It's its sibling, Already Invented Here. Every kludge makes sense in the context of the system state when it was implemented, but that leads to a system where /everything/ ends up a kludge. Management didn't want to invest any time in fixing a system that already "worked", so 30 years of temporary fixes turned it into an absolute monstrosity.
- ethbro 11y ago
  Both of these are important observations. And, as far as I can tell, both of them exist largely because of corporate politics. If you could design a corporate culture that minimized the incentives towards both, I think you'd be a long way towards having a more sustainable company viz technology.
  - webjprgm 11y ago
    There was one time (2006 at a company of less than 50 people) where I absolutely refused to build the new pricing features in a billing system (which would have been a big hack on a long series of smaller hacks) unless they let me refactor it (meaning rebuild it from scratch). After waiting for a few months without their new pricing features they finally gave in. That's at a young, nimble company where the co-CEO had a programming background. I can only imagine how hard a fight that would be at a larger, slower company.
    - api 11y ago
      I've worked for a government data center where they ran instances of Ultrix in a commercial VM to run old data processing software that ingested and barfed out CSVs that contained UUencoded blobs of image data. All this was written in a mixture of very very old Perl, shell scripts in some gawd-awful Ultrix shell CSH script, and C code full of weird -isms that made it difficult to port to anything made after the rise of mammals. This was of course mission critical. At least they virtualized it instead of continuing to run physical Ultrix boxes, which was apparently a huge fight with management. A side dish of LULZ was that they were experimenting with moving the VMs to EC2, so there is probably Ultrix in EC2 now. This stuff is really, really common in large "mature" industries and government.

PaulHoule 11y ago

The only way to stop this madness is to go on strike against it, as you did. I work very hard to never mention online that I've had anything to do with SharePoint or ColdFusion because it will bring recruiters with a job that will kill you.

foobarbecue 11y ago
Not hard enough, apparently.
mring33621 11y ago
Dear %prospect_name%; I was in our recruiting DB the other day, and came across an old version of your résumé. I am very impressed with your background and think you might be a perfect fit for an opening that we have for a Senior ColdFusion Developer. Please get back to me at your earliest convenience regarding this exciting opportunity. Thank you. Sincerely, %recruiter_sig%
- Kalium 11y ago
  You're going to give me nightmares. I still get emails like this from a six-year-old version of my resume.
  - ErikRogneby 11y ago
    Where is the national do-not-call list for clueless tech recruiters?
    - Kalium 11y ago
      I don't know. And half of appear to be operating out of India anyway, judging by the noise in the background on the lines.
    - rogerbinns 11y ago
      I maintain one at http://www.rogerbinns.com/blocked_recruiters.txt http://www.rogerbinns.com/blocked_recruiters.txt My resume has a "note to recruiters" attached. It is very clear if a recruiter has made any effort to read any of that. The clueless ones end up in the list above. http://www.rogerbinns.com/recruiters.html http://www.rogerbinns.com/recruiters.html

nasalgoat 11y ago

Although nothing compared to what was in this story, you should see the level of obsolete crap in place at adult website providers. Systems that haven't changed since the 90's that still generate revenue, so they'll never be shut off until there's a flood or a 40MB hard drive finally dies.

kenko 11y ago
Have to admit, that approach makes a lot of sense to me. It's still chugging along and bringing in $, so why not just have a policy of benevolent neglect?
- nasalgoat 11y ago
  That Pentium-90 is probably sucking a ton of power at this point, when it could be virtualized and you could run 50 of those boxes on an $800 Dell pizza box. I mean, they are literally running the same hardware from the 90s.
  - wtallis 11y ago
    That Pentium 90 is a 9W processor. Even after accounting for the inefficiency of a 90's era PSU, there's not much power to be saved.
    - nasalgoat 11y ago
      Plus the hard drive(s), the space it's taking up in the DC, the cooling overhead, etc.
      - wtallis 11y ago
        There's not really any significant room for improvement on any of those factors. You might be able to save a rack unit or two, and several watts by replacing hard drives with a SSD, but if you don't have 50 boxes to consolidate, you're still facing the baseline cost of having a box taking up some amount of physical space and requiring several watts of power. Unless you make the much bigger up-front investment of migrating to a shared hosting platform.

russnewcomer 11y ago

I think the most interesting lessons of this story have to do with the following expectations. First, the expectation of support timelines. It seems in tech that support is promised for products far beyond when the vendor should reasonably be expected to support them. Companies make large investments in technology that ultimately aren't long-term enough. Lesson: Any company doing a major project where there is significant cost (as a percentage of the overall cost) in hardware or software, and that selected hardware or software is not commodity level or has low market penetration in their industry should seriously reconsider the projections of TCO, as long-term support will get increasingly expensive over the life of the project (Contrived example, supporting Windows XP via virtualization vs OS/2 Warp) A second expectation is the timeline of viability for technologies. Non-technical business people of the past generally seemed to have an expectation of viability that was around 15-20 years. More recently, stated expectations seem shorter, but are actually not. Lesson: Businesses that make serious investments in technology need to consider length of that investment, and technical people should probably take those estimates and double or triple them, because that's probably how long hardware/software will be in use. Companies providing technology or support need to have realistic timeframes of support. As a corollary to this lesson, I have spent some of my career thus far doing IT support-type roles for small organizations, and while I haven't encountered quite the same level of obsolescence as the author, I have seen people using NT 4 for critical server infrastructure in 2012, I've set up DOSBOX[1] in Windows 7 for clients to continue to use that version of Lotus 1-2-3 they paid for and have been using since 1987 , and I've tried to figure out, in 2011, why Netscape Navigator on, I think System 7, ( using dial-up! ) wouldn't properly render a webpage. People keep tech a long time, and people in the tech world need to expect that, and plan accordingly. The final expectation is one that technology companies have, which is that tech should be replaced. While there are many use cases where tech gets old, breaks, has to be replaced, there are lots of cases where it can be functional for almost indefinite periods of time, and industries that will expect such. Lesson: Tech companies in should not neglect the future, but should also strive to design objects of technology that can last a very, very long time. Companies that do so will find a niche and make money because there will always[2] be enough customers to support them. [1]It was going to get some use on an iPad via LogMeIn and there was some reason I've forgotten why DOSBOX was a better choice than the command prompt [2] Subject to prevailing economic winds, of course. You can still buy buggies and buggy whips, but it's not a growth business.

JohnTHaller 11y ago

I was completely expecting 8 inch floppies or punch cards to show up once I was half way through reading.

sthorn 11y ago
LOL, we debugged our PL/M with an Intel ICE ("blue box") that used 8" floppies. Good times.
- jsnell 11y ago
  One of these? http://www.computinghistory.org.uk/det/731/Intel-MDS-80-Microcomputer-Development-System/ http://www.computinghistory.org.uk/det/731/Intel-MDS-80-Micr... Wow, that looks just amazing.

ddoscampaign 11y ago

Reminds me of a contract gig I passed on to support Microsoft's creaky inheritance of Danger's Hiptop/Sidekick stuck in cheapest maintenance mode. Everyone associated were trying to rotate out and it seemed like morale was less than zero. Also never work for slave-driving shops like Taos (aka Tause Mountain) unless you really need the money because it will crush your soul. (eg master cool techs and be awesomely personable to get cooler gigs/referrals.)

rietta 11y ago
That transition was a real mess. We had some games in the T-Mobile Sidekick store and getting paid the royalties by Microsoft was way more difficult than it should have been after the transition. It took months to get added as a vendor and I had to even send them a "final invoice" using the numbers that they had provided. The next collection step was going to be at least small claims court. The Danger team did built a great system for its time and by the time Microsoft took over, it went downhill fast. And then Microsoft's Kin flopped too.
- hga 11y ago
  The Danger team did built a great system for its time and by the time Microsoft took over, it went downhill fast. Microsoft managed one of the biggest cloud computing screw ups in history to date: https://en.wikipedia.org/wiki/2009_Sidekick_data_loss https://en.wikipedia.org/wiki/2009_Sidekick_data_loss The incident caused a public loss of confidence in the concept of cloud computing, which had been plagued by a series of outages and data losses in 2009. It also was problematic for Microsoft, which at the time was trying to convince corporate clients to use its cloud computing services, such as Azure and My Phone. I've heard good things about Oracle's RAC, but it's understandably intolerant of your screwing up its disks (SAN mis/re-configuring) when you aren't properly maintaining backups. I also heard the consultants you have to hire after you manage such a feat are expensive.
  - bro-stick 11y ago
    Yup. I managed some dataguard (not RAC) instances on AWS for Palm pre HP. Thankfully we had DR plans and snapshots to cover our asses. Edit: Fixed HA techs.

Theodores 11y ago

I supported booking systems that could have bookings stretching far into the future, this made switching systems tricky as a lot of quite complex bookings would need recreating with each taking proper time to do. The scope for automation was there but that wasn't possible. With the software we supported there was no expertise in the company as the original developers left a long time ago. Therefore there was no way to understand why something broke or to write a patch. Instead everything was fixed by rote, if this went wrong then you did this. If that went wrong then you did whatever was needed for that. All of these fixes were handed down in the oral tradition and you took your own notes - there was no manual. We further out sourced to India so there was no trace of the people responsible for the code, just us clueless idiots. Some of these systems were MS DOS based and this was about a decade ago. What I did manage though was a means of getting on to any computer anywhere in the world even if it was totally air gapped and owned by someone that did not speak English. Even better, in most cases this would take seconds from a standing start. If I needed to access a computer that was on a remote network but turned off, I could usually Magic Packet it on and be fixing it as soon as it booted. This was a golden era for my remote access skills, nowadays I have difficulties getting my own computer online! There is another way of looking at this, from the perspective of what will be out there for you to do in 25 years time, I am optimistic that there will be legacy work for us if we cannot keep up with the new stuff.

rwallace 11y ago
> What I did manage though was a means of getting on to any computer anywhere in the world even if it was totally air gapped and owned by someone that did not speak English. Even better, in most cases this would take seconds from a standing start. I'm curious - how did you do this?

myth_buster 11y ago

I'm wondering whether the use of ingenious (adj) (of a person) clever, original, and inventive. in place of indigenous (adj) originating or occurring naturally in a particular place; native. is a Freudian slip or sarcasm. Edit: Never mind; they used it with in-house

kelvin0 11y ago
Sarcasm would be my bet ...

russell 11y ago

As soon as I saw the title, I thought of the furor when Gingrich and Clinton discovered that the FAA was using vacuum tube computers and radar in the air traffic control system. http://govinfo.library.unt.edu/npr/library/clinton.htm http://govinfo.library.unt.edu/npr/library/clinton.htm The foreign (sole) vacuum tube supplier that Clinton mentioned was Russia. Oh, the embarrassment! I knew that there were some delays in rolling out the the replacement system, so I did some googling to see when it actually occurred. My god, it's now. We are in the middle of it. The upgrade was responsible for the recent air traffic meltdown. Please somebody, point out my flawed research skills. I cant believe this.

steve-howard 11y ago
I found this article from Wired earlier this year: http://www.wired.com/2015/02/air-traffic-control/ http://www.wired.com/2015/02/air-traffic-control/ It doesn't specificaly mention vacuum tubes, but it does explain why it's entirely plausible that they're still around: >This technology is complicated and novel, but that isn't the problem. The problem is that NextGen is a project of the FAA. The agency is primarily a regulatory body, responsible for keeping the national airspace safe, and yet it is also in charge of operating air traffic control, an inherent conflict that causes big issues when it comes to upgrades. Modernization, a struggle for any federal agency, is practically antithetical to the FAA's operational culture, which is risk-averse, methodical, and bureaucratic. Paired with this is the lack of anything approximating market pressure. The FAA is the sole consumer of the product; it's a closed loop.
- _delirium 11y ago
  I'm skeptical that the culprit is mainly to be found in specific features of the FAA, and the article doesn't seem to provide much causal evidence of that (though I recognize that it's hard to prove that kind of causality). Major upgrades/replacements/migrations either failing or coming in a decade late and billions over budget is the norm in Fortune 500 companies, too, which leads to a hypothesis that it's a more general problem with big rewrites of safety/business-critical systems in large organizations.

jackgavigan 11y ago

In a previous life, I was a systems engineer at a company[0] that had several client with what would be charitably termed "legacy" systems, including SCO UnixWare and "Olivetti Unix". 0: https://news.ycombinator.com/item?id=10048801 https://news.ycombinator.com/item?id=10048801

kjs3 11y ago
Is that Olivetti X/OS or their SVR4 port. X/OS is really legacy.
- jackgavigan 11y ago
  It was the late '90s so it was probably the SVR4 port (I actually think a lot of those "ports" were little more than rebranding).
  - kjs3 11y ago
    Yeah...pretty much everyone with a vaguely ISA/EISA/MCA box in those days just shipped a 99% unchanged reference port of i386 SVR4. It was interesting that one of the exceptions was, of all folks, Dell. They had a really nicely done SVR4 distribution with a hugely improved installer, a lot of open source stuff prebuilt and lots of bug fixes. Ran it for a while as my primary desktop and liked it alot. Shame Dell didn't stick with it.

hahamrfunnyguy 11y ago

The first part sounded actually kind of interesting, being stuck in this role for eternity would suck though.

notacoward 11y ago

How ancient this all seems . . . until you realize people are doing the exact same thing by shipping "golden" VM/container images around instead of doing proper package and configuration management.

1971genocide 11y ago
As a engineer I do not have time to get into this new VM/container thing. I fail to see how it leads to anything productive ( no offence ). Call me a simpleton but I am happy with better package management and config systems.
- DannoHung 11y ago
  Containers are pretty flexible (unfortunately?). They can be used as a stop-gap to everyone jumping ship to Nix (we can only pray to god) somewhere down the line in that they let you get some decent isolation properties out of existing package management. Since you can pretty much use any config management system you like with containers, they also play nice with that too. There is one other ancillary benefit in that they enforce a little bit tighter process isolation than you might get if you just installed a bunch of stuff on a given server. So that lets you do things like overcommit available resources (put more on the server than it can run at one time, but stochastically, it can probably handle) and it helps ensure you have "livestock" and not "pets" (servers configured automatically, repeatably, and with no concern with their demise rather than servers you need to baby). They're pretty much just a (mostly) handy abstraction for a bunch of things that you probably want to be doing but is exceptionally frustrating to set up piece by piece.
- icedchai 11y ago
  "New" VM thing? It's only been around for eons now.
- GFK_of_xmaspast 11y ago
  "I do not have time to get into this new .... thing" is pretty much the kind of thing that leads to problems like the original article.

mekael 11y ago

Ah legacy enterprise software/hardware, it keeps me employed and insane. Where i currently work we have a system which was originally designed in '69(coding started in 75) running on an ibm iSeries. It's old old tech and most of the system is cludged together so poorly that instead of being able to write patches we just write wrappers around the applications(micro services really) that modify the program as its running. Yeahp. We have a system written in something called "Workstation Basic" which was designed/written in '79( a decade before I was born). The original author no longer works for us and can only be contacted certain times of the year because he's a snowbird. Millions of dollars of business is done in access databases and custom made excel workbooks. It's a fun time.

siavosh 11y ago
It's an interesting time when one can build a living/breathing piece of engineering that can outlive us indefinitely.
- smeyer 11y ago
  This isn't specific to computer or software engineering, of course. There are still Roman aqueducts in use that were engineered thousands of years ago.
  - mekael 11y ago
    I'm fascinated by things like the aqueducts. The skill to make them is something i can never comprehend.
    - grayclhn 11y ago
      Overengineeing doesn't take nearly as much skill as you'd think. It would be pretty easy for any of us to design a wall that, if left alone, would last for thousands of years.
      - kragen 11y ago

SeanLuke 11y ago

Mine. Pick Basic, with only an ed-like line editor, in all caps, on Pick OS running inside an emulator called VMark UniVerse on HP/UX. I lasted a year.

csixty4 11y ago
Glad you got out of it so quickly. I spent five years in Pick at the turn of the century and it set my career back quite a bit. Just out of curiosity, how did you get out and what are you working on these days?
- SeanLuke 11y ago
  > Just out of curiosity, how did you get out and what are you working on these days? I quit and went to grad school. I'm now a computer science professor.
  - csixty4 11y ago
    Awesome.

unoti 11y ago

It's the people that make a job good or bad; the technology plays a comparatively small role. I've worked with people I'd do anything for, including spending a couple years wading through old technology. You do it for the team, because you love your people. The same is true using new technology: it's easiest when you're doing it for your team.

navanit 11y ago
Or your end users. Everything becomes easier when you really care about the demographic you're serving.

hinkley 11y ago

Jesus, man. Company X? Company Y? Product Z? Variable naming is a very important skill.

kevin_thibedeau 11y ago
On legacy systems you may only have six significant characters for identifiers. Gotta scrounge for every last character.
- me2i81 11y ago
  Six characters! Luxury. I worked on a system where we only had five.
  - rwallace 11y ago
    Five characters! Luxury. I learned to program on a system where we only had two. (Not joking: on the Commodore eight bit machines, only the first two characters of variable names were significant.)
    - cecilpl 11y ago
      Applesoft BASIC (for the Apple ][e) was like this too.
      - StillBored 11y ago
        Yes, because both apple and commodore license microsoft basic as the core of their basic implementations.
    - mark-r 11y ago
      I learned to program on a BASIC system that had only one. Although strings were suffixed with $, so you could have both A and A$. Did I also mention that only upper case was available?
    - nocman 11y ago

peeters 11y ago

> That idea was nixed since the machine was old and fragile, we couldn't risk poking around in the inside. Well that pretty much sums up everything doesn't it. I don't see how a rational brain could produce that thought.

egwor 11y ago
I once worked somewhere where we never switched an old machine off. It was 15 years old and would cost £15K to replace. They were recommended to never switch it off - on the basis that it was most likely to fail on power up.

userbinator 11y ago

Aside from the bureaucracy, that actually sounds like a fun job to me, solving obscure problems and working with things almost no one has heard of, instead of churning out and maintaining boring "cookie-cutter" applications. Then again, I do like retrocomputing, reverse-engineering, and the demoscene... I imagined myself in a job interview in 2010, trying to explain how useful my extensive knowledge of Xenix, PL/M build systems, and VMS would be to my prospective new employer. You can phrase it as having gained that knowledge through your problem-solving skills. Working on obscure things, that you can't just Google the answer to when you get stuck, really helps build those skills.

russell 11y ago
I first read the term as "necrocomputing" instead of "retrocomputing". I think I have a preference for "necrocomputing", but then I may have been reading too much fantasy lately.
stdbrouw 11y ago
Wouldn't the sheer pointlessness of it all bother you after a while? If someone wanted to pay you to dig a ditch and fill it up again, for the right price, would you do it? I mean, I'm not trying to claim that working for that hot new startup which is going to revolutionize ___ or endlessly tweaking some megacorp's ERP system is necessarily such a boon to humanity, but still.
- saalweachter 11y ago
  Most things we do are a little pointless in the end. In the long run, the universe will cool to uniformly distributed inert matter, and all of our efforts will scarcely affect the distribution. I currently work at big company X doing cutting edge work on the latest rewrite of big system Y, used by millions of people each day. In five years, the next rewrite will begin, and probably within ten years every last line of code I've written will be deprecated and deleted.
Joeri 11y ago
I expect the job of programmer archaeologist will become a proper category sometime in the next 50 years.

crystalmeph 11y ago

Not for technical reasons, but my worst job in software was also my first, a part time gig in college back in the late 90s. It was a software company owned by two people that didn't write code or do anything on a computer other than send email. They had somehow managed to hire somebody to make a moderately successful version 1.0 of their product, but now none of the original programmers were there and they were trying to come up with a brand new release. They mentioned during the interview that they had been "burned" by every programmer they'd ever hired. In retrospect, this was a red flag that the business owners were the only common denominator there, but again, this was my first ever software job, I had no idea what to expect. They stopped paying me after 1 month, and kept promising that I'd get paid as soon as the big contract they were pursuing landed. I quit shortly after I took a phone call from one of their creditors demanding that they fax over a check.

mey 11y ago
If this was in the US, you should seek back wages through your state.
- crystalmeph 11y ago
  I got another job quickly, and it didn't hurt me too much financially (I was still living at home with my parents at the time). But you're right, I should have, if only to force them them out of business and out of the next guy's misery.
  - danielweber 11y ago
    It still may not be too late. Plus other employees who were screwed but unaware of their rights may automatically get checks.
    - vacri 11y ago
      You're suggesting legally chasing a few weeks salary after nearly 20 years? That's quite a grudge you're encouraging, and likely to cost much more than it would gain, even if it could be proven.

zem 11y ago

wow, my worst job infrastructure experience ever was when we were all forced to run a windows virtual machine to run outlook as our mail client, even though we were developing on linux and there was no real reason to prefer outlook over a linux mail client. (our company was making software for the windows vm market, and the ceo got the bright idea that everyone should run a windows vm to dogfood our product.) it was a pretty miserable experience, but this kind of puts it into perspective :)

bbcbasic 11y ago

Excellent! It feels like I am on the Daily WTF. A severe WTF followed by a thread of everyone's battle stories. I've worked on .NET most of my career. Biggest old-tech WTF is JS in IE5, or hours of fun getting those DCOM balls spinning again.

helmut_hed 11y ago

I do love war stories, but I think the OP has missed something: After the meeting one of the managers told me that it was really our job to come up with projects that the customer wanted to buy, not the other way around. And it usually couldn't just be a general project for minor improvements, it'd need clear and ideally measurable goals I think the OP has misunderstood the nature of consulting. This is exactly what needs to be done to convince X to spend some of their cash on having Y do work for them. You need to make a business case, with concrete duration, cost, goals, and benefits. That this system is a mess is really not in dispute by anyone... but no one will pay to improve it "just because".

markbnj 11y ago

I won't say it was my worst job ever, at all, but I can empathize. I joined a consulting company a few years back and ended up doing performance optimization on a legacy point of sale system written twenty years before, in C, for a large home improvement supplies company. Some of it actually proved interesting, but in terms of career moves it wasn't the best.

knodi123 11y ago

My first job in software was also my worst. I was learning a lot by building and enhancing a SaaS applicant tracking system for a small firm in Denton. But it was run by really shady people; our CEO would hire grossly unqualified people that he was friends with back in his home country... Our director of QA had previously been a high school history teacher, and had never used a computer before... it was nuts. We were also actively screwing one of our business partners. They sued us, and my boss considered me a very dangerous witness, so he sent me on a paid vacation so I couldn't be subpoenaed. Crazy place.

dan_m2k 11y ago

Not as vintage as this, but back in '10, I, along with a couple of other freelancers that I'd worked with in the past got asked to work in a <legacy clothing manufacturer> in a big, old mill in Yorkshire. We were immediately isolated from the modern-ish IT department and placed in a room at the end of the mill that contained a couple of dozen vacuum drive tape machines. So loud that they silenced the room when they were warming up. Our task was to "check for security problems" in their ecommerce platform because there was some serious financial motivation from the men upstairs. Their 'platform' was a decade-ish old install of osCommerce that had been hacked to death to make it support multi-side and multi language, together with some customisations to it's templating engine to provide asset reuse across sites. It was immediately apparent that all the SQL injection and most other vulnerabilities present years ago in that old version of OSc were still there. They were alongside the many, many vulns that were introduced with the hacks. Oh, and most of the hacks were done by a mixture of programmers on the continent, so variable names and comments were easy to understand. They didn't want to even consider upgrading, just patch it up and get outta there. Out of professionalism it was necessary to document the many times management passed on our recommendations to fix issues we found. Looking through that file, most of those issues seem to be present even today.

veddox 11y ago

I love it :-) What a time travel! I've been reading a bit about computing history of late, and that article sounds like it's straight from the pages of some old Jargon File entry...

sytringy05 11y ago

Great story! I worked at a large aerospace manufacturer who had their most important production systems (just for a single factory) running on an ancient VAX. It was written in the 60's and the original author's son, who was the sole person on the planet who knew anything detailed about it - he was the architect, the dev, the BA, the sysadmin and so on - was due to retire after 35 years with the company. The DR plan was similar to this - they had found a model of the same computer in a museum somewhere that still worked and they had dibs on it. In the end they shut down the factory, one of the major factors in that decision was this system. Just too hard to replace it

bigger_cheese 11y ago

My work is pretty bad. We have a box running windows 3.1 hooked up to an instrument on our plant. The answer I got when I asked about it was it has a proprietary ISA DAQ card in it with 'flaky' drivers' that prevents us upgrading it. It's a non critical instrument but is providing 'useful' data I think its some type of microwave radar measurement instrument - very specialised hardware. It's been working well since the 90's so not worth spending money until it fails. "Proprietary hardware cards" are notorious in industrial control system world they inevitably lock you into some ancient infrastruture when modern hardware and O/S's stop supporting ISA slots and things like that.

cnvogel 11y ago
As late as 2008 or so, when leaving university, there was still a PC running DOS for it's pulse height analyzer card for spectroscopy. Same thing: still works, and the pain to use it is not bad enough to warrant the $10k or whatever it takes to replace it with a completely new measurement system.... Which often won't even have better specs regarding the actual measurement.
cptnbob 11y ago
One of our clients still has a DOS box connected to a 14.4k modem that we have to drag EDI interchanges off. You wouldn't believe how much it cost to get a modem and POTS line installed in our DC so we could talk to it. There's a windows service written in .net that talks to it via a serial port and POSTs it to a REST endpoint.
- cnvogel 11y ago
  Of course you could put a small internet connected embedded box to your clients place... (maybe even a commercial terminal server, if you trust the security of it) and directly emulate the whole modem+POTS line.
  - cptnbob 11y ago
    They paid for our modem setup so meh ;-)

andreapaiola 11y ago

In 2007 for a big bank I fixed an intranet site for a version of Mozilla browser (pre-Firefox) that run on an old version of Debian that were used in their branches.

totallystupid 11y ago

This is so funny, thank you.

DonHopkins 11y ago

Would the author mind sharing that tape dump from an old ITS system please?

hga 11y ago
See if you can get Devon, plus the people worried about privacy etc., to get their act together.
- DonHopkins 11y ago
  I remember running across a Turing machine emulator implemented in TECO in Minsky's home directory that I'd REALLY love to get ahold of.
  - hga 11y ago
    Ask and yea shall receive: MSG: APL 1 DISTRIB: *BBOARD EXPIRES: 03/17/81 23:08:54 MINSKY@MIT-MC 03/11/81 23:08:54 Re: too-short programs APL is compact, I suppose. So is TECO. When I wrote the following Universal Turing Machine, which works, I actually understood it. [ I've interpolated the non-printing characters as displayed by (Gnu) EMACS, escape is ^], ^^ is one character, as is \356: ] i1Aul qq+^^0:iqm^[29iiq\356y0L1 00L1 11L2 A1L1 y0L1 0yR2 1AR2 AyR6 yyL3 00L0 1AL3 A1L4 yyL4 0yR5 11L7 A1L4 yyR5 0yL3 1AR5 A1R5 yyR6 0AL3 1AR6 A1R6 y0R7 0yR6 11R7 A0R2 ^[j<sR^[;-d-2ciql-^^^[ci"ed^^^[cii^[ciuq'^[> j<sL^[;-d-2ciql-^^^[ci"ed^^^[cii-2c^[ciuq'^[>jxblx1lx2lx3lx4lx5lx6lx7hk iyyAyyAyy^[32<i0^[>ji110101110000010011011^[ 1uq<htmbqq=> I do not advise attempting to understand this code, which is almost as bad as that for the Universal Turing machine. ADDED: or http://ancell-ent.com/share/minsky-TECO-turing-machine.txt http://ancell-ent.com/share/minsky-TECO-turing-machine.txt Please ack receipt of this and/or send me email (in my HN info); for others, note this is ITS TECO, which I was told was by far the most powerful version of it (fortunately, by the time I showed up learning it was no longer really necessary).
    - DonHopkins 11y ago
      OOP ACK! It was a shot in the dark, but I am SO GLAD I asked!!! Thank you Harold! It looks just like I remember. ;)

amyjess 11y ago

I went to college at the University of Texas at Dallas. Our online Student Information System (SIS) was one of the very rare online services with operating hours. It wasn't available at night (and possibly on weekends too, but I graduated eight years ago, so my memory is fuzzy). At one point, I started asking around, and I found out that it was because SIS ran on an old mainframe that couldn't read and write the disk at the same time. During operating hours, it reads the disk and makes all changes in memory. When operating hours are over, it writes all its changes to disk. This was in 2003-2007 and they still had that mainframe controlling all their class schedules and student data. UTD was considered the biggest tech-oriented university in the southwest, too.

hadeharian 11y ago

What people don't always understand is outside of the tech world, there are people who still think computer upgrades are unnecessary expenditures. It's true that the theoretical lifetime of a computer is probably something like 50 years, minus the hard drive replacement and the power supply which will generally go bad much quicker. The more difficult thing is in a field like aeronautics any minor changes have the possibility of generating catastrophic events. In these situations, the cost of replacement, and testing, and phasing stuff in is enormously high and the risk is obscene when change needs to be made.

Neil_in_Chicago 11y ago

Ten or fifteen years ago, I spent a few years maintaining a "system" in FoxPro for unix on bastard hardware . . . I once had to maintain a system using "Westi" 1.1, Westighouse's knockoff of CICS . . . The day I completed my first serious project and linked it, I crashed the company because the operating system was named "test", so I'd just overwritten it. About the time I was fired, I figured out that the entire system was bootleg. IF NO ONE WILL ADMIT EVEN KNOWING WHO BOUGHT THE IDIOTIC PILE OF MISMATCHED PARTS, WALK OUT IMMEDIATELY.