6 ms·

My worst codebase story: In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java. In

My worst codebase story: In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java. In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server. Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files). Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released. This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files. Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why. It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.

sir-dingleberry 2y ago
This is insane. Thanks for the post.
pelagicAustral 2y ago
This sounds so much like dealing with MS Access databases... Unfortunately, part of my responsibility in my current role is to manage a handful of Access applications... they are ancient, I am talking early 2000's. They are the most unruly thing I have ever had to work with, and I will not go into details but your story reminds me so much of having to work on these apps...
- DaiPlusPlus 2y ago
  MSAccess is a tragedy, imo. Right-up until, say, Access 2007, it was a simple (in a good way!) RAD platform following those zombie 4GL predecessors, but it’s been left to stagnate without any real effort to modernise it, and it’s clear from how Microsoft’s been downplaying Access that it’s a product they’d really rather not have to support, but they know if they do kill it then it would weaken on-prem Office’s moat and everyone will probably just rush to Google Firebase. Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.
stavros 2y ago
We all take too much for granted how much git "just works", even when it doesn't.
- Izkata 2y ago
  Even those before git - what they're describing sounds a bit like RCS, the precursor to CVS, which came before SVN. I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.
  - wolf550e 2y ago
    VSS6 had file checkout, which IIRC locked the file preventing other devs from editing it. It was popular with Visual C++ 6.0, circa the year 2000. https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe
- deleted 2y ago
  [deleted]
whatever1 2y ago
I bet now they deal with broken pipelines and dependency hell. Tools do not fix bad design.
mikeocool 2y ago
I recall something similar from my first job, except the shared file locking was a full on feature in Macromedia dreamweaver. CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.
- q7xvh97o2pDhNrh 2y ago
  ...and whoever did that "accidentally" when they had to leave around lunchtime on Friday was, presumably, celebrated as a local legend.
- mnahkies 2y ago
  Macromedia flash / .fla files weren't particularly ideal for collaboration either, though I still feel a bit nostalgic for working with flash in general
- evan_ 2y ago
  It just did this by creating a sentinel file- so if you needed to, you could just delete the file manually.
  - masklinn 2y ago
    SVN provided the ability to steal locks, and locks were opt-in so e.g. you wouldn't have made CSS need locks because it's a text file and merges fine. Mandatory locking was mostly for binary work files e.g. PSD and the like.
  - mikeocool 2y ago
    Good to know, though I’m optimistic this is knowledge I’ll never need again!
cjbgkagh 2y ago
So they had a problem, got 2 years of approved development effort of a small team to solve it property which they did successfully, and then you resigned? After they fixed the problem? Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1. I presume there were other reasons for resigning, or you just like massive refactoring projects.
- Nexialist 2y ago
  It was a little tongue in cheek, but yes. I had large grievances with the software culture there, but after I got sign off on the project to modernise our build process, I couldn't bring myself to abandon ship in the middle of trying to fix it. After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.
  - gnat 2y ago
    Thank you for the clarification. Because you said “it took a small team … and then I resigned”, it was unclear that you were part of that small team and instead made it sound like you left because the problem was fixed.
    - dfee 2y ago
      For what it’s worth, it wasn’t unclear when I read it.
      - thanksgiving 2y ago
        I worked at a company recently where I couldn't get a team to do the fairly simple task of cleaning up our git ignore (so we wouldn't have to painstakingly add certain files to the index every time we changed something) so I take it as a massive accomplishment moving to git within two years. If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.
werdnapk 2y ago
Visual SourceSafe would show that a file was checked out hinting to maybe stay away. Good times.
- heywire 2y ago
  We still use VSS for a couple codebases. We’re a small enough team that conflicts are rare, but the occasional Teams message “hey, let me know when you’re done with module.cpp” is not unheard of.
  - cebert 2y ago
    I’m impressed VSS still works. I used it as part of my first professional software engineering job in 2008 and it felt old then.
    - jsrcout 2y ago
      I'm so sorry. We were using it in the late 90s, and on a monthly basis it would be down for a day or two while the admin plunged out the database. Good times.
    - heywire 2y ago
      It surprisingly works well. We do use a front end to it called VssConnect, but to be honest, I really don’t have any complaints.
  - varjag 2y ago
    Fascinating. An anachronism worthy of steampunk novels!
- telgareith 2y ago
  Did you know, theres a GitHub repo called [you-poor-bastard](https://github.com/SirkleZero/you-poor-bastard https://github.com/SirkleZero/you-poor-bastard)? It converts a vss repo to git (not very well, but well enough), ignoring the VSS "passwords."
- SoftTalker 2y ago
wredue 2y ago
I’ve worked on “check out” code bases very similar to this. I mean. Nothing so insane as scripts patching class files in to jars Willy nilly, but it seems the “just check out a file so nobody else can work on it” thing is a “common” “solution” to this. I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)
brightball 2y ago
Had a similar thing on a small team at a bigger company. We didn't have any type of version control so the team did all development on a shared dev server that we SSH'd into. We'd use vim to edit files one at a time and rely on vim's file locking to make sure that we weren't editing anything at the same time as anybody else. Oddly, the project was one of the best I've ever worked on.
ufmace 2y ago
I think a lot of people who've only learned programming in the last 10 years or so don't realize that Git, and even more so the large-scale popularity of Git, is actually pretty new. A lot of programming happened before Git was created, and before it became mature and popular enough to be the default for everyone. Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that. Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it. I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it. [0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
- fiddlerwoaroof 2y ago
  I use CVS, RCS and Subversion on my personal projects for several years before I learned git. I don’t remember any of them being a pain to setup for small projects and Sourceforge provided free CVS hosting for small projects.
  - brabel 2y ago
    I started with Rational Clearcase, which I interacted with via Eclipse, and I remember that you did have to lock a file to make changes to it... pretty sure there was no merge capability (but maybe I was just too junior to know stuff like that, it was definitely not something you would do as a matter of routine, like we do now with git). Someone leaving for vacation and forgetting to unlock a file could be a huge pain in the ass. After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.
wiether 2y ago
> In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java. > In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares! But we were using Subversion's lock system and had a proper CI/CD with Jenkins. So you won, your project was worse!
matheusmoreira 2y ago
> It took a small team two years to migrate the entire codebase to git with proper CI I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.
alt187 2y ago
That's just maddening enough to start a cult! The Rube-Goldberg VCS-mutex is hilarious though.

sir-dingleberry 2y ago

This is insane. Thanks for the post.

pelagicAustral 2y ago

This sounds so much like dealing with MS Access databases... Unfortunately, part of my responsibility in my current role is to manage a handful of Access applications... they are ancient, I am talking early 2000's. They are the most unruly thing I have ever had to work with, and I will not go into details but your story reminds me so much of having to work on these apps...

DaiPlusPlus 2y ago
MSAccess is a tragedy, imo. Right-up until, say, Access 2007, it was a simple (in a good way!) RAD platform following those zombie 4GL predecessors, but it’s been left to stagnate without any real effort to modernise it, and it’s clear from how Microsoft’s been downplaying Access that it’s a product they’d really rather not have to support, but they know if they do kill it then it would weaken on-prem Office’s moat and everyone will probably just rush to Google Firebase. Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.

stavros 2y ago

We all take too much for granted how much git "just works", even when it doesn't.

Izkata 2y ago
Even those before git - what they're describing sounds a bit like RCS, the precursor to CVS, which came before SVN. I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.
- wolf550e 2y ago
  VSS6 had file checkout, which IIRC locked the file preventing other devs from editing it. It was popular with Visual C++ 6.0, circa the year 2000. https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe
deleted 2y ago
[deleted]

whatever1 2y ago

I bet now they deal with broken pipelines and dependency hell. Tools do not fix bad design.

mikeocool 2y ago

I recall something similar from my first job, except the shared file locking was a full on feature in Macromedia dreamweaver. CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.

q7xvh97o2pDhNrh 2y ago
...and whoever did that "accidentally" when they had to leave around lunchtime on Friday was, presumably, celebrated as a local legend.
mnahkies 2y ago
Macromedia flash / .fla files weren't particularly ideal for collaboration either, though I still feel a bit nostalgic for working with flash in general
evan_ 2y ago
It just did this by creating a sentinel file- so if you needed to, you could just delete the file manually.
- masklinn 2y ago
  SVN provided the ability to steal locks, and locks were opt-in so e.g. you wouldn't have made CSS need locks because it's a text file and merges fine. Mandatory locking was mostly for binary work files e.g. PSD and the like.
- mikeocool 2y ago
  Good to know, though I’m optimistic this is knowledge I’ll never need again!

cjbgkagh 2y ago

So they had a problem, got 2 years of approved development effort of a small team to solve it property which they did successfully, and then you resigned? After they fixed the problem? Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1. I presume there were other reasons for resigning, or you just like massive refactoring projects.

Nexialist 2y ago
It was a little tongue in cheek, but yes. I had large grievances with the software culture there, but after I got sign off on the project to modernise our build process, I couldn't bring myself to abandon ship in the middle of trying to fix it. After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.
- gnat 2y ago
  Thank you for the clarification. Because you said “it took a small team … and then I resigned”, it was unclear that you were part of that small team and instead made it sound like you left because the problem was fixed.
  - dfee 2y ago
    For what it’s worth, it wasn’t unclear when I read it.
    - thanksgiving 2y ago
      I worked at a company recently where I couldn't get a team to do the fairly simple task of cleaning up our git ignore (so we wouldn't have to painstakingly add certain files to the index every time we changed something) so I take it as a massive accomplishment moving to git within two years. If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.

werdnapk 2y ago

Visual SourceSafe would show that a file was checked out hinting to maybe stay away. Good times.

heywire 2y ago
We still use VSS for a couple codebases. We’re a small enough team that conflicts are rare, but the occasional Teams message “hey, let me know when you’re done with module.cpp” is not unheard of.
- cebert 2y ago
  I’m impressed VSS still works. I used it as part of my first professional software engineering job in 2008 and it felt old then.
  - jsrcout 2y ago
    I'm so sorry. We were using it in the late 90s, and on a monthly basis it would be down for a day or two while the admin plunged out the database. Good times.
  - heywire 2y ago
    It surprisingly works well. We do use a front end to it called VssConnect, but to be honest, I really don’t have any complaints.
- varjag 2y ago
  Fascinating. An anachronism worthy of steampunk novels!
telgareith 2y ago
Did you know, theres a GitHub repo called [you-poor-bastard](https://github.com/SirkleZero/you-poor-bastard https://github.com/SirkleZero/you-poor-bastard)? It converts a vss repo to git (not very well, but well enough), ignoring the VSS "passwords."
SoftTalker 2y ago

wredue 2y ago

I’ve worked on “check out” code bases very similar to this. I mean. Nothing so insane as scripts patching class files in to jars Willy nilly, but it seems the “just check out a file so nobody else can work on it” thing is a “common” “solution” to this. I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)

brightball 2y ago

Had a similar thing on a small team at a bigger company. We didn't have any type of version control so the team did all development on a shared dev server that we SSH'd into. We'd use vim to edit files one at a time and rely on vim's file locking to make sure that we weren't editing anything at the same time as anybody else. Oddly, the project was one of the best I've ever worked on.

ufmace 2y ago

I think a lot of people who've only learned programming in the last 10 years or so don't realize that Git, and even more so the large-scale popularity of Git, is actually pretty new. A lot of programming happened before Git was created, and before it became mature and popular enough to be the default for everyone. Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that. Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it. I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it. [0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

fiddlerwoaroof 2y ago
I use CVS, RCS and Subversion on my personal projects for several years before I learned git. I don’t remember any of them being a pain to setup for small projects and Sourceforge provided free CVS hosting for small projects.
- brabel 2y ago
  I started with Rational Clearcase, which I interacted with via Eclipse, and I remember that you did have to lock a file to make changes to it... pretty sure there was no merge capability (but maybe I was just too junior to know stuff like that, it was definitely not something you would do as a matter of routine, like we do now with git). Someone leaving for vacation and forgetting to unlock a file could be a huge pain in the ass. After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.

wiether 2y ago

> In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java. > In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares! But we were using Subversion's lock system and had a proper CI/CD with Jenkins. So you won, your project was worse!

matheusmoreira 2y ago

> It took a small team two years to migrate the entire codebase to git with proper CI I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.

alt187 2y ago

That's just maddening enough to start a cult! The Rube-Goldberg VCS-mutex is hilarious though.