27 ms·

When I started at my first company, they had a very complex VB application running on dozens of customers around the country, each having some particular needs

When I started at my first company, they had a very complex VB application running on dozens of customers around the country, each having some particular needs of course. There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio. I don't know how they convinced the users to do this and how they managed with the license but it was done. What happened next was even worse. There was no version control of course, the code being available on a shared disk on the local network of the company with the code copied over in multiple folders each having its own version, with no particular logic to it either, V1, V2, V2.3, V2a, V2_customer_name, V2_customer_name_fix, ... After that, when there was a problem for a customer, the programmer went there to debug and modified the code on site. If the bug/problem was impacting other customers, we had to dispatch some guys for each customer to go copy/edit the code for all of them. But if the problem was minor, it was just modified there, and probably saved on the shared folder in some new folder. What happened next was to be expected: there was no consensus on what was the final version, each customer having slightly different versions, with some still having bugs fixed years before for others.

fouronnes3 2y ago
This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.
- mleo 2y ago
  Anecdote seems long before git creation, so Visual SourceSafe maybe. Which did not work well over a WAN. Needed other tools to replicate and synchronize VSS.
  - masklinn 2y ago
    You can remove the "over a WAN" part: VSS had been designed as a local VCS, so until the addition of a proper server in the mid aught using it over a network share was the only way to actually use it. And it really wasn't good. I don't know if that made it better, I assume not much, VSS was really trash.
  - breput 2y ago
    I was working at a game development company in the mid 90s that used Visual Source Safe and to avoid file corruption due to concurrent commits, we had a shiny silver baseball cap which you had to find and physically possess to commit. After we had off-site devs, the physical cap wouldn't work. So the project had a "silvercap.txt" file and you had to exclusively check out . And of course people forgot to release that and work ground to a halt.
- kolanos 2y ago
  I'm sure I'm not alone in actually having lived such an experience. I joined a dynamic DNS provider once that had been around since 1999. Their tech, sadly, had not progressed much beyond that point. Showing the higher ups version control was like showing cavemen fire. Of course once the higher ups arranged to have training sessions led by the new hire for the entire dev team the VP of Engineering couldn't handle it and had me fired. Fun times.
- The_Colonel 2y ago
  At this level of dysfunction, installing git won't do anything. You need a holistic change in thinking which starts with convincing people there's a problem.
  - parpfish 2y ago
    Yeah, this level of dysfunction takes years to cultivate. You need the “Dead Sea effect” to be in effect long enough that not only have the good people left, but for them to have been gone long enough that people rising into management have never even worked with somebody competent so they don’t know there’s a better way
    - HdS84 2y ago
      I am occasionally on firefighting projects for customers. It's super hard to get out of this mess. The code base has gone to shit, customers are often passed enough that they want functioning code NOW, the reaming developers are bottom of the barrel, management is either cause of the problems or has now clue. Getting out of that is difficult especially if management doesn't understand it has to change first.
    - mlhpdx 2y ago
      Years, and a special kind of myopia in leadership to not see the seeds sprouting. It doesn’t happen quickly, but once established is very, very expensive to fix.
    - lutoma 2y ago
      Oh man that description immediately made me think of a government IT job a friend used to work in
  - lloydatkinson 2y ago
    I’ve yet to see people actually change their mind at this level. Firing is often the only way.
- djbusby 2y ago
  I was one of those once. Tried to get CVS in a project. Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow. I left.
  - qup 2y ago
    Then they probably fired the guy who deleted 2MM lines of tabs for not meeting LoC metrics.
    - djbusby 2y ago
      Many years later there was a scene in Silicon Valley where their VC set a tequila bottle on the delete key and caused havoc. That's when I figured out what happened in 1998.
      - dtech 2y ago
        And what's the less opaque reference to 1998?
        djbusby 2y ago
        Me, up thread, I was the junior in 1998, trying to introduce CVS.
      - collinmanderson 2y ago
        I have had a website taken down by (pretty sure) a person setting something on their F5 key.
        qup 2y ago
        Let's hear it!
        collinmanderson 2y ago
- hleszek 2y ago
  I started in 2008. This is what I did eventually. Over the years I introduced the small company to Linux, git, defensive programming, linting, continuous integration, Scrum..., but only for the new projects and stayed 13 years there. That old project though was never fixed though, probably still running that way now.
  - refactor_master 2y ago
    Same story here. Introduced static Python to our team’s entire codebase. Best place I’ve ever worked.
  - FpUser 2y ago
    I think of Scrum as a crime against humanity
- llmblockchain 2y ago
  I've been that person a few times. 1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS. 2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master. The list goes on!
  - malux85 2y ago
    Here’s a few of my horror stories where I was a consultant at various companies: 1. Previous consultants left no documentation or anything, and a running Hadoop cluster handling (live!) 300 credit card transactions a second. Management hired 8 junior sysadmins - who were all windows sysadmins, had never used Linux before, and were expected to take over running this Linux cluster immediately. They all looked at me white as ghosts when I brought up SSH prompt, that’s the point where I learned they were all windows sysadmins. 2. Another company: all Java and MySQL developers who were trying to use Spark on Hadoop, refusing to learn anything new they ended up coding a Java app that sat on a single node, with a mysql database on the same node, that “shelled out” to a single trivial hello-world type function running in spark, then did the rest of the computation in Java on the single node, management celebrated a huge success of their team now using “modern cluster computing” even though the 20 node cluster did basically nothing and was 99.99% idle. (And burning huge $ a month) 3. Another company: setup a cluster then was so desperate to use the cluster for everything installed monitoring on the cluster, so when the cluster went down, monitoring and all observability went down too 4. A Cassandra cluster run by junior sys-admins and queried by junior data scientists had this funny arms race where the data scientists did what was effectively “select * from *” for every query and the sysadmins noticing the cluster was slow, kept adding more nodes, rather than talk to each other things just oscillated back and forwards with costs spiralling out of control as more and more nodes were deployed Any many more! This might sound like I’m ragging on juniors a bit but that’s definitely not the case - most of these problems were caused by bad management being cheap and throwing these poor kids into the deep end with no guidance. I did my best to upskill them rapidly and I’m still friends with many of them today, even though it’s nearly 10 years later now, Good times!
    - llmblockchain 2y ago
- sergiotapia 2y ago
  I did this at my first, learned quick oldheads would get flustered and feel challenged if not eased into things a certain way. Ultimately by the time I left I tried to introduce redbeanphp (orm), git for source control, and CakePHP for some structure. Nothing stuck. When I left it was still raw sql string queries, .zip files when they remembered for backups, and 400,000 line php files with everything caked on there.
  - kolanos 2y ago
    Yes, there is a good lesson here. If you walk onto a dev team still using stone tools, there is likely a culture that distrusts of new technology, so tread lightly or you may be perceived as a heretic.
- fragmede 2y ago
  I'm assuming this story predates git and the Internet, but I'm not OP. Things were different back in those days.
- donall 2y ago
  I had a visceral reaction to this comment! I once joined a company doing ETL with Apache camel and a half dozen underpowered pet machines. Ingesting their entire dataset and running a suite of NLP models took 3-6 months (estimated; it was so slow nobody ever reprocessed the data to fix bugs or release improvements). I drew up a simple architecture using Kafka, hbase, and MapReduce to implement a lambda architecture. The CTO very patronizingly told me that just because something is shiny and new it doesn't mean we need to implement it. This was 2017 :laugh-cry:.
  - bjconlan 2y ago
    It's a shame the CTO was patronizing. I've generally found this to be the attitude of many IT workers in similar positions. I would recommend trying to allocate (work) time to prototype and get important back of the envelope metrics that they think are valuable along with those that you think are valuable. At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)
  - yyyfb 2y ago
    But maybe this isn't really what they felt that they needed at the time? I don't mean to defend bad practices, but your comment makes it sound like nobody had tasked you with re-architecting the business, and you took it upon yourself to show them how it should be done (in your opinion), without having earned the necessary trust. This might have also come across as patronizing, or at least antagonistic, and in any case unbeneficial. Not saying that's the case as I obviously wasn't there, just something to think about.
    - donall 2y ago
      Fair comment. And I'm usually suspicious of young engineers wanting to implement the new hotness and I'm also a fan of "if it ain't broken don't fix it". In this case, though, the system was in very rough shape. Our customers were complaining about data problems which we had no way to fix (short of manually editing the prod db, which was the SOP). I definitely took it upon myself to do something that nobody had asked for, but it was because the people in charge were entirely asleep at the wheel! They did not last long in their positions.
- blacklight 2y ago
  Been there. There was this old fashioned developer in one of the companies I worked for a decade ago who never understood nor embraced version control (we were talking of SVN at the time, not even git). Luckily that wasn't the case for all the others developers in the company. But when it came to the projects he owned, I witnessed several scenes along the lines of "hey, customer X has an issue with your component Y, what version do they have?" He had a spreadsheet where he kept track of the versions used by every customer. Once identified the version, he would open (no joke) a drawer in his desk and pick the right USB stick with that version on it. I've always wondered whether this overhead was a worth price to pay for not wanting to learn a couple of SVN commands.
  - icedchai 2y ago
    My first job out of college, way back in the 90's, I had to "train" an entire department on how to use CVS. This wasn't a startup. This was a billion dollar, established company with 1000's of employees.
  - Aloha 2y ago
    He was in essence doing manual version and release control - just.. like you would in 1990 with floppy disks.
  - cyanydeez 2y ago
    Perhaps he was more concerned with job control, and saw centralized code as a detriment to his job security
    - throwaway03254 2y ago
      I have a confession to make. I was working in a company where the main development center was located in a different country. The main development center wanted to centralize all code on a single SVN server and move the build process to Jenkins I said we already had source control (CVS) and a a working build process (a script on a server that pulled from CVS, built a jar file and deployed it to X servers). We were too "busy" at the moment but would look into it in the future. This never happened. The real reason was that I was concerned that the central development center would take over the project, as they had tried to do so in the past. Looking back, I should probably have let them take over as there was more than enough work for everyone.
- kwhitefoot 2y ago
  Some of these stories sound a bit far fetched, especially those that involve Unix systems. RCS was released in 1982 and CVS in 1990 so Unix systems have had version control available for over forty years.
  - bdbdhxhdh 2y ago
    There is a lag between availability and mindshare to apply a technology. I myself introduced SVN as versioning solution to a company in 2007, years after git was available I and I was aware of it git simply didn't have the mind share jet, thus I went with what I knew
    - Iwan-Zotow 2y ago
      Hear, hear Git was too weird for a team, but svn with its linear timeline was a hit. They are still with svn, and for a small shop it's ok
    - zimpenfish 2y ago
      > I myself introduced SVN as versioning solution to a company in 2007 In 2013, I was tasked with writing wrapper scripts around SVN to make it look like SCCS to avoid confusing the development people who only knew how to use SCCS whilst the company migrated to SVN. Fun but traumatising.
  - cruffle_duffle 2y ago
    I can assure you they are true. Version control was still “controversial” in a lot of shops for quite some time. Plenty of places had the classic “v1, v2.3, v2_next_debug_cruffle_duffle” way of managing versions for a very, very long time. Well into the subversion era…
    - CRConrad 2y ago
      I know places that still do it that way. Not necessarily tiny places either.
  - teractiveodular 2y ago
- ics 2y ago
  I was in this position before but would point out that there is a tactical approach when you know that others will not follow. I set up a cron job (on Windows, not really cron) to check scan the network location for updated source files. The git repo was on my drive and on the corporate GitHub account, safe from those who should have been using it. Whenever files changed it would just auto commit on the main branch with the username included in the message. I could do whatever I wanted on my own branches, keep track of what others were doing, and essentially wield git. You don’t have to try to be a hero inflicting proper source control upon your teams (their perspective) to still occasionally appear like a wizard to save them from inevitable, oft-occurring peril.
- gumby 2y ago
  Astonishingly, It took a long time for revision control to become widespread. Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forking was generally considered a tragedy, something I managed to change five or six years later). Rich Pixley came and said “all of our code should be in revision control, and I want to use this newfangled thing called CVS.” Michael was OK with it but John and I were steadfastly opposed. We agreed to an experiment, grudgingly, subject to a whole whiteboard of absurd conditions (“must be transparently integrated into emacs so we don’t have to know it’s there). Pixley agreed to all of that and then ignored all of it completely. It was immediately such a win that everybody adopted it without complaint, including us two obstreperous a-holes. A few years later a preventable crisis was how revision control first became client-server.
  - asveikau 2y ago
    Linux was managed with tarballs and patch emails until a little over 20 years ago when they started using bitkeeper, then Linus wrote git.
    - NetOpWibby 2y ago
      What a legend
      - fijiaarone 2y ago
        In the story above, Linux Torvalds is Gitfolye.
  - jimbokun 2y ago
    > Pixley agreed to all of that and then ignored all of it completely. Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.
- ragebol 2y ago
  Have been that person before. As an intern, and they even listened! In the days of SVN, just before git, so I ran a server in my laptop and my manager somehow decided we needed a big Red hat server or something, IIRC. In a 20 ppl company.
- MBCook 2y ago
  I never had to deal with “we don’t use source control”, luckily. One company I joined was cleaning up the last vestiges of “customize it for every customer by letting people edit their copy on server,” which predictably turned into a mess. They were all very small customizations to styles or images but enough to make upgrades a total mess. I did work at a company where despite having full source control they didn’t actually know of they could ever deploy the server component again. Edits got made to the live server but then made again in source control, or vice versa. There was one more senior person who couldn’t be talked out of their old workflow. In theory everything matched.Eventually they even checked and got it all under control where they were positive it was the same and kept it that way. But it had only ever been deployed from scratch… once. And for like 15 years it lived there and kept getting upgraded. It would all be moved when new hardware was brought in. But it wasn’t installed from scratch. We truly did not know if we were capable of doing that. It is possible if that server was destroyed and we couldn’t restore from a back up it would take us an unknown amount of time. Even though in theory deploying should be as simple copying the files and starting the web server. Were there odd configurations that had been made eight years ago that kept it running? Some strange permission changed somewhere? I wasn’t on that team. But it always made me nervous. That was absolutely a core application of the business. I really like small shops sometimes. You get a lot of freedom and get to have your hands in a lot of disciplines. You learn a lot of things, including things that’s should never be duplicated.
  - yyyfb 2y ago
    That's not an exception, I think? Most online service businesses that I've worked with wouldn't be able to run their operations from a "cold start". That takes a lot of effort to engineer and in practice it doesn't happen enough, so that's a risk that most are willing to run with.
    - MBCook 2y ago
      Really? Everywhere else I’ve worked has been able to. I’m not saying all the data in the database. Pretend your DB is fine or you have a full up to date backup. I know losing that KILLS businesses. But in my example just the web server immolated and the backup tapes sitting next to it melted. Can you set up a new web server? As a Java developer it’s usually install OS, add Apache + Tomcat, set a few basic config parameters, copy WARs. PHP or Python is even easier. They didn’t know they could successfully do that.
- camilla4151 2y ago
  [dead]
- deleted 2y ago
  [deleted]
- freetanga 2y ago
  Software Configuration Management has existed as a discipline and with proper tooling for at least 50 years. Mainframe and VAX machines had tooling in the early 80s. For VB Sourcesafe was the go to tool if memory serves. This is not a case of new vs old, rather incompetence vs competence.
- yyyfb 2y ago
  It's maybe better than to take the pain to set up git only to see people use it in the same way, setting up a gazillion branches called V1, v2_customer_fix, v3_final, etc...
- fencepost 2y ago
  I think you misspelled "Visual SourceSafe" - and depending on when this VB was written it might actually predate VSS as a Microsoft product. Source code management back in the days of VB (long before VB.Net) was not the same as what you see today.
- BobbyTables2 2y ago
  Wait until you work in an industry where customers are extremely reluctant to upgrade and each one lands on a different release. Then you have to start tracking the limitations and issues in 2-3 year old releases and the solution to issues is never “upgrade to the latest”. At that point, source control starts feeling worthless.
- samspot 2y ago
  Setting up git is the easy part. We all used it. Except the owner of the company who would fix bugs in prod and not tell anyone. Then next release we'd unintentionally un-fix those bugs because the fixes never made it back to source control.
deleted 2y ago
[deleted]
culi 2y ago
Can I ask roughly what year this was? I can't even imagine something like that today
- hleszek 2y ago
  2008
- nine_k 2y ago
  Visual Basic was a smash hit released in 1991. For reference, Perforce was released in 1995, Subversion in 2000, git in 2005. RCS existed, but only in the Unix world.
  - icedchai 2y ago
    Don't forget CVS, released in 1990. It was popular for about a decade before Subversion took over.
    - nine_k 2y ago
      Did it run under DOS? I'm not even asking about a win16 native GUI client. (You can still enjoy CVS if you want to contribute to OpenBSD, for instance.)
      - icedchai 2y ago
        I'm not sure. In the mid-90's, I worked at a Solaris and NT shop, which was where I first used it.
      - mike_hearn 2y ago
        Yes, and for Win95 users there was actually a very nice GUI plugin to explorer called TortoiseCVS. The funny thing about CVS and Subversion is that they were last version control systems that non-programmers could actually use. TortoiseCVS/TortoiseSVN were easy enough that with a lot of training you could get technical writers or artists to use it too. Valuable for game projects. With git, forget about it. It can't even handle binary files well.
thayne 2y ago
> There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. I once ran across a (c) program that had 26 variables, each one letter long, one for each letter of the alphabet. They were all global variables. And many of them were re-used for completely unrelated things.
- bdbdhxhdh 2y ago
  wow this is just evil I want an obfuscator doing such a transformation:D
- leni536 2y ago
  Ah, the "user-defined registers" paradigm.
- tengwar2 2y ago
  I inherited a control program from Risø National Laboratories. It had roughly 600 globals of the form A9$, three local variables, and one comment - "Midlertidig". However on a more practical note, the "Java" used on Smartcards effectively requires that all variables be treated as constants, other than one array. You dynamically allocate the array when handing an event, and it only lasts for the duration of that event.
sghiassy 2y ago
This is nightmare-fuel. How does this happen? As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices?? Honest question. How does our profession root out intrinsically obvious bad practices?
- lazide 2y ago
  It happens because it’s easier - until it’s impossible, anyway. The training and best practices you’re talking about is learned experience about how to avoid it getting impossible. But that almost always involves expense that the business side considers ‘stupid’.
insane_dreamer 2y ago
How long ago was this?
lowdownbutter 2y ago
My first role was with a company who had hit a limit for VB6 variable names iirc. So they'd all been renamed to shorter names. This may be the same issue. They were in the process of rewriting in VB.net.
- DidYaWipe 2y ago
  That sounds like they were doing something dumb. VB6 did not have a limit for variable names, unless it was 255 characters or something. And yep, I just checked: The limit was 255 characters.
  - lowdownbutter 2y ago
    I think more like globally. Like there was a fixed amount of space for all variable names.
matheusmoreira 2y ago
> with some still having bugs fixed years before for others Hopefully they won't discover at some point that customers were in fact depending on those bugs. That's poison for the programmer's soul.
- disqard 2y ago
  Yup! Hyrum's Law :)
osigurdson 2y ago
There is "do things that don't scale" and the there is bending over backward to do things in the dumbest possible way. Not sure where this lands.
makmanalp 2y ago
As much as this stuff is nuts to think of today and there's tons to hate, I am kinda nostalgic for some aspects of my experience of working at a place where software is maybe needed and/or valued but isn't a core competency. Or maybe a time when software was a new fangled thing that hadn't fully been integrated into corporate structure yet: - No one having any preconception of how you're /supposed to/ do things or whether you'd even be the type of person to know, so you just kinda figure it out yourself. You spend a lot of time on reading and learning skills. Version control? Wow, cool, what's git, let's try that! A new graphing library? Let's give that a shot, maybe it'll make things easier! You want XYZ? Let me go read about that for a day. - No one having any idea what's even possible: being treated like a wizard for introducing the tiniest piece of automation or improvement that makes someone's day easier or doing something they never thought was possible. Lots of appreciation and excitement for showing and teaching people new things (... and I know this is somewhat selfish and ego driven, but who doesn't like being appreciated?) - Similarly people having no idea how long those things should take which, tbh, can be a nightmare if you're not trusted and respected enough to be consulted but also great if people believe you when you say it's gonna take 3 months. - Beyond the basics just being mostly kinda left alone to do your job however: no standups or tickets or the 30 other kinds of daily (micro)management that is probably necessary but ends up feeling tiresome and stifling at an individual level - Not being part of software company "culture": no performance review driven development and promo packet madness, no weird rating and ranking systems, no OGPs or KPIs. No ladder. Your bosses think you did what was required of you, so then you're good, and if it's a good year you get a raise, and that's that. I do recognize that with a bad boss this can be a terrible and unfair spot to be in - but again, subjectively with a decent enough boss it felt like a lot less weight on my shoulders at the time. - No hacker ninja pirate segway mini quadcopter you're the smartest people in the world and we're the best company to work for sort of b.s. - Socializing with people who are good at and love to talk about stuff other than software Reading over that, I'm thinking maybe I lucked out a lot and that wasn't most people's experience from that era. And there's some level of rose tinted glasses going on. And/or maybe my years in the rat race are starting to show :-)
- Aeolun 2y ago
  Don’t think so. My first job was kind of like that. I don’t even know how they thought that little old me just out of university could be left alone to successfully build applications on my own, but I think people trusted a lot more during that era because eternal september hadn’t arrived yet. Working directly for the users without any weird BA/PM/TA shit in between is glorious, both because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue), and in a place like that you can likely deploy it in the middle of the day and immediately improve their workflow. It still amuses me that IT was located together with finance, because we did reports xD
TacticalCoder 2y ago
> At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio. Holy smoke! That's actually the most creative solution (horrible, but creative) I've ever heard to fix a Heisenbug: https://en.wikipedia.org/wiki/Heisenbug https://en.wikipedia.org/wiki/Heisenbug
- pletnes 2y ago
  Often it’s enough to compile in debug mode and maybe set an env var! This sounds like a horrible approach by all measures I think.
tansan 2y ago
These type of stories are a humbling reminder that PMF always beats engineering excellence
- zarathustreal 2y ago
  Well that depends entirely on what you consider to be the goal - as a software engineer, your role is entirely concerned with engineering excellence. As a member of a team, especially a team of extremely highly paid and highly educated individuals, it is your duty to spend your time (and thus, the company’s resources) efficiently by doing what you’re educated, qualified, and hired to do.
  - jahewson 2y ago
    Engineering excellence is a means to an end. Great engineers solve problems - sometimes by writing software.
  - williamdclt 2y ago
    Few people agree that the goal of SWE is engineering excellence. It is to solve business problems. Engineering excellence is a means to a goal: to be able to solve _difficult_ problems _correctly_ and allow solving new problems _in the future_. All these things can be traded off, and sometimes aren’t even needed at all
onion2k 2y ago
I don't know how they convinced the users to do this and how they managed with the license but it was done Enterprise and business users will wade through endless swamps of crap if the software provides enough value. This is a lesson in why "it must be perfect before we release it" is such nonsense - that just says the value your app provides is so low that users barely care and they'll abandon it at the first problem. If that's the case you're not providing anything worth paying for.
croes 2y ago
That's why cloud solutions exist. Now you only need to run the app in debug mode on your own server.
feketegy 2y ago
This is that meme when the pizza guy walks into a burning room. Also, this comment is worthy of The Daily WTF.
Forgeties79 2y ago
This sounds like what I see when an inexperienced filmmaker takes on a big project and hands me the “organized” drive to edit, but way way worse and more consequential lol
llm_trw 2y ago
I was in a broker trader in 2016 where this was still the case. I was brought in when an old sheet cost them $10m on a bad trade because the yahoo finance end point it was hitting stopped responding and it just used the last value it had gotten - three months before.
DragonMaus 2y ago
This is entirely typical of especially VB scripts. When I was a software engineer for a Fortune-20 company, I spent more time debugging (and trying to normalize, though that met with mixed levels of resistance) VB applets than anything else.
fencepost 2y ago
Oh dear god no. The solution is not throw VS at it and run from the code, the next step is some combination of excessive logging (which to be fair may resolve the issue all by itself) and/or throwing in a ton of DoEvents because Visual Basic.
acwan93 2y ago
Dear god this is pretty much what I went through when I started taking over a company with a 35-40 year old codebase. Files spread everywhere, no consensus, and supporting customizations for thousands of customers who we didn’t know if they were even still using the system. It took five years and the firing of the long-time “head” programmer until some meaningful change was made.
titusjohnson 2y ago
Oh man, that brings me back. My first tech job was an ecommerce company that a basic online cart with backed by incredibly in-depth set of industry catalogs. We also sold a marketing package as an add-on to the online store where we would proactivly contact our customers and get the info from them to replicate their monthly/weekly physical advertising on the web. This was back in '05-ish, lots of money to be made just helping people get their business online. We had a group of talented designers/artists wrangling Photoshop and pumping out a few of these designs a day, each, and as we scaled up and gained a lot of repeat customers, tracking these PSDs became a big problem. The designers were graphically talented, not technically savvy. The PSDs were stored on a shared NAS drive that the whole company could see. The designers had a complex naming system to manage major revisions, but overall there was no "history" beyond the classic "_v2_2008_09_best.psd" naming technique. Several times every week I had to fix "accidentally dragged the PSD folder somewhere and lost it" level problems. Getting IM'd from tech support because the underlying server was falling over trying to clone the multi-GB folder 3 times, logging into a workstation as Admin and searching for an updated PSD that the vacationing Designer hadn't synced back to the NAS before leaving for work, that kind of thing. As soon as I was promoted to Supervisor I made my first big move. It took a lot of training, far more talking than I thought it should (back then I didn't know anything about politics), but I was able to get SVN implemented to replace the NAS share. I wrote quick-reference documents, in-depth guides, (this was before I knew that no one reads anything, ever, for any reason) and eventually just had to do one-on-one training with everyone just to explain the concept and basic useage. One of the most satisfying feelings of my career continues to be watching attitudes change over the course of a summer. None of the design-y people liked the new set of hoops they had to jump through. Check-Out, Check-In, Lock, etc, it was "too much". Then, at a happy hour someone mentioned how we hadn't lost the PSD folder in a while. Later someone came to me panicking because a client wanted to re-run an ad from 2 months ago with a couple tweaks, and she didn't have the source PSD or the source material -- I did a live demo on how to get a historical version back, and that's when it really clicked with everyone. With internal political will behind the toolset, it now became an IT problem, as our SVN useage was nothing like Engineering's usage. Of course file locking was a huge PITA, that feature replaced "forgot to copy the changed file back before vacation" as a problem category. But it also eliminated the problem where 2 people would open the same PSD directly from the NAS share, make their changes, and only the last one to save gets their work persisted. So, a toss-up I guess.

fouronnes3 2y ago

This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.

mleo 2y ago
Anecdote seems long before git creation, so Visual SourceSafe maybe. Which did not work well over a WAN. Needed other tools to replicate and synchronize VSS.
- masklinn 2y ago
  You can remove the "over a WAN" part: VSS had been designed as a local VCS, so until the addition of a proper server in the mid aught using it over a network share was the only way to actually use it. And it really wasn't good. I don't know if that made it better, I assume not much, VSS was really trash.
- breput 2y ago
  I was working at a game development company in the mid 90s that used Visual Source Safe and to avoid file corruption due to concurrent commits, we had a shiny silver baseball cap which you had to find and physically possess to commit. After we had off-site devs, the physical cap wouldn't work. So the project had a "silvercap.txt" file and you had to exclusively check out . And of course people forgot to release that and work ground to a halt.
kolanos 2y ago
I'm sure I'm not alone in actually having lived such an experience. I joined a dynamic DNS provider once that had been around since 1999. Their tech, sadly, had not progressed much beyond that point. Showing the higher ups version control was like showing cavemen fire. Of course once the higher ups arranged to have training sessions led by the new hire for the entire dev team the VP of Engineering couldn't handle it and had me fired. Fun times.
The_Colonel 2y ago
At this level of dysfunction, installing git won't do anything. You need a holistic change in thinking which starts with convincing people there's a problem.
- parpfish 2y ago
  Yeah, this level of dysfunction takes years to cultivate. You need the “Dead Sea effect” to be in effect long enough that not only have the good people left, but for them to have been gone long enough that people rising into management have never even worked with somebody competent so they don’t know there’s a better way
  - HdS84 2y ago
    I am occasionally on firefighting projects for customers. It's super hard to get out of this mess. The code base has gone to shit, customers are often passed enough that they want functioning code NOW, the reaming developers are bottom of the barrel, management is either cause of the problems or has now clue. Getting out of that is difficult especially if management doesn't understand it has to change first.
  - mlhpdx 2y ago
    Years, and a special kind of myopia in leadership to not see the seeds sprouting. It doesn’t happen quickly, but once established is very, very expensive to fix.
  - lutoma 2y ago
    Oh man that description immediately made me think of a government IT job a friend used to work in
- lloydatkinson 2y ago
  I’ve yet to see people actually change their mind at this level. Firing is often the only way.
djbusby 2y ago
I was one of those once. Tried to get CVS in a project. Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow. I left.
- qup 2y ago
  Then they probably fired the guy who deleted 2MM lines of tabs for not meeting LoC metrics.
  - djbusby 2y ago
    Many years later there was a scene in Silicon Valley where their VC set a tequila bottle on the delete key and caused havoc. That's when I figured out what happened in 1998.
    - dtech 2y ago
      And what's the less opaque reference to 1998?
      - djbusby 2y ago
        Me, up thread, I was the junior in 1998, trying to introduce CVS.
    - collinmanderson 2y ago
      I have had a website taken down by (pretty sure) a person setting something on their F5 key.
      - qup 2y ago
        Let's hear it!
        collinmanderson 2y ago
hleszek 2y ago
I started in 2008. This is what I did eventually. Over the years I introduced the small company to Linux, git, defensive programming, linting, continuous integration, Scrum..., but only for the new projects and stayed 13 years there. That old project though was never fixed though, probably still running that way now.
- refactor_master 2y ago
  Same story here. Introduced static Python to our team’s entire codebase. Best place I’ve ever worked.
- FpUser 2y ago
  I think of Scrum as a crime against humanity
llmblockchain 2y ago
I've been that person a few times. 1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS. 2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master. The list goes on!
- malux85 2y ago
  Here’s a few of my horror stories where I was a consultant at various companies: 1. Previous consultants left no documentation or anything, and a running Hadoop cluster handling (live!) 300 credit card transactions a second. Management hired 8 junior sysadmins - who were all windows sysadmins, had never used Linux before, and were expected to take over running this Linux cluster immediately. They all looked at me white as ghosts when I brought up SSH prompt, that’s the point where I learned they were all windows sysadmins. 2. Another company: all Java and MySQL developers who were trying to use Spark on Hadoop, refusing to learn anything new they ended up coding a Java app that sat on a single node, with a mysql database on the same node, that “shelled out” to a single trivial hello-world type function running in spark, then did the rest of the computation in Java on the single node, management celebrated a huge success of their team now using “modern cluster computing” even though the 20 node cluster did basically nothing and was 99.99% idle. (And burning huge $ a month) 3. Another company: setup a cluster then was so desperate to use the cluster for everything installed monitoring on the cluster, so when the cluster went down, monitoring and all observability went down too 4. A Cassandra cluster run by junior sys-admins and queried by junior data scientists had this funny arms race where the data scientists did what was effectively “select * from *” for every query and the sysadmins noticing the cluster was slow, kept adding more nodes, rather than talk to each other things just oscillated back and forwards with costs spiralling out of control as more and more nodes were deployed Any many more! This might sound like I’m ragging on juniors a bit but that’s definitely not the case - most of these problems were caused by bad management being cheap and throwing these poor kids into the deep end with no guidance. I did my best to upskill them rapidly and I’m still friends with many of them today, even though it’s nearly 10 years later now, Good times!
  - llmblockchain 2y ago
sergiotapia 2y ago
I did this at my first, learned quick oldheads would get flustered and feel challenged if not eased into things a certain way. Ultimately by the time I left I tried to introduce redbeanphp (orm), git for source control, and CakePHP for some structure. Nothing stuck. When I left it was still raw sql string queries, .zip files when they remembered for backups, and 400,000 line php files with everything caked on there.
- kolanos 2y ago
  Yes, there is a good lesson here. If you walk onto a dev team still using stone tools, there is likely a culture that distrusts of new technology, so tread lightly or you may be perceived as a heretic.
fragmede 2y ago
I'm assuming this story predates git and the Internet, but I'm not OP. Things were different back in those days.
donall 2y ago
I had a visceral reaction to this comment! I once joined a company doing ETL with Apache camel and a half dozen underpowered pet machines. Ingesting their entire dataset and running a suite of NLP models took 3-6 months (estimated; it was so slow nobody ever reprocessed the data to fix bugs or release improvements). I drew up a simple architecture using Kafka, hbase, and MapReduce to implement a lambda architecture. The CTO very patronizingly told me that just because something is shiny and new it doesn't mean we need to implement it. This was 2017 :laugh-cry:.
- bjconlan 2y ago
  It's a shame the CTO was patronizing. I've generally found this to be the attitude of many IT workers in similar positions. I would recommend trying to allocate (work) time to prototype and get important back of the envelope metrics that they think are valuable along with those that you think are valuable. At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)
- yyyfb 2y ago
  But maybe this isn't really what they felt that they needed at the time? I don't mean to defend bad practices, but your comment makes it sound like nobody had tasked you with re-architecting the business, and you took it upon yourself to show them how it should be done (in your opinion), without having earned the necessary trust. This might have also come across as patronizing, or at least antagonistic, and in any case unbeneficial. Not saying that's the case as I obviously wasn't there, just something to think about.
  - donall 2y ago
    Fair comment. And I'm usually suspicious of young engineers wanting to implement the new hotness and I'm also a fan of "if it ain't broken don't fix it". In this case, though, the system was in very rough shape. Our customers were complaining about data problems which we had no way to fix (short of manually editing the prod db, which was the SOP). I definitely took it upon myself to do something that nobody had asked for, but it was because the people in charge were entirely asleep at the wheel! They did not last long in their positions.
blacklight 2y ago
Been there. There was this old fashioned developer in one of the companies I worked for a decade ago who never understood nor embraced version control (we were talking of SVN at the time, not even git). Luckily that wasn't the case for all the others developers in the company. But when it came to the projects he owned, I witnessed several scenes along the lines of "hey, customer X has an issue with your component Y, what version do they have?" He had a spreadsheet where he kept track of the versions used by every customer. Once identified the version, he would open (no joke) a drawer in his desk and pick the right USB stick with that version on it. I've always wondered whether this overhead was a worth price to pay for not wanting to learn a couple of SVN commands.
- icedchai 2y ago
  My first job out of college, way back in the 90's, I had to "train" an entire department on how to use CVS. This wasn't a startup. This was a billion dollar, established company with 1000's of employees.
- Aloha 2y ago
  He was in essence doing manual version and release control - just.. like you would in 1990 with floppy disks.
- cyanydeez 2y ago
  Perhaps he was more concerned with job control, and saw centralized code as a detriment to his job security
  - throwaway03254 2y ago
    I have a confession to make. I was working in a company where the main development center was located in a different country. The main development center wanted to centralize all code on a single SVN server and move the build process to Jenkins I said we already had source control (CVS) and a a working build process (a script on a server that pulled from CVS, built a jar file and deployed it to X servers). We were too "busy" at the moment but would look into it in the future. This never happened. The real reason was that I was concerned that the central development center would take over the project, as they had tried to do so in the past. Looking back, I should probably have let them take over as there was more than enough work for everyone.
kwhitefoot 2y ago
Some of these stories sound a bit far fetched, especially those that involve Unix systems. RCS was released in 1982 and CVS in 1990 so Unix systems have had version control available for over forty years.
- bdbdhxhdh 2y ago
  There is a lag between availability and mindshare to apply a technology. I myself introduced SVN as versioning solution to a company in 2007, years after git was available I and I was aware of it git simply didn't have the mind share jet, thus I went with what I knew
  - Iwan-Zotow 2y ago
    Hear, hear Git was too weird for a team, but svn with its linear timeline was a hit. They are still with svn, and for a small shop it's ok
  - zimpenfish 2y ago
    > I myself introduced SVN as versioning solution to a company in 2007 In 2013, I was tasked with writing wrapper scripts around SVN to make it look like SCCS to avoid confusing the development people who only knew how to use SCCS whilst the company migrated to SVN. Fun but traumatising.
- cruffle_duffle 2y ago
  I can assure you they are true. Version control was still “controversial” in a lot of shops for quite some time. Plenty of places had the classic “v1, v2.3, v2_next_debug_cruffle_duffle” way of managing versions for a very, very long time. Well into the subversion era…
  - CRConrad 2y ago
    I know places that still do it that way. Not necessarily tiny places either.
- teractiveodular 2y ago
ics 2y ago
I was in this position before but would point out that there is a tactical approach when you know that others will not follow. I set up a cron job (on Windows, not really cron) to check scan the network location for updated source files. The git repo was on my drive and on the corporate GitHub account, safe from those who should have been using it. Whenever files changed it would just auto commit on the main branch with the username included in the message. I could do whatever I wanted on my own branches, keep track of what others were doing, and essentially wield git. You don’t have to try to be a hero inflicting proper source control upon your teams (their perspective) to still occasionally appear like a wizard to save them from inevitable, oft-occurring peril.
gumby 2y ago
Astonishingly, It took a long time for revision control to become widespread. Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forking was generally considered a tragedy, something I managed to change five or six years later). Rich Pixley came and said “all of our code should be in revision control, and I want to use this newfangled thing called CVS.” Michael was OK with it but John and I were steadfastly opposed. We agreed to an experiment, grudgingly, subject to a whole whiteboard of absurd conditions (“must be transparently integrated into emacs so we don’t have to know it’s there). Pixley agreed to all of that and then ignored all of it completely. It was immediately such a win that everybody adopted it without complaint, including us two obstreperous a-holes. A few years later a preventable crisis was how revision control first became client-server.
- asveikau 2y ago
  Linux was managed with tarballs and patch emails until a little over 20 years ago when they started using bitkeeper, then Linus wrote git.
  - NetOpWibby 2y ago
    What a legend
    - fijiaarone 2y ago
      In the story above, Linux Torvalds is Gitfolye.
- jimbokun 2y ago
  > Pixley agreed to all of that and then ignored all of it completely. Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.
ragebol 2y ago
Have been that person before. As an intern, and they even listened! In the days of SVN, just before git, so I ran a server in my laptop and my manager somehow decided we needed a big Red hat server or something, IIRC. In a 20 ppl company.
MBCook 2y ago
I never had to deal with “we don’t use source control”, luckily. One company I joined was cleaning up the last vestiges of “customize it for every customer by letting people edit their copy on server,” which predictably turned into a mess. They were all very small customizations to styles or images but enough to make upgrades a total mess. I did work at a company where despite having full source control they didn’t actually know of they could ever deploy the server component again. Edits got made to the live server but then made again in source control, or vice versa. There was one more senior person who couldn’t be talked out of their old workflow. In theory everything matched.Eventually they even checked and got it all under control where they were positive it was the same and kept it that way. But it had only ever been deployed from scratch… once. And for like 15 years it lived there and kept getting upgraded. It would all be moved when new hardware was brought in. But it wasn’t installed from scratch. We truly did not know if we were capable of doing that. It is possible if that server was destroyed and we couldn’t restore from a back up it would take us an unknown amount of time. Even though in theory deploying should be as simple copying the files and starting the web server. Were there odd configurations that had been made eight years ago that kept it running? Some strange permission changed somewhere? I wasn’t on that team. But it always made me nervous. That was absolutely a core application of the business. I really like small shops sometimes. You get a lot of freedom and get to have your hands in a lot of disciplines. You learn a lot of things, including things that’s should never be duplicated.
- yyyfb 2y ago
  That's not an exception, I think? Most online service businesses that I've worked with wouldn't be able to run their operations from a "cold start". That takes a lot of effort to engineer and in practice it doesn't happen enough, so that's a risk that most are willing to run with.
  - MBCook 2y ago
    Really? Everywhere else I’ve worked has been able to. I’m not saying all the data in the database. Pretend your DB is fine or you have a full up to date backup. I know losing that KILLS businesses. But in my example just the web server immolated and the backup tapes sitting next to it melted. Can you set up a new web server? As a Java developer it’s usually install OS, add Apache + Tomcat, set a few basic config parameters, copy WARs. PHP or Python is even easier. They didn’t know they could successfully do that.
camilla4151 2y ago
[dead]
deleted 2y ago
[deleted]
freetanga 2y ago
Software Configuration Management has existed as a discipline and with proper tooling for at least 50 years. Mainframe and VAX machines had tooling in the early 80s. For VB Sourcesafe was the go to tool if memory serves. This is not a case of new vs old, rather incompetence vs competence.
yyyfb 2y ago
It's maybe better than to take the pain to set up git only to see people use it in the same way, setting up a gazillion branches called V1, v2_customer_fix, v3_final, etc...
fencepost 2y ago
I think you misspelled "Visual SourceSafe" - and depending on when this VB was written it might actually predate VSS as a Microsoft product. Source code management back in the days of VB (long before VB.Net) was not the same as what you see today.
BobbyTables2 2y ago
Wait until you work in an industry where customers are extremely reluctant to upgrade and each one lands on a different release. Then you have to start tracking the limitations and issues in 2-3 year old releases and the solution to issues is never “upgrade to the latest”. At that point, source control starts feeling worthless.
samspot 2y ago
Setting up git is the easy part. We all used it. Except the owner of the company who would fix bugs in prod and not tell anyone. Then next release we'd unintentionally un-fix those bugs because the fixes never made it back to source control.

deleted 2y ago

[deleted]

culi 2y ago

Can I ask roughly what year this was? I can't even imagine something like that today

hleszek 2y ago
2008
nine_k 2y ago
Visual Basic was a smash hit released in 1991. For reference, Perforce was released in 1995, Subversion in 2000, git in 2005. RCS existed, but only in the Unix world.
- icedchai 2y ago
  Don't forget CVS, released in 1990. It was popular for about a decade before Subversion took over.
  - nine_k 2y ago
    Did it run under DOS? I'm not even asking about a win16 native GUI client. (You can still enjoy CVS if you want to contribute to OpenBSD, for instance.)
    - icedchai 2y ago
      I'm not sure. In the mid-90's, I worked at a Solaris and NT shop, which was where I first used it.
    - mike_hearn 2y ago
      Yes, and for Win95 users there was actually a very nice GUI plugin to explorer called TortoiseCVS. The funny thing about CVS and Subversion is that they were last version control systems that non-programmers could actually use. TortoiseCVS/TortoiseSVN were easy enough that with a lot of training you could get technical writers or artists to use it too. Valuable for game projects. With git, forget about it. It can't even handle binary files well.

thayne 2y ago

> There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. I once ran across a (c) program that had 26 variables, each one letter long, one for each letter of the alphabet. They were all global variables. And many of them were re-used for completely unrelated things.

bdbdhxhdh 2y ago
wow this is just evil I want an obfuscator doing such a transformation:D
leni536 2y ago
Ah, the "user-defined registers" paradigm.
tengwar2 2y ago
I inherited a control program from Risø National Laboratories. It had roughly 600 globals of the form A9$, three local variables, and one comment - "Midlertidig". However on a more practical note, the "Java" used on Smartcards effectively requires that all variables be treated as constants, other than one array. You dynamically allocate the array when handing an event, and it only lasts for the duration of that event.

sghiassy 2y ago

This is nightmare-fuel. How does this happen? As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices?? Honest question. How does our profession root out intrinsically obvious bad practices?

lazide 2y ago
It happens because it’s easier - until it’s impossible, anyway. The training and best practices you’re talking about is learned experience about how to avoid it getting impossible. But that almost always involves expense that the business side considers ‘stupid’.

insane_dreamer 2y ago

How long ago was this?

lowdownbutter 2y ago

My first role was with a company who had hit a limit for VB6 variable names iirc. So they'd all been renamed to shorter names. This may be the same issue. They were in the process of rewriting in VB.net.

DidYaWipe 2y ago
That sounds like they were doing something dumb. VB6 did not have a limit for variable names, unless it was 255 characters or something. And yep, I just checked: The limit was 255 characters.
- lowdownbutter 2y ago
  I think more like globally. Like there was a fixed amount of space for all variable names.

matheusmoreira 2y ago

> with some still having bugs fixed years before for others Hopefully they won't discover at some point that customers were in fact depending on those bugs. That's poison for the programmer's soul.

disqard 2y ago
Yup! Hyrum's Law :)

osigurdson 2y ago

There is "do things that don't scale" and the there is bending over backward to do things in the dumbest possible way. Not sure where this lands.

makmanalp 2y ago

As much as this stuff is nuts to think of today and there's tons to hate, I am kinda nostalgic for some aspects of my experience of working at a place where software is maybe needed and/or valued but isn't a core competency. Or maybe a time when software was a new fangled thing that hadn't fully been integrated into corporate structure yet: - No one having any preconception of how you're /supposed to/ do things or whether you'd even be the type of person to know, so you just kinda figure it out yourself. You spend a lot of time on reading and learning skills. Version control? Wow, cool, what's git, let's try that! A new graphing library? Let's give that a shot, maybe it'll make things easier! You want XYZ? Let me go read about that for a day. - No one having any idea what's even possible: being treated like a wizard for introducing the tiniest piece of automation or improvement that makes someone's day easier or doing something they never thought was possible. Lots of appreciation and excitement for showing and teaching people new things (... and I know this is somewhat selfish and ego driven, but who doesn't like being appreciated?) - Similarly people having no idea how long those things should take which, tbh, can be a nightmare if you're not trusted and respected enough to be consulted but also great if people believe you when you say it's gonna take 3 months. - Beyond the basics just being mostly kinda left alone to do your job however: no standups or tickets or the 30 other kinds of daily (micro)management that is probably necessary but ends up feeling tiresome and stifling at an individual level - Not being part of software company "culture": no performance review driven development and promo packet madness, no weird rating and ranking systems, no OGPs or KPIs. No ladder. Your bosses think you did what was required of you, so then you're good, and if it's a good year you get a raise, and that's that. I do recognize that with a bad boss this can be a terrible and unfair spot to be in - but again, subjectively with a decent enough boss it felt like a lot less weight on my shoulders at the time. - No hacker ninja pirate segway mini quadcopter you're the smartest people in the world and we're the best company to work for sort of b.s. - Socializing with people who are good at and love to talk about stuff other than software Reading over that, I'm thinking maybe I lucked out a lot and that wasn't most people's experience from that era. And there's some level of rose tinted glasses going on. And/or maybe my years in the rat race are starting to show :-)

Aeolun 2y ago
Don’t think so. My first job was kind of like that. I don’t even know how they thought that little old me just out of university could be left alone to successfully build applications on my own, but I think people trusted a lot more during that era because eternal september hadn’t arrived yet. Working directly for the users without any weird BA/PM/TA shit in between is glorious, both because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue), and in a place like that you can likely deploy it in the middle of the day and immediately improve their workflow. It still amuses me that IT was located together with finance, because we did reports xD

TacticalCoder 2y ago

> At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio. Holy smoke! That's actually the most creative solution (horrible, but creative) I've ever heard to fix a Heisenbug: https://en.wikipedia.org/wiki/Heisenbug https://en.wikipedia.org/wiki/Heisenbug

pletnes 2y ago
Often it’s enough to compile in debug mode and maybe set an env var! This sounds like a horrible approach by all measures I think.

tansan 2y ago

These type of stories are a humbling reminder that PMF always beats engineering excellence

zarathustreal 2y ago
Well that depends entirely on what you consider to be the goal - as a software engineer, your role is entirely concerned with engineering excellence. As a member of a team, especially a team of extremely highly paid and highly educated individuals, it is your duty to spend your time (and thus, the company’s resources) efficiently by doing what you’re educated, qualified, and hired to do.
- jahewson 2y ago
  Engineering excellence is a means to an end. Great engineers solve problems - sometimes by writing software.
- williamdclt 2y ago
  Few people agree that the goal of SWE is engineering excellence. It is to solve business problems. Engineering excellence is a means to a goal: to be able to solve _difficult_ problems _correctly_ and allow solving new problems _in the future_. All these things can be traded off, and sometimes aren’t even needed at all

onion2k 2y ago

I don't know how they convinced the users to do this and how they managed with the license but it was done Enterprise and business users will wade through endless swamps of crap if the software provides enough value. This is a lesson in why "it must be perfect before we release it" is such nonsense - that just says the value your app provides is so low that users barely care and they'll abandon it at the first problem. If that's the case you're not providing anything worth paying for.

croes 2y ago

That's why cloud solutions exist. Now you only need to run the app in debug mode on your own server.

feketegy 2y ago

This is that meme when the pizza guy walks into a burning room. Also, this comment is worthy of The Daily WTF.

Forgeties79 2y ago

This sounds like what I see when an inexperienced filmmaker takes on a big project and hands me the “organized” drive to edit, but way way worse and more consequential lol

llm_trw 2y ago

I was in a broker trader in 2016 where this was still the case. I was brought in when an old sheet cost them $10m on a bad trade because the yahoo finance end point it was hitting stopped responding and it just used the last value it had gotten - three months before.

DragonMaus 2y ago

This is entirely typical of especially VB scripts. When I was a software engineer for a Fortune-20 company, I spent more time debugging (and trying to normalize, though that met with mixed levels of resistance) VB applets than anything else.

fencepost 2y ago

Oh dear god no. The solution is not throw VS at it and run from the code, the next step is some combination of excessive logging (which to be fair may resolve the issue all by itself) and/or throwing in a ton of DoEvents because Visual Basic.

acwan93 2y ago

Dear god this is pretty much what I went through when I started taking over a company with a 35-40 year old codebase. Files spread everywhere, no consensus, and supporting customizations for thousands of customers who we didn’t know if they were even still using the system. It took five years and the firing of the long-time “head” programmer until some meaningful change was made.

titusjohnson 2y ago

Oh man, that brings me back. My first tech job was an ecommerce company that a basic online cart with backed by incredibly in-depth set of industry catalogs. We also sold a marketing package as an add-on to the online store where we would proactivly contact our customers and get the info from them to replicate their monthly/weekly physical advertising on the web. This was back in '05-ish, lots of money to be made just helping people get their business online. We had a group of talented designers/artists wrangling Photoshop and pumping out a few of these designs a day, each, and as we scaled up and gained a lot of repeat customers, tracking these PSDs became a big problem. The designers were graphically talented, not technically savvy. The PSDs were stored on a shared NAS drive that the whole company could see. The designers had a complex naming system to manage major revisions, but overall there was no "history" beyond the classic "_v2_2008_09_best.psd" naming technique. Several times every week I had to fix "accidentally dragged the PSD folder somewhere and lost it" level problems. Getting IM'd from tech support because the underlying server was falling over trying to clone the multi-GB folder 3 times, logging into a workstation as Admin and searching for an updated PSD that the vacationing Designer hadn't synced back to the NAS before leaving for work, that kind of thing. As soon as I was promoted to Supervisor I made my first big move. It took a lot of training, far more talking than I thought it should (back then I didn't know anything about politics), but I was able to get SVN implemented to replace the NAS share. I wrote quick-reference documents, in-depth guides, (this was before I knew that no one reads anything, ever, for any reason) and eventually just had to do one-on-one training with everyone just to explain the concept and basic useage. One of the most satisfying feelings of my career continues to be watching attitudes change over the course of a summer. None of the design-y people liked the new set of hoops they had to jump through. Check-Out, Check-In, Lock, etc, it was "too much". Then, at a happy hour someone mentioned how we hadn't lost the PSD folder in a while. Later someone came to me panicking because a client wanted to re-run an ad from 2 months ago with a couple tweaks, and she didn't have the source PSD or the source material -- I did a live demo on how to get a historical version back, and that's when it really clicked with everyone. With internal political will behind the toolset, it now became an IT problem, as our SVN useage was nothing like Engineering's usage. Of course file locking was a huge PITA, that feature replaced "forgot to copy the changed file back before vacation" as a problem category. But it also eliminated the problem where 2 people would open the same PSD directly from the NAS share, make their changes, and only the last one to save gets their work persisted. So, a toss-up I guess.