67 ms·

"We ran out of columns"

• Waterluvian 2y ago
  The first line really hits me hard. There’s something so incredibly freeing about being a kid and doing stuff like coding. There’s simply no expectations. Even the smallest project felt like such an achievement. But now I code professionally and I don’t know how to turn off engineering brain. I don’t know how to be okay doing something poorly, but on my terms.
  • actionfromafar 2y ago
    What ever small success I had outside work has been whenever I managed to turn off Enterprise brain.
  • captn3m0 2y ago
    Writers often say that your first N books will be crap, and the answer to that problem is to just write more than N books. I feel that’s true of software as well - the first years of the magical programming I did without any expectations meant those projects were utter crap yet functional.
  • datavirtue 2y ago
    It's because you don't get to write apps from scratch enough. Everything is a POC. You get it working. Optimization is just dumb until the project is so big it doesn't fit in the developers brain anymore. Then you write tests for the core part so you can keep interating fast. You will see parts that clearly need refactorwd before you can move on. Also, be aware of, but don't get hung up on, engineering best practices. Chances are that someone is going to see your POC and put it in front of a client or promise it to someone. It will enter service being less than perfect, if you are lucky. When you pull this off people will tell stories about you and brag about you when you are not around. None of them will know anything about the code engineering.
    • zbentley 2y ago
      > When you pull this off people will tell stories about you and brag about you when you are not around. True! > None of them will know anything about the code engineering. Not true. In environments where it's very easy to fuck up (hyperconnected codebases, lots of legacy tech debt, fragile systems generally), programmers who create tools/libraries that let other engineers get work done without blowing their feet off or spending huge amounts of time on silly things do get praised. Maybe the praise comes from different people than in your example, but it definitely is rewarded. Two caveats: this only happens in certain environments; some gigs are just culturally unable to care about quality tools (death march feature factories and the like). Also, you actually have to save time and reduce defect rates with your engineering work; starting an architecture-astronaut rewrite that isn't actually useful for other programmers' day-to-day work doesn't count. Making "one tool to rule them all" that requires a Ph. D or pairing session with the author doesn't count either.
  • neilv 2y ago
    I think you can add a mode to engineer brain, in which it's aware when it's kludging something, but kludging is the appropriate thing to do. Might help to have enough experience that you have (well-placed) confidence in your gut feel for how much engineering a particular thing needs. (More common is to not have an engineer mode, or to not have enough experience to do it well.) If you don't have the kludge mode, try some things where kludges are outright necessary. One time I recall this was when writing Emacs extensions. For example, it used to be that you'd hit a point where normal programmatic navigation/parsing of the buffer was just too slow, and a well-placed regexp operation that worked 99.99% of the time (and the other 0.01% of the time it's OK enough, and the user understands) made the rest of the code viable. Another example is personal open source projects, when you really want to experiment with an usual implementation approach, and you give yourself permission. (Maybe don't do this when the open source is done only for resume purposes, where you're trying to demonstrate you follow all the current fashionable conventions. You're almost guaranteed that someday a new-grad on a public forum or not-very-technical hiring manager will stumble across that bit of code, and fixate on the one thing they think they recognize as a bad practice. Documenting the kludge and rationale, in, say, a code comment, is great practice, but, again, it will also draw the attention of the least-skilled. Much like, if you're relating an anecdote in a job interview, and implied are 20 things you did right and 1 thing you did brilliantly, and you note as an aside one mistake you made, most of that will whoosh over the head of the dimmest FAANG interviewer, but they'll latch onto that mistake you spelled out, as a wise insight they have, and that's what's going in their report. :) Then, armed with an experienced multi-skilled brain, when your startup's imminent MVP launch is facing crazy constraints, you triage what bits need to be rock-solid so they'll absolutely work and not fail, what bits need creative kludging so you can hit your launch window for other key requirements, and what bits you need to mitigate any compromises. Ideally, you have the wisdom to know which is which, and can activate different skills for each kind of work. That Turbo Encabulator needs to be well-machined, but the new sensor you just decided it needs for a test doesn't need a week to be drilled into the engine block, nor a mount to be designed and CNC'd or 3D-printed, but the nearest Velcro or zip-tie or maybe wad of chewing gum will do.
    • Waterluvian 2y ago
      All such great advice. The one time I’ve found success is when I figure out the entire scope of the little project up front. Then I find myself not even worrying about maintenance or scalability. I know I won’t need that extra class abstraction or modularity or whatever. I just don’t like planning at the start, but it’s probably, counterintuitively, part of the answer. P.S. you opened a parenthesis and forgot to close it so now everything I read all day is part of your anecdote)
  • jimmyhmiller 2y ago
    This is something I've had to train myself to overcome. The first step for me was having a place where I clearly signaled that this wasn't my best work. Where the rules were allowed to be broken. That place is my junk drawer of code[1]. I have a zsh alias `alias changes='git add . && git commit -am "Changes" && git push'` that I use for my commits in the repo. This all may feel silly, but it's what I needed to get back to that time of playing. Where I never worried about the code. But where I also didn't feel I was wasting my time working on code no one could see. I'd definitely recommend trying something like that if you are struggling with it. [1]: https://github.com/jimmyhmiller/PlayGround https://github.com/jimmyhmiller/PlayGround
    • neontomo 2y ago
      next step: not using git ;-) i considered a few times to make an alias like this but i feel like git is so precious to get right that i would rather type out each command just to "feel" what i'm doing and be active in it.
• PreInternet01 2y ago
  > I miss that direct connection. The fast feedback. The lack of making grand plans. There's no date on this article, but it feels "prior to the MongoDB-is-webscale memes" and thus slightly outdated? But, hey, I get where they're coming from. Personally, I used to be very much schema-first, make sure the data makes sense before even thinking about coding. Carefully deciding whether to use an INT data type where a BYTE would do. Then, it turned out that large swathes of my beautiful, perfect schemas remained unoccupied, while some clusters were heavily abused to store completely unrelated stuff. These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob. Then, some indexes specified by `json_extract` expressions, some clever NULL coalescing in the consuming code, resulting in a generally-better experience than before...
  • jrochkind1 2y ago
    Oh good question on date essay was written -- put dates on your things on the internet people! Internet Archive has a crawl from today but no earlier; which doesn't mean it can't be earlier of course. My guess is it was written recently though.
    • lexicality 2y ago
      created 14 hours ago https://github.com/jimmyhmiller/jimmyhmiller.github.io/commit/1e007eba3843eadaba97c1d713b7076580a616da https://github.com/jimmyhmiller/jimmyhmiller.github.io/commi...
      • __MatrixMan__ 2y ago
        But clearly in retrospect. It sounds like some of the things I was doing in 2009.
        • chatmasta 2y ago
          It includes a reference to Backbone and Knockout JS, which were released in 2010, so presumably it was around that era. The database, though, was probably much older...
• cowsandmilk 2y ago
  He acts like sequence key is odd, but that’s quite normal in database world. https://www.postgresql.org/docs/current/sql-createsequence.html https://www.postgresql.org/docs/current/sql-createsequence.h...
  • williamdclt 2y ago
    yeah, I've seen tables with a single row and single columns a few times, for alright reasons. Sometimes you do just want to store a single global value!
  • MarceColl 2y ago
    Yes, but both these have very different properties. He said (I don't know if its the case) that the db didnt have an autoincremental type. Postgres uses these sequence objects to implement autoincremental ids as he was referring to, they are implemented in-engine and are very fast and have already solved data races. In the article, what he complains about is not about what a sequence is, but about implementing it manually with a table that is read, incremented and then saved. This us more expensive, and depending on how it was implemented you need to take care of the whole data flow so you are unable to allocate the same id twice. That's what he considers odd
    • pbronez 2y ago
      The scary thing to me about that setup is how the global value is updated. Every individual script must successfully increment the value to avoid duplicate keys. Really hope they had a “get key” stored procedure to handle that.
      • SoftTalker 2y ago
        It's possible they had some requirement for gapless ID numbers. You can't do that with a sequence.
  • _elf 2y ago
    Databases have built-in features for this now. What the author is talking about is a regular table. In reality, that wasn't too unusual to see because frameworks would use that technique because it's a lowest common denominator across RDMS.
• codetrotter 2y ago
  > went by the name Munch How do you pronounce that? Was it like the word munch in “munching on some snacks”? Or like the name of the painter Edward Munch? https://www.nrk.no/kultur/nrk-endrer-munch-uttale-1.539667 https://www.nrk.no/kultur/nrk-endrer-munch-uttale-1.539667 (note: this link is in Norwegian)
  • jimmyhmiller 2y ago
    As in munching on snacks
    • thrwaway1985882 2y ago
      Hey former colleague - just had to say hello on a comment where you might see. I started reading this article and everything started feeling so familiar... as soon as you told me Munch was the resident shaman, everything clicked. My favorite factoid for others was that when I was there, we used split-horizon DNS to squat on an in-use domain name for tons of internal services, including Github. I kept wondering what would happen if the owner realized & set up his own services to catch people who weren't on the VPN.
      • rendall 2y ago
        ::Edvard Munch scream emoji::
• trte9343r4 2y ago
  Slicing columns into multiple tables is fairly common type of sharding. Sort of SQL way to do columnary store.
  • pelagicAustral 2y ago
    This is such an horrendous practice, I am yet to find a database were this makes sense. Maybe my brain is not wired for it
    • zo1 2y ago
      The problem is maybe not so much the splitting and putting extra columns in a separate table. It's that you even have a table that large that it necessitates such a thing. Worst case you have a main table and a detail table that has a one to one correlation to the main entity table.
      • magicalhippo 2y ago
        Why is that worse than a couple of dozen joins?
        • zo1 2y ago
          Because that means your data is highly denormalized and has plenty of duplicates. But in all likelihood it means no one knows wtf this table actually represents and you should be firing people. I've seen this play out. Usually the many columns is because everyone misuses the table and eventually their special little business scenario or "filter" needs to be a column. Bonus points is whoever has to reference this table, they have to copy over whatever the hell your PK seems to be, and the cycle repeats, this time a bit worse. Last place I did a brief project in had this. Queue 1000 tables spread across 25 schemas, each table having wide PKs, 20 redundant indexes on each table, and despite all this the database performs poorly. No one can tell you what each table represents, the table names are meaningless and the same data is everywhere. In order to get anything done you have to ask a small cabal of priests that knows the processes that write between these tables. After about 10 years, a partial rewrite happens and you now have 1/3rd of the data on each side with plenty of duplicate and overlap because hey. I feel torn, I really wanna name&shame this company as a warning to all developers thinking about working there.
• dakiol 2y ago
  Honestly, I like to work on such systems because: - there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it) - it’s back to school again and everything goes. You can implement features in any way you want because the constraints the system imposes. Granted, sometimes it’s painful to add functionality - there’s usually no room to subjective topics like clean code and architecture. The most important thing with these systems is correctness (and this is usually an objective topic) - nobody can blame you for something that doesn’t work. It’s always the fault of the legacy system I wouldn’t recommend working on such systems to junior engineers, though. I don’t really like to work on “perfect” codebases where everyone follows the same pattern, with linters, where if something breaks is because of your shitty code (because the codebase is “clean”). It’s very frustrating and limiting.
  • korhojoa 2y ago
    I mean, it is kind of nice to notice that something isn't going to work because you have the linters and tests. When your developer count goes up, chances that erroneous behavior is included also goes up. I've created proof-of-concepts that worked perfectly but would make you cry if you looked at how they worked. Eventually they became that mess. Everything is a self-contained unit so it doesn't mess anything else up. Of course, there is always time to keep adding new stuff but never to refactor it into what it should be. I prefer the way with linters and tests, it at least lessens the chances of whatever is put in being broken (or breaking something else). (Then again, somebody putting "return true" in a test _will_ surprise you sooner or later)
  • poikroequ 2y ago
    > there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it) And there is nothing worse than a crappy codebase the company won't let you improve.
  • SoftTalker 2y ago
    You also get to see some genuinely creative stuff that works well, written by people who aren't indoctrinated in a particular approach.
• gamepsys 2y ago
  > All that remained were ragtag interns and junior developers. For many people, their first job in software engineering is the worst codebase they will deal with professionally for this reason. The first job hires lot of people with little/no experience. As soon as someone gains some experience than can move on to better paying jobs, where there are better developers with better standards.
  • prudentpomelo 2y ago
    This is the exact boat I am in. I always say the best thing about our codebase is the worst thing: junior developers can do whatever they want.
  • mleo 2y ago
    Worst code bases are often ones taken over from IT consultancies. They drive young, inexperienced developers working many hours to deliver functionality. While the project may start out “clean” using whatever is the current hotness in technology, at some point getting stuff developed and throwing over the wall to QA is the important part.
• surfingdino 2y ago
  I had dubious pleasure of working with similar codebases and devs. I'll remember one of those guys forever, because whenever he wanted to work on a new branch he would clone the repo, make changes to the master branch, and push code to a new repo numbered repo0001, repo002, ... He refused to change his ways, because "I have a PhD so you are wrong". Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type. That made porting code fun.
  • marcosdumay 2y ago
    > Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type. The standard does not have a boolean type. It's a postgres extension that the other open source databases adopted (because, yeah, it's obvious). But the proprietary ones insist on not having. The official recommendation is using byte on MS SQL and char(1) on Oracle. Both are ridiculous.
    • nolist_policy 2y ago
      C programmer joins the chat.
      • mort96 2y ago
        Hey we've had _Bool and <stdbool.h> since 1999!
      • dvfjsdhgfv 2y ago
        Well, everything is a boolean in ANSI C. If I use `if (a)` I know the code will be executed only if `a` is nonzero.
    • bdcravens 2y ago
      Pedantic, but the type is bit in MSSQL.
    • breakingcups 2y ago
      I don't think that's true. Even 2005 had a bit type and it's better optimized for space: https://learn.microsoft.com/en-us/sql/t-sql/data-types/bit-transact-sql?view=sql-server-ver16 https://learn.microsoft.com/en-us/sql/t-sql/data-types/bit-t...
• Nexialist 2y ago
  My worst codebase story: In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java. In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server. Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files). Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released. This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files. Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why. It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.
  • sir-dingleberry 2y ago
    This is insane. Thanks for the post.
  • pelagicAustral 2y ago
    This sounds so much like dealing with MS Access databases... Unfortunately, part of my responsibility in my current role is to manage a handful of Access applications... they are ancient, I am talking early 2000's. They are the most unruly thing I have ever had to work with, and I will not go into details but your story reminds me so much of having to work on these apps...
• tryauuum 2y ago
  Would be great to work in such a company as a Linux guru. There are so many entangled services and machines that you feel like an Indiana Jones. You ssh into a machine and feel century-old dust beneath your footsteps. And you never know what will you find. Maybe a service which holds the company together. Maybe a CPU eating hog which didn't do anything useful last 3 years. I don't enjoy writing new code much. But in such an environment even with my limited skills I can do decent improvements, especially from security point of view. Feels great
  • noisy_boy 2y ago
    Basically the entire application is loaded with low hanging fruits and things that look like fruits but are bombs that explode upon contact.
  • arnorhs 2y ago
    Well it sounded like there are exactly 0 Linux machines running there.. it's all windows .net / c# and a bunch of native windows apps, as I understood the article. But maybe you can replace your statement with "windows guru" and SSH with "remote desktop" and perhaps that would be fun
    • tryauuum 2y ago
      well, with linux at least I get get the source code of the kernel and of the MySQL when given a database server which hasn't been restarted for 7 years
  • doublepg23 2y ago
    When I started my recent job the team kept referring to a box running "weird linux". After getting on-boarded and accessing the box it turned out to be running a very old version of OpenBSD. To this day I'm curious who had the wherewithall to install OpenBSD in prod but was seemingly ignorant of the 1yr support cycle.
    • xp84 2y ago
      I started reading and was hoping someone had installed one of the Linux distros that were popular at the turn of the millennium, like Mandrake, Slackware, etc. and it was still trucking along 25 years later.
• airstrike 2y ago
  This is glorious. I can imagine people could write versions of this for every industry out there and they would all be equally fun to read.
• Twirrim 2y ago
  Back about 15 years ago, I worked for a web hosting company that provided some sysadmin consultation services. Customer paid us, and I would take a look. I had one customer who came back with the same request, slightly differently worded, every single month, and every single month I'd say the same thing. They had this site they were running that was essentially a Yellow Pages type site. They had a large set of companies with contact details, each with multiple business categories associated with it. You'd choose a category, and they'd return a list of matching companies. The problem was the site was really slow. I took a quick look around, and saw that all the time was lost querying the database. Taking a quick look at the schema I discovered that their approach to categorisation was to have a TEXT column, with semicolon separated 4 character strings in it. Each 4 character string mapped to a business category. So when someone wanted to load up, say, all pest control companies, it would check the category mapping table, get the 4 character string, and then go to the companies table and do: SELECT * FROM companies WHERE categories LIKE "%PEST%" So on each page load of the main page type the site was there to provide, it did a full text search over the category field for every single record in the company table. I guess that's probably okay for the developer without real world scale data, and real world traffic counts to worry about. But they had lots of data in the database, and that category field could have dozens of categories against a company. As soon as they had more than about 4-5 simultaneous customers performance started tanking. I could never get them to accept that they needed to rethink the database schema. One month they were bleating about how is it possible that Google can manage to do such a search across a much larger amount of data, much faster. They really didn't like my answer that amounted to "By having a sane database schema". All they were willing to do was pay over the odds for our most powerful server at the time, which had enough capacity to hold the entire database in memory.
  • throwup238 2y ago
    In case anyone is looking for a performant way to implement categories like that in Postgres: https://news.ycombinator.com/item?id=33251745 https://news.ycombinator.com/item?id=33251745 I stumbled across that comment a few years back and it changed the way I handle tags and categories so just sharing it here. If anyone has an equivalent for Sqlite, I’d love to hear it!
• dmd 2y ago
  Probably some of the worst code I ever worked on was a 12k+ line single file Perl script for dealing with Human Genome Project data, at Bristol-Myers Squibb, in the late 1990s. The primary author of it didn't know about arrays. I'm not sure if he didn't know about them being something that had already been invented, or whether he just didn't know Perl supported them, but either way, he reimplemented them himself on top of scalars (strings), using $foo and $foo_offsets. For example, $foo might be "romemcintoshgranny smithdelicious" and $foo_offsets = "000004012024", where he assumes the offsets are 3 digits each. And then he loops through slices (how does he know about slices, but not arrays?) of $foo_offsets to get the locations for $foo. By the time I was done refactoring that 12k+ was down to about 200 ... and it still passed all the tests and ran analyses identically.
  • rokkamokka 2y ago
    Goes to show that solutions don't need to be good to be impressive. Because damn, I'm impressed he did that
  • dezsiszabi 2y ago
    That's a very impressive achievement, reducing that to around 200 lines. Congrats :)
    • itsameputin 2y ago
      Well the lines could very well be 1000s of bytes long. Perl oneliners used to be a thing in a company that I used to work for.
  • sandebert 2y ago
    Nice. But I have to ask, considering it was Perl: Could an outsider understand it after you reduced it to 200 lines?
    • dmd 2y ago
      Certainly better than they could when it was 12000.
• deleted 2y ago
  [deleted]
• lemme_tell_ya 2y ago
  My first day at my first full-time programming gig, I was asked to look at some reporting job that had been failing to run for months. I logged in, found the error logs, found that it needed a bit more memory assigned to the script (just a tweak in the php.ini) and let the team lead know it should run fine that night. He was shocked, "Dude, if you just fixed that report you probably just got a promotion, no one has been able to figure that out for months." He was joking about the promotion, but my boss was just as shocked. I'd realize later that most the other people on the dev team didn't like linux and wanted to rewrite everything in .NET and move everything to Windows so no one even tried with anything related to any of the linux machines.
  • mleo 2y ago
    I know things have gotten somewhat better, but the amount of wasted time and latency of using RDP and Windows UI for development, testing and production maintenance is insane. Throw in some security requirements of RDP into host 1 to the RDP jump to host 2 and companies are just wasting money on latency. There is, often, not an appreciation of the administrative costs of the delivery. Not necessarily system admin costs, but developer and QA time associated with delivering and ongoing maintenance.
    • lemme_tell_ya 2y ago
      Oh yeah, been there. I personally find Windows to be a terrible server OS. I'd rather be stuck with a fleet of AIX machines or something equally uncommon (but still nix-ish) than any Windows system.
    • cyanydeez 2y ago
      Windows programming has a natural job security dependency that is often overlooked.
      • vasili111 2y ago
        What do you men? Can you please tell me more about it?
    • 2y ago
• interactivecode 2y ago
  sure it might be a mess, but at least it's purpose built. I love that kind of performance gains. Honestly most companies die before purpose built code like that becomes a problem.
  • bloaf 2y ago
    Truly, the Tao was alive in that company. https://www.mit.edu/~xela/tao.html https://www.mit.edu/~xela/tao.html
• qxmat 2y ago
  Jira's now discontinued server version had a sequence table to stop you sharding it. It also made disaster recovery from a hard shutdown awful. I have nothing good to say about Atlassian.
  • Atreiden 2y ago
    Atlassian, and JIRA specifically, are responsible for so much wasted time and capital expenditure. If I could get ahold of metrics like "hours spent building, using, and maintaining JIRA" versus "value obtained from JIRA" for each of the companies I've worked at, I'm pretty sure I could generate a report so scathing that nobody would ever use it again. Gatekeeping your work organization system from the people working on, and often MANAGING it is such a huge friction point I'm amazed they ever got clientele. I'm an Admin on a project right now and I can't change our status types without going through our assigned Atlassian rep. And don't even get me started on the dumpster fire that is BitBucket. Ever tried to use the API? it's somehow even more worthless than the UI.
• mrighele 2y ago
  > Now the story I heard at the time was that once upon a time SQL Server didn't support auto-incrementing ids. This was the accepted, correct answer. At a company that I used to work, they heard the same rumor, so instead of using identity columns or sequences, they kept a table with a number of ids "available" (one row per id). Whenever unique id was needed, the table would be locked, an id selected and marked as used. If there were no ids available, more ids would be added and then one used. A scheduled job would remove ids marked as used from time to time. Note that there was a single "sequence table", that was shared among all of the entities. That was not even the weirdest part. That id was unique, but NOT the primary key of the entity, only part of it. The structure of the database was fairly hierarchical, so you had for example a table CUSTOMER in 1-to-many relation with a USER table, with a 1-to-many relation with an ADDRESS table. while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
  • noisy_boy 2y ago
    > while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key. Maybe they wanted to avoid maintaining a separate CUSTOMER_ADDRESS 1-to-many table or maybe it was done to make easy reverse lookups.
• bdcravens 2y ago
  A couple of weeks ago I had a flat on the way to the airport. It was a total blowout, and my car doesn't include a spare. We were already over budget on our trip, so I had the car towed to the closest tire shop and had them put on the cheapest tire that could get me to the airport. I know I'll need to replace other tires, as it's an AWD, and I know it's not a tire I really want. I made a calculated choice to make that a problem for future me due to the time crunch I was under. Programming is a lot like this.
  • alt227 2y ago
    Aside - the fact that cars dont include spares these days is infuriating. IMO its a classic example of real world enshittification
    • bdcravens 2y ago
      Supposedly there are reasons (for example, efficiency by avoiding the extra weight) and they give you a kit that's essentially a "fix a flat" product, but in the 2 times I've had a flat in a car outfitted like this, what they provided wasn't useful, usually due to the type of damage to the tire.
    • fragmede 2y ago
      that's not enshittification! it sucks, yes, but when coined, the term referred to a platform extracting value from its users by degrading their experience, forcing them to upgrade to a premium plan. your car not having a spare isn't an example of that! https://en.wikipedia.org/wiki/Enshittification https://en.wikipedia.org/wiki/Enshittification
      • sebastiennight 2y ago
        Somebody at Toyota has read your comment and is now adding "spare tyres" to their subscription offering.
      • CRConrad 2y ago
        Many of us grew up with cars having spare tyres as a matter of course. Now car manufacturers are extracting value from their customers by degrading our experience (of safety and expedience of repair), forcing us to pay extra for a spare tyre (if even available), or even upgrade to a better car. Ordinary cars not having a spare tyre as standard equipment any more is a perfect example of enshittification.
• rendall 2y ago
  I'm glad OP was able to maintain a good sense of humor about it. Such discoveries as the nested classes of empty methods have sent me into teeth-gnashing fury followed by darkness and despair. One such experience is why I would rather change careers if my only option were to build on the Salesforce platform, for instance.
• jkestner 2y ago
  Early on in my career, at a large company, I encountered someone who took “codebase” a little too literally. At the time every department had their own developers, sometimes just employees who had an aptitude for computers. This one guy established himself by making an Access database for their core business, and when the web became a thing, built a customer site. But not on it—in it. He simply served ASP pages directly from the database, inserting dynamic content in queries. When I was asked to help improve their terrible design, I was forced to untangle that unholy mess of queries, ASP (new to me) and HTML. It was easiest to write all the HTML and insert their ASP right before I sent the files back (because I wasn’t given access to their DB/web server). Thinking “I could do better than this” got me into programming. He was a Microsoft-everything head. Finally went too far when he presented a new web interface starring a Clippy-like parrot using Microsoft’s DirectX avatar API. The executives were unimpressed and then I noted that 20% of our customers couldn’t use his site. (I probably still had a “best viewed with IE” badge on the main site, lol)
• brunoarueira 2y ago
  Once not long enough, I'd worked on 4 projects which was literally copied from the first made and changed parts of the customers and internal users according to each use of it. So, the main problem is bugs found in one project was found on the other 3 and I'd to fix the same bug! Codebases like this or from the OP is cool to learn how to not do certain things.
• omoikane 2y ago
  This codebase sounds like a haunted graveyard[1], where everyone just fixes their local corner of things and avoid the risk of untangling the existing mess. Not needing to conform to some company-wide standard is probably really pleasant while it lasted, but every such effort adds to the haunted graveyard, and the lack of consistency will eventually come back to bite whoever is still around. [1] https://www.usenix.org/sites/default/files/conference/protected-files/srecon17americas_slides_reese.pdf https://www.usenix.org/sites/default/files/conference/protec...
  • collinmanderson 2y ago
    It could maybe work if every team is responsible for their own area of the code base, where the code base starts to match the company tree
• mg 2y ago
  That is why people these days tend to use a single JSON blob instead of multiple columns. And because it is so popular, SQLITE and other DBs are building better and better JSON support into the DB. I wonder if better support of EAV tables would solve this issue better. If one could do "SELECT price,color,year FROM cars! WHERE status='sold'" and the "!" would indicate that cars is an EAV table ... entity attribute value 1 price 20750 1 color red 1 year 2010 1 status sold ... and the result of the query would be ... 20750 red 2010 That would solve most of the use cases where I have seen people use JSON instead.
  • throwaway7e8te 2y ago
    Have you looked at the "hstore" type in Postgres? It seems to cover that use case.
• telgareith 2y ago
  First real job, stayed for 10yrs, 5yrs too long, put VB6 COM objects into the asp.net codebase.
• arnorhs 2y ago
  I really love these kinds of stories. Does anybody know if there's a collection of similar stories/code bases anywhere? I guess there is dailywtf but that's mostly bugs. Probably good enough though
• jancsika 2y ago
  Oh man, this article reminds me of an article that was a parody of some horrid business logic. Something like this: a "genius" programmer was somehow, for some reason using svn commits as a method dispatcher. Commit ids were sprinkled throughout the codebase. A new hire broke the entire system by adding comments, and comments weren't compatible the bespoke svn method dispatcher. Does anybody remember this article? I really want to read it again.
  • carderne 2y ago
    The Inner JSON Effect. Most recent HN discussion of it: https://news.ycombinator.com/item?id=40923258 https://news.ycombinator.com/item?id=40923258
    • dang 2y ago
      Thanks! Also: The Inner JSON Effect (2016) - https://news.ycombinator.com/item?id=35964931 https://news.ycombinator.com/item?id=35964931 - May 2023 (11 comments) The Inner Json Effect - https://news.ycombinator.com/item?id=12185727 https://news.ycombinator.com/item?id=12185727 - July 2016 (142 comments)
  • ivanjermakov 2y ago
    Tom is a genius! https://thedailywtf.com/articles/the-inner-json-effect https://thedailywtf.com/articles/the-inner-json-effect
    • jancsika 2y ago
      Yes! Has anyone tried implementing it? If not I'm going to give it a shot. :)
      • masklinn 2y ago
        It'd be way easier (and actually not completely insane, though still far from sane) in git: you'd store the functions in blobs instead of misusing svn commits. It'd probably be a lot faster too as Git is fairly good at giving you object content. Admittedly the GC would be an issue, but I think that would be fixable: instead of a `functions` key in a json, link all the functions in a tree object, the link that tree object from a commit that way you have the author for free. And then the class name can be a ref' in a bespoke namespace. Well that's the issue that it's not necessarily clear what commits go together to compose the system, so might actually be better to replace the ref' per class by a tree for the entire thing, with each class being a commit entry, and an other commit capping the entire thing. Of course git will not understand what's happening as commit entries in trees are normally for submodules but that's fine, git is just used as a data store here.
• tbm57 2y ago
  Working on something like that would drive me absolutely batty. I am happy you were able to find your zen in the middle of that chaos. This post truly speaks to the human condition
• jbkkd 2y ago
  Sounds awfully like my first job, with the addition of not having _any_ sort of test - functional, integration or unit. Nothing. A few months in, when I approached the CTO and asked if I could start writing a test framework, he deemed it a waste of time and said "by the time you'd commit the test, it would go out of date and you'd need to rewrite it". Naturally, the build would break about 5 times a week. Boeing was a major customer of this system, so when shit hit the fan at Boeing a while ago, I wasn't surprised.
• hleszek 2y ago
  When I started at my first company, they had a very complex VB application running on dozens of customers around the country, each having some particular needs of course. There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything. At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio. I don't know how they convinced the users to do this and how they managed with the license but it was done. What happened next was even worse. There was no version control of course, the code being available on a shared disk on the local network of the company with the code copied over in multiple folders each having its own version, with no particular logic to it either, V1, V2, V2.3, V2a, V2_customer_name, V2_customer_name_fix, ... After that, when there was a problem for a customer, the programmer went there to debug and modified the code on site. If the bug/problem was impacting other customers, we had to dispatch some guys for each customer to go copy/edit the code for all of them. But if the problem was minor, it was just modified there, and probably saved on the shared folder in some new folder. What happened next was to be expected: there was no consensus on what was the final version, each customer having slightly different versions, with some still having bugs fixed years before for others.
  • fouronnes3 2y ago
    This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.
    • mleo 2y ago
      Anecdote seems long before git creation, so Visual SourceSafe maybe. Which did not work well over a WAN. Needed other tools to replicate and synchronize VSS.
      • masklinn 2y ago
        You can remove the "over a WAN" part: VSS had been designed as a local VCS, so until the addition of a proper server in the mid aught using it over a network share was the only way to actually use it. And it really wasn't good. I don't know if that made it better, I assume not much, VSS was really trash.
• nikodunk 2y ago
  What a beautiful, humorously written, bitter-sweet piece of writing!
• agentultra 2y ago
  The ending is pure gold. Some of the best times in my career were working on a codebase for an application serving folks I knew on a first name basis and had had lunch with. I could talk through pain points they were having, we’d come up with a solution together, I’d hack up a quick prototype and launch it just to them to try out. We’d tweak it over a couple of weeks and when it was good I’d launch it to all customers. Messy, ugly code base. But it worked well because it wasn’t over-managed. Just developers doing development. Amazing what happens when you get out of the way of smart, talented people and let them do their work.
  • rochak 2y ago
    [Deleted]
  • failbuffer 2y ago
    ... and get them talking directly to the user. I feel that's where the real magic happens.
    • Aeolun 2y ago
      So many “Oh, I just didn’t ask because I thought it would take months” saved!
    • rblatz 2y ago
      When you actually understand the problem you are solving, and the users you solve it for you start to care about a solution and not just the technology.
    • switch007 2y ago
      Surely it is much more efficient for a PM to ask all the wrong questions and relay the answers using totally different words to the developers. As many many companies love hiring tons of PMs, this is surely the optimal system
  • kuon 2y ago
    The key here is to put the devs and users together.
• deathanatos 2y ago
  Two early databases I worked on. The first contained monetary values. These were split over two columns, a decimal column holding the magnitude of the value, and a string column, containing an ISO currency code. Sounds good so far, right? Well, I learned much later (after, of course, having relied on the data) that the currency code column had only been added after expanding into Europe … but not before expanding into Canada. So when it had been added, there had been mixed USD/CAD values, but no currency code column to distinguish them. But when the column was added, they just defaulted it all to USD. So and USD value could be CAD — you "just" needed to parse the address column to find out. Another one was a pair of Postgres DBs. To provide "redundancy" in case of an outage, there were two such databases. But no sort of Postgres replication strategy was used between them, rather, IIRC, the client did the replication. There was no formal specification of the consensus logic — if it could even be said to have such logic; I think it was just "try both, hope for the best". Effectively, this is a rather poorly described multi-master setup. They'd noticed some of the values hadn't replicated properly, and wanted to know how bad it was; could I find places where the databases disagreed? I didn't know the term "split brain" at the time (that would have helped!), but that's what this setup was in. What made pairing data worse is that, while any column containing text was a varchar, IIRC the character set of the database was just "latin1". The client ran on Windows, and it was just shipping the values from the Windows API "A" functions directly to the database. So Windows has two sets of APIs for like … everything with a string, an "A" version, and a "W" version. "W" is supposed to be Unicode¹, but "A" is "the computer's locale", which is nearly never latin1. Worse, the company had some usage on machines that were set to like, the Russian locale is, or the Greek locale. So every string value in the database was, effectively, in a different character set, and nowhere was it specified which. The assumption is the same bytes would always get shipped back to the same client, or something? It wasn't always the case, and if you opened a client and poked around enough, you'd find mojibake easily enough. Now remember we're trying to find mismatched/unreplicated rows? Some rows were mismatched in character encoding only: the values on the two DBs were technically the same, just encoded differently. (Their machines' Python setup was also broken, because Python was ridiculously out of date. I'm talking 2.x where the x was too old, this was before the problems of Python 3 were relevant. Everything in the company was C++, so this didn't matter much to the older hands there, but … god a working Python would have made working with character set issues so much easier.) ¹IIRC, it's best described as "nearly UTF-16"
• throwaway93982 2y ago
  When it comes to building things, the outcome is dictated by the constraints. But not the way most people realize. Constraints affect outcomes in (at least) three ways: - by absolute limitation (you cannot violate these) - by path of least resistance (the constraint makes X easier) - by strategy (creative use of knowledge & skills leads to novel solutions) --- There is an injured person on top of a mountain, and they need medical attention. You need to go up the mountain, get them, and bring them down. You only have so much strength/endurance, so nothing you do will ever be faster than what your body is capable of. You need to get up and down in less than two hours. You need to carry an injured person. The mountain has two slopes: a long gradual one, and a short steep one. Most people would climb the long gradual slope, because it's the path of least resistance. But it will take 4x as long to climb up and down it. Climbing straight up the steep slope would be incredibly tiring, and unsafe to bring someone down. You can, however, zig-zag up and down the steep hill. It will take more time than going straight up, but faster than the long way, you will be less tired, and it's safer to bring someone down hill. --- Constraints can be good and bad. Good use of constraints can allow you to get something done effectively. Bad use of constraints leads to failure. So it's important to have someone who can utilize those constraints effectively. Vision, leadership, and wisdom is more important than the people, skills, materials, or time involved. The former determines the quality of the outcome more than the latter.
  • crngefest 2y ago
    So to summarise: „It depends“
    • throwaway93982 2y ago
      Well, no... the summary is, you have to think hard in order to use your constraints as effectively as possible or the outcome will suck.
      • crngefest 2y ago
        „You have to think hard“ is just „it depends“ worded differently. If you don’t have to think hard then it also doesn’t depend on the details/circumstances and the outcome will be fine regardless of constraints.
• RadiozRadioz 2y ago
  I built a system as horrible as this. One of my many terrible inventions was as follows: Originally our company only did business in one marketplace (the UK). When we started doing business in multiple marketplaces, it came time to modify the system to cope with more than one. Our system assumed, _everywhere_, that there was only ever one marketplace. I had a plan: I would copy-paste the system, make an "international" version that supported many marketplaces, then transition over our original marketplace to the international version and shut down the old system. This way, everyone could keep working on the original marketplace like normal, they'd get the new marketplaces on the new system, and we'd do a clean cutover once ready. It started out quite well. I got the international version working and onboarded our first new marketplace. The business was very happy, there was lots of opportunity in the new marketplaces. They asked that I delay the cutover from the old system and focus on developing things for these new marketplaces. After all, the old system still works for our original marketplace, we can move it over once our work is done on the new marketplaces. I said yes, of course! It's now 5 years later, it turns out there's a lot to do in those other marketplaces. To say that the system is split-brained is an understatement. When training new hires, we teach them the difference between a "product" and an "international product". When either system does something, it double checks and cross-references with the other system via a colourful assortment of HTTP APIs. There are a variety of database tables that we "froze" in the old system and continued in the new system, so you could tell that an ID referred to something in the new system or the old system if it was above or below the magic number we froze it at. We have overarching BI dashboards that query both systems to produce results, and they have to heavily manipulate the old system's data to fit with the new multi-marketplace model, and any other changes we made. Both systems are extremely tightly coupled, the old one is a ball and chain to the new one, but the new one is so tightly integrated into it that there's no hope of separating them now. I've learned to say no since then.
  • noisy_boy 2y ago
    Time to create a third "Global" system.
    • tommica 2y ago
      "We got this customer on Mars, can we make a "Planetary" variant of the product?"
• collinmanderson 2y ago
  I’m seeing a lot of comments about terrible code bases, but I do think there’s something really beautiful here, something like “worse is better”: > This may sound like a mess to you. But it was remarkably enjoyable to work in. Gone were the concerns of code duplication. Gone were the concerns of consistency. Gone were the concerns of extensibility. Code was written to serve a use, to touch as little of the area around it as possible, and to be easily replaceable. Our code was decoupled, because coupling it was simply harder.
• MBCook 2y ago
  This reminds me of my first job at a very small shop. Here’s two stories: The calendar table struck a chord. We had one for holidays. One system needed to know when they were to calculate pay dates. Except once a year it would “run out” and someone would have to go add in the next year’s worth after a bad calculation was spotted. The second time I was told to do it, I put in 10 years worth. The company didn’t survive long enough to need more. My first “big” project was actually that pay date code. Every once in a while the server would hang, and people had deduced the date calculation was the problem. But no one knew why. And it wasn’t frequent enough to be worth taking time from the other two programmers. But I was new and thus “spare”. After not being able to find any errors, I brute forced it. I ran that calculation for every day of the year for every pay type (weekly, monthly, bi-monthly, etc) and it quickly got locked in an infinite loop. Armed with the “bad” data that triggered it, it was easy to solve and provide tests to prove it would never happen again. I don’t remember what it was, exactly.
• thayne 2y ago
  I've seen something very similar to the Sequence key table. But there wasn't just one of them, it was a common pattern. And it had 2 rows and 2 columns. The reason we had it is we had master-master replication, but without requiring the peer to acknowledge befor committing a transaction. To avoid inconsistencies, we preferred one server for even ids and the other for odd ids. But when writing a new record, an autogenerating sequence would just give the next id without regard to what "side" the request was on. So we had a table to keep track of the next id to use for each side, where we incremented the next id by 2 each time a new id was allocated. It was a little weird, but it worked fairly well. Although we eventually changed the architecture and removed the need for those tables.
• codeptualize 2y ago
  What an amazing read, funny, nostalgic, the duality of the perfect mess but still so much opportunity and ability to make progress, and somehow it all just chugging along. I feel a lot of this is the difference between theory and practice. Sure each of these things are bad, but probably a lot might have been the right choice at the time, and in a way, most companies, even most projects running for many years, end with similar quirky processes, messes, and hacks. It's the war stories of what used to be, often only told by the code and the database as the creators have long left. When I look at our couple year old startup, we have some of these things. Prototypes that just keep functioning and only get more attention when they break, manual processed that sort of make sense but also don't, integrations with systems that were build in the early 2000's (it works, but it ain't pretty). We fix many, but I'm sure some will survive way too long. As software engineers, especially online, we like to discuss the ideal way to do things, best practices, testing strategies, redundancies, the whole nine yards. When time is of the essence and stakes are high, it all goes out of the window and you gotta just make it work in whatever way is possible to fight another day.
  • ptrik 2y ago
    This is the main takeaway for me. The decentralized way of software development in a large scale. It does echoes with microservices a lot, but this can be done with a more traditional stack as well. It's ultimately about how you empower teams to develop features in parallel, and only coordinate when patterns emerge.
• compiler-guy 2y ago
  It's hard for those who came into the discipline in the past twenty years to realize just how much things have changed around version control and building. Joel Spolsky released the "Joel Test" for determining if the software team you were interviewing had good practices in 2000. One of the requirements was that they used version control. Not all that many teams actually did. Especially during the dotcom craze. Today, passing the Joel Test is table stakes, and it's the rare shop that doesn't. But it took years for that sort of thing to become ubiquitous. https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
  • khalilravanna 2y ago
    This reminds me of my first software job, an internship in college in 2012 building an application from scratch to facilitate evaluating teachers. It was basically an app to allow people to create a form for their school’s evaluation criteria and then submit those forms. Sounds super straightforward, right? It was. The catch was our team was 2 CS undergrad, a masters CS student, and a high school student. All with no professional experience. We knew nothing. Well kind of but more on that in a second. Our manager was absolutely non technical. In fact they were the second highest person in the company (fairly small company) and were managing our project and a bunch of other stuff at the company. And somehow with almost 0 oversight we built a functional Django application that the business was able to sell and make money from. My favorite highlights were 1) the codebase was initially shared over FTP (“Hey you’re not editing file X, right? Oh you are? Ah woops I just overwrote all your changes.”) till someone intelligently suggested “Uhhh Git?” 2) the actual best programmer amongst us was the high schooler. They suggested Django, picked the DB, they suggested using Celery to speed up async work, Redis for caching, and yes, “Uhh Git?” In retrospect the only reason we succeeded was because of them. They were like top 5 on the stack overflow Code Golf site IIRC. 3) My interview was basically showing my aforementioned manager who had never coded in his life a project I worked on at school and him being like “Yeah looks good. You’re hired.” With 10 years of hindsight, I cringe thinking back to all the bad decisions I pushed for and the no-doubt terrible code I wrote. But I also marvel and look back fondly at being given a shot and being in an environment where I could just build something from the ground up and learn everything soup to nuts on the job. God bless whoever inherited that codebase.
• grishka 2y ago
  The worst codebase I've ever worked on was the Telegram client for Android. I mean just look at this file, it's so huge GitHub gives up rendering it: https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/src/main/java/org/telegram/ui/ChatActivity.java https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/... Or this one, it implements EVERYTHING for rendering and interacting with a message in a single class, all non-service message types, all drawn manually on a canvas "for performance", and all input processed manually too: https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/src/main/java/org/telegram/ui/Cells/ChatMessageCell.java https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/... I even had the permission from the main developer to refactor some of it, but I never got around to actually doing it. It's a miracle this app works at all.
  • bartekpacia 2y ago
    It’s even more crazy because Telegram is by far the smoothest messenger app on Android. BTW, would you mind sharing how you got that job?
    • Klonoar 2y ago
      They didn't specify it was their job, the project is open source after all.
    • grishka 2y ago
      > BTW, would you mind sharing how you got that job? I started working with Pavel Durov in 2011 on the VKontakte Android app, after winning several of his developer contests, and it all just went from there. I was the last one of the "old" team to transfer to Telegram. My main job was actually the libtgvoip[1] library and its integration into the Android app (the other platforms were done by their corresponding devs). I got fired in 2019 because the calls weren't working satisfactorily for Pavel. He got furious when he was unable to call some journalist, cursed at me in front of the entire team and sent me a termination notice. In hindsight, I should've definitely not taken building a VoIP library from scratch alone upon myself. It's a massive undertaking. But at least I learned a lot, and many people say that my implementation performed noticeably better than the current WebRTC-based one ¯\_(ツ)_/¯ I'm now working on two fediverse projects at the same time — Smithereen[2], a VK-like ActivityPub server, and the Mastodon Android app[3]. [1] https://github.com/grishka/libtgvoip https://github.com/grishka/libtgvoip [2] https://github.com/grishka/Smithereen https://github.com/grishka/Smithereen [3] https://github.com/mastodon/mastodon-android https://github.com/mastodon/mastodon-android
• xpressvideoz 2y ago
  Elasticsearch and OpenSearch have a similar issue. They have a soft limit on the number of "searchable" fields (called "mapping") a table can have (by the way, a table is called an "index" in their terminology. How confusing!), which is 1000. We ran into this problem because we tried to jam all the logs in every single microservice into a single table. Each microservice had a different log format, so after combining them all the number of different fields surged. We did it this way because the operations team wanted to maintain a single table, by the name of simplicity, which to this day is a reason I can't completely fathom. We were surprised because we thought Elasticsearch and OpenSearch were some kind of magic box that can somehow ingest all the data we put yet still are perfectly performant. After that incident, we introduced a common log format that applies to every microservice.
  • deleted 2y ago
    [deleted]
• deleted 2y ago
  [deleted]
• ddgflorida 2y ago
  In one of my early career jobs, we had applications that were so buggy we had 1 or 2 developers dedicated to logging into customers databases and fixing data.
• peteforde 2y ago
  Man, this post is gold. I think we all have our own version of the best worst codebase. My take on this story was a backend "deal capture" system for an emissions trading desk about 25 years ago. The application was written in classic ASP, T-SQL and VB6 COM components. Oh, and VBScript. It only ran in Internet Explorer 6. The heart of the machine was a massive, unholy stored procedure. It was about 35kb, and what it did was generate a very long string which was - you guessed it - a gigantic SQL statement with countless JOINs across many temporary tables. The business logic was deemed sophisticated enough that it was decided we needed a workflow engine and some genius decided that we should use an expensive proprietary enterprise-y tool which used Microsoft Exchange Server as its datastore and operating environment. This was as terrible an idea as it sounds, because the "API" to talk to this engine was single-threaded and massively bottlenecked by whatever made Exchange Server suck. Most significantly, it also could only be properly accessed by a very specific version of a DLL, requiring that a server setup follow an agonizingly specific software installation procedure that was scientifically proven to be impossible for IT to execute because they would always jump to the end and install the latest version of the service packs and suddenly the DLL we needed would be impossible to access without wiping the machine and starting over. This is what people meant when you hear the term "DLL Hell". The most interesting aspect of this system was the client UX. This was 1999, and Ajax was literally not a thing yet. However, the precursor to XmlHttpRequest was an IE-only thing called ActiveXObject. You could use VBScript to wire up form elements to be updated by an ActiveXObject in the way we'd consider familiar today, except that instead of FORM GET/POST, your ActiveXObject would talk to a VB6 COM+ DLL running inside of Microsoft Transaction Server. Looking back on it now, the whole thing was simultaneously an abomination and years ahead of its time. The security profile would likely give a modern developer night terrors; I'm pretty sure that if you were on a page that was being served by an IIS instance that had COM+ registered, your ActiveXObject could call methods on that object so long as the VBScript had the UUID that identified the COM+ object. End of story. Just wow. Final detail that my memory retains is that the room full of chain smoking women who operated this system did not use mice. That is, they were entering data almost exclusively through the number pad and furiously slamming the tab key to move to the next field. They would develop intense muscle memory for the 200-300 fields on this form. They needed repeatability or it would literally break them. Thing is, there were many instances where they didn't want to fill fields in the order they are defined in the HTML, so over time about 90% of those form elements gained in-line event handlers, again, written in VBScript, which replaced the built-in tab key handling functionality of the element with whatever was deemed the correct next step in their insanely specific process. If we accidentally broke their expectation, there would be hell to pay. That is, we would login to the live server with VNC and live edit the changes into the codebase while they were on the phone over the remote screen share connection. There was no source control, and if there were backups, I'm dubious that they would have been restorable.
• prudentpomelo 2y ago
  It's honestly relieving to read all these stories because I feel like I am in the middle of this right now. Our legacy product is a mishmash of perl and php that was probably started in the early 2000s. Which I wouldn't feel bad about supporting if it wasn't for the new system that has been bolted on top of it. The previous team that started the migration is all gone and now we are halfway between two two systems that we have to support. On top of that my manager is afraid to say no to our eccentric CEO so we are flooded with new feature requests but can never get anything done because we spend so much time chasing down problems in the old system. Is there any sane way out of this? I just feel like I am white-knuckling it through every day.
  • hiddew 2y ago
    Get you, your manager and the CEO in one room, and tell them the facts. Once those are on the table, discuss solutions. Otherwise nobody wins.
• dave7 2y ago
  For online Poker, one of the current database tools runs in to this issue - PokerTracker 4 (PT4). These tracker databases are usually used to generate a HUD - numerical readouts surrounding each player shown on the table. PT allows for custom HUDs to be saved/exported/shared, and there is a cottage industry building and selling such HUDS. These HUDs can often bundle hundreds of custom stats - basically a subset of SQL queries, a simple example would be "times_raised / opportunities_to_raise WHERE position= 'button'", that sort of thing. For performance reasons these custom stats are cached, which obviously makes some sense. However, each cached stat creates a new column in the custom_cache table of PT4's current database, a PostgreSQL 9.0 backend. If you play mostly Heads-Up SNG or the 3-handed Spins, it's actually quite easy to go over the (IIRC) 4096 column limit there by purchasing and importing one too many fancy HU HUD packages! This completely borks PT4, it can no longer open - and players can no longer make money! In my previous work, I've supported many a player and fixed this "too many columns" error numerous times (by deleting enough custom HUDs from their account until the PT4 can start up correctly once more). Funny to see this pop up elsewhere!
• sanktanglia 2y ago
  Im currently deep in rewriting a c# monolith thats 10+ years old that has thousands of lines of extra code that i was able to throw away because most of it was written before there were optional arguments so they made overloads for every permutation of arguments for every framework function
  • noisy_boy 2y ago
    Boring but satisfying.
• gouggoug 2y ago
  I worked many years with an open source eCommerce platform called Magento[0] which, at the time, used something called the "Entity Attribute Value" (or EAV) system[1]. One particularity of the EAV system is that you end up having tables with hundreds (and growing) of columns. It made Magento itself extremely hard to work with and optimize. I hope they moved away from this model since. To be fair, this was before nosql databases were a thing. [0]: https://en.wikipedia.org/wiki/Magento https://en.wikipedia.org/wiki/Magento [1]: https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...
• jabart 2y ago
  This codebase sold for $4.3 billion. That table was cursed.
• alexpotato 2y ago
  And weird code can lead to crazy outages. I put together a thread of all of the wild incidents I've seen over my many years of working as a FinTech SRE: https://x.com/alexpotato/status/1215876962809339904 https://x.com/alexpotato/status/1215876962809339904
• cmpalmer52 2y ago
  I once had a project to turn a customer’s Excel VBA application into a real application (I used ASP.Net). He had been hacking on this Excel spreadsheet for like 15 years. Once I printed the VBA code because it was hard to navigate and it was like 250+ pages printed out rather compactly. The worse part wasn’t the code itself (although it was bad), but the fact that there was so much abandoned paths in it and there would be three or more different versions of crucial functions with the same name (or sometimes a different name but doing the same thing) in different places and sometimes all being called by different paths in the workflow. Or not being called at all. And it was very math heavy (calculating natural gas pressures and flow rates through different sized pipes and fittings and sizing regulators to meet certain loads). Think Excel formulas on cells that referenced 15-20 other cells, each of which was a formula on their own that referenced other pages and cells, some of which were filled by VBA. And that wasn’t even involving the VBA code full of brute force solvers for multi-variable equations that used heuristics he’d worked out by trial and error (if it’s a delta over 1.5, add 5 to this variable, otherwise subtract 3, but if the delta was less than 0.5, add 1 and so on - it eventually converged or found no solution, but a binary solved did the same thing, only faster and easier). It took me and a junior developer several months, during which, of course, multiple change requests were going through to make it multiuser and secure. Both my nightmare project and one that I’m very proud of once it was completed.
  • jahewson 2y ago
    By the end of the first sentence I thought “that sounds like engineering” (the real kind). I know people still dealing with this kind of thing!
  • IIsi50MHz 2y ago
    That's more adventurous than all the Excel projects I've been given. Many of the most satisfying solutions are the most simple that nobody's implemented yet. Like, at a job where all the lead devs and, apparently, the whole internet agreed that you can't do proper source control for Excel because you can't export-and-import code for all modules (including sheets & ThisWorkbook) without copy-paste and because a running module can't replace itself. The solution ending up so simple, that I was embarrassed to hear it called "a stroke of genius". I still have that code somewhere.
• odyssey7 2y ago
  It’s a cute story, it made me laugh. If this story reminds you of the codebase you work on, don’t let its romanticism deceive you into thinking that suffering is somehow actually good.
• 20after4 2y ago
  When I was still in college studying Computer Science (~2001) I got a part time job at a Medical Claims clearing house doing VB and Java programming. My task was to re-write in Java the ancient and archaic system which would read in medical claims that were uploaded by providers and then translate them into the one of several formats, depending on which insurance provider we were sending them to. The inputs were big batches of data in one of two formats, "ASC X12"¹ or "NSF"² and the outputs were some ideocyncratic version of those two, or a 3rd option which was just ascii text, layed out (monospace) such that it would line up when you dump the file directly to a printer loaded with the pre-printed UB-92³ forms. None of this is particularly interesting, unless you have a fascination with archaic file formats and/or an interest in historical and highly idiosyncratic government bureaucracy. The really interesting (horrifying) thing about the job, though, was the state of the VB6 code which I was asked to translate into well structured and performant Java. There were some really hilarious and nightmare inducing subroutines like "BunchOfIfs" and "BigSelect", each of these monsters were several thousand lines long and consisted of exactly what you'd expect. Huge branching structures with absolutely no reasonable organization or design. I'm not even exaggerating to say it was just the accumulated cruft of 10 years of dirty hacks tacked on by the cheapest coders they could find where the only standards were if it works it ships. Literally the worst procedural code you can imagine with zero factorization, modularization, plenty of duplicated code copied and pasted then modified to do something slightly different than the 3 other versions of similar code elsewhere in the project. Somehow, after a year of part-time work (20 hours a week, between classes) I managed to produce a working system to translate claims from any one of the weird formats into any other one of the weird formats, including 5 or 6 variants of said formats, each one which violated the spec in a unique way in order to satisfy the strange requirements of some particular insurance company. The Java version was less than 10% the size (lines of code) of the old system, ran 30x faster and produced correct output. Still to this day it's objectively the most difficult, painstaking, excruciating, but also probably the best, most impressive work I've done. And it's the least I've ever been paid for writing code. Oh and I forgot to mention, nothing was in source control and there were several variants of the codebase that had been modified slightly to do a similar but different task and then just continued to drift away from the codebase it was copied from. 1. https://en.wikipedia.org/wiki/ASC_X12 https://en.wikipedia.org/wiki/ASC_X12 2. https://manuals.momed.com/edb_pdf/NSF%20(National%20Standard%20Format)%20User%20Manual.pdf https://manuals.momed.com/edb_pdf/NSF%20(National%20Standard... 3. https://owcprx.dol.gov/static/UB-92.pdf https://owcprx.dol.gov/static/UB-92.pdf
• jiggawatts 2y ago
  In case anyone is curious about what the proper solution to some of these problems is in modern SQL Server: 1) Many columns can happen with per-customer customisations to a shared table. The common way is to have a "customcolumns" table with "ID,ColumnName,ColumnValue" linked to the base table that has an "ID" key, but SQL Server also supports this natively now with Sparse Columns: https://learn.microsoft.com/en-us/sql/relational-databases/tables/use-sparse-columns?view=sql-server-ver16 https://learn.microsoft.com/en-us/sql/relational-databases/t... 2) Shared identity or globally sequential numbers have a built-in schema object type now: https://learn.microsoft.com/en-us/sql/t-sql/statements/create-sequence-transact-sql?view=sql-server-ver16 https://learn.microsoft.com/en-us/sql/t-sql/statements/creat... 3) Manually populated calendar tables are actually a common schema design pattern, especially in manufacturing, shipping, and OLAP reporting systems. This is not that bizarre, it's just a bit weird that it would break logins! These tables can let you define all sorts of things such as international holidays, manufacturing calendars, tax years, finance reporting schedules, etc... 4) Dropping and recreating tables is also common, albeit usually done in a transaction. The fancy way to do this is with partition switching, where a new table is populated from whatever (external data, business logic, etc...) and then instantly swapped for the original without any long-running operations like truncate & insert would. See: https://pragmaticworks.com/blog/table-partitioning-in-sql-server-partition-switching https://pragmaticworks.com/blog/table-partitioning-in-sql-se... 5) Delayed reporting replicas ("here was a copy of the database. Data in this copy was about 10 minutes out of date.") is also a common pattern. At this point, the blog author is just complaining about the realities of business databases. Typically you'd have a bunch of read only replicas with different delays: Synchronous for HA failover, Asynchronous for DR failover and real-time reporting, and deliberately delayed on a schedule ETL copies for "point in time" consistent reporting. These would typically be done at 3am or something to minimise inconsistencies in the data. The modern way to do this is a SQL Server Always On readable secondary: https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/active-secondaries-readable-secondary-replicas-always-on-availability-groups?view=sql-server-ver16 https://learn.microsoft.com/en-us/sql/database-engine/availa... 6) "The main codebase I worked in was half VB, half C#." this is also surprisingly common. It's often not worth rewriting old code, there's not enough return on the invested money. These days there are automated conversion tools for VB to C#, such as: https://marketplace.visualstudio.com/items?itemName=SharpDevelopTeam.CodeConverter https://marketplace.visualstudio.com/items?itemName=SharpDev... 7) The Gilfoyle character is honestly the only thing that stands out here as an actual problem, not just a typical thing that happens in large enterprise bespoke software development.
• vstollen 2y ago
  This made me think of my first job. I was the sole developer on a project because the old developer left. Nothing was documented and nobody knew why things were designed the way they were. We had no code reviews, no design docs, no tests, nothing. We made the changes the way we thought they were right and would git pull them onto the production server. After I struggled to get productive for the first four months, my manager went on a four-week Christmas vacation. In a moment of frustration, I seized the opportunity and rewrote the whole project from scratch. I don’t remember if my manager ever noticed, but that was the moment I finally got productive.
  • sebastiennight 2y ago
    > my manager went on a four-week Christmas vacation. Is that kind of stuff common? People checking out on Black Friday and coming back for New Year's?
    • louwrentius 2y ago
      I bet that’s Europe, it’s not uncommon
      • vstollen 2y ago
        Yess, we were the software team for a university library. There wasn't a lot of pressure and people on my team generally had a chill and comfortable life. However, there wasn't a lot of room for ambition.
    • eddd-ddde 2y ago
      As we all should. Life is too short to be surprised about not working for a month.
    • folmar 2y ago
      In Europe totally, and for example in Germany it's customary to get you six week vacation in the summer. This also has the benefit that the workplace has to have real back-up person for all matters, as six weeks is too long to shove everything under the carpet waiting for your return.
• EFruit 2y ago
  Oh, those kinds of columns. I thought we were talking text columns, and I was about to relate. I work at a small business. Despite computer software being about the literal opposite of our business (plants), the founder built an entire suite of interconnected tools that runs off MS BASIC for Xenix, on a single HP machine running SCO OpenServer. The machine has so many customizations, self-scheduling cron/at jobs, odd nooks for files, weird tweaked programs, and special conventions that if a server with a dedicated hostname qualifies as a pet (as opposed to cattle), I'd be THIS THING'S pet. The system handled EVERYTHING. Accounting, payroll, pesticide management, inventory, attendance, business contacts, shipping label printing... all out of a bunch of terminal menus (which are actually text files with control codes that get `cat`ed out). But by God, the most horrifying part of it all are those BASIC files. They're IMPENETRABLE. Firstly, I don't believe this version of BASIC supports named functions or subroutines. At all. But that's fine. MS BASIC being what it is, the interpreter only can deal with a certain number of characters per logical line, and that includes data definitions. This version of BASIC (like so many others) includes its own serialization format and record/file access scheme. You declare the layout of the data file you want, open that file, and BASIC will handle (most of) the rest. So when the founder started hitting the internal line limit while defining the data file's fields, he would cut the names of the fields down to fit more on that one line. Over time `30 AS EMPLOYEENAME` became `30ASEMPLNAME`, which became `30ASEMNAME` which became `30ASAF(1)`. Every cent we transact, and every employee's timecards still flow through this old system, some even using bona fide Wyse terminals. To reiterate, this man was, first and foremost, a farmer. His contraption is terrifying, but commands immense respect. It's lasted 30-some years with continuous tweaking and refining, and we still have yet to replicate even half of its functionality. (Though there are other organizational issues that are making that difficult.) On a personal note, aside from the calcified codebase and occasional spelling errors, it's a stellar business application. It's fast, mostly coherent, and keyboard-driven in such a way that experienced employees can navigate it faster than the terminal can refresh. We've been working for years to replace it, but at the same time, there's a lot our newfangled Angular+PHP+MySQL replacement could learn from it.
  • 392 2y ago
    Why a webpage instead of another TUI?
• curiouscavalier 2y ago
  This reminds me of working with a company that provided market-wide pricing data for a particular commodity across the US. They were the de facto aggregator of pricing for the sector and at the time I worked for one of their larger customers. We requested they add another vendor’s pricing in a particular region and received a response along the lines of “Sure, as soon as we can figure out how to add another entry. We’re at the maximum row count on the system.” Needless to say it gave my team a few days of flabbergast and speculation on what system they must have built on to hit a row limit at only 5 digits. And yet for a non-tech industry it was mostly working.
• ChilledTonic 2y ago
  Man, I remember my Gilfoyle. We had an employee who would do one off programs for anyone with a use case - but ours wiped his computer before giving it back, so frequently we'd get tickets for software we'd never heard of that was somehow mission critical, and we'd have a few days to spin up a new version on the spot. Probably some of the most fun I've ever had writing software was making Gilfoyle-2s.
• recursive 2y ago
  In SQL Server, you're likely to run into the 8000 byte row size limit before the column count limit. Ask me how I know.
• 100pctremote 2y ago
  "Merchants2 was the solution."
• MikePlacid 2y ago
  Oh, the shipping! Now, from the customer’s point of view. My youngest worked in a furniture chain over the summer. And they got sent a big, heavy furniture set from the central warehouse, which the store actually didn't want. So, they sent it back. The problem was that the system didn't allow them to say: please, don't send this again. And the non-natural intelligence at the central base decided to send this set again. When my youngest started working - they were loading this set back for the seventh time. Why 'loading'? Because no one could find a way in the program to send this ill-fated set back on the same truck that brought it. No one, except my youngest, that is. He managed to find the combination of keys and checkboxes that allowed them not to unload the unwanted set and ship it back on the same truck - and he immediately got a raise. I suspect the set is still traveling. But now they only load-unload it at the central warehouse.
• summerlight 2y ago
  My favorite, obligatory reference when we discuss horror of legacy codebase: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
• nnurmanov 2y ago
  I know about the limits on max column number per table. The other day I was thinking about the best table layout for AirTable like systems, I finally decided to use relational tables with json fields. E.g. row is a several fields (id, name, who fields) and one json field where I can put any number of fields. Hopefully, this is going to be the best from both worlds and work good from performance view.
• bubblebeard 2y ago
  This reminds me of my first internship. The company I worked at backed everything up on CD:s. I was tasked with writing a new service for their intranet, indexing the content of all CD:s, so staff members could lookup which CD contained a certain file. I wrote a little program to scan the CD:s as I inserted them into my computer, indexing the data into a database I had created, and then labelling the CD. It wasn’t exactly exciting work but I still miss those days sometimes, everything was new and unexplored.
• Nurbek-F 2y ago
  Databases must not be a place to store any logic
  • noisy_boy 2y ago
    Tell that to millions of lines of stored procedure code across the globe doing heavy lifting on database side. To be clear, I really do not like stored procedures, but they do have their place.
• greenthrow 2y ago
  This is obviously a work of fiction. Not because there isn't bad code similae to this out there, but because the way it is told and many of the details don't add up and have all the hallmarks of fabrication.
• philjohn 2y ago
  I can second the sequence key table being because auto increment wasn't available on some databases. I ran into the same some years back at a company who's software dated back to the late 60's.
• nicopappl 2y ago
  Wow, this is exactly how I felt with regard to my first job as well. This old codebase no one wants to touch but works somehow. The quite nice later-on additions. Even the "build a pipe-separated string using reflection and a 150 classes hierarchy" rings something. The old codebase was an hairy ball of scala using a very outdated version of a (now) infamous actor framework. Before they figured out that untyped messages kinda left out one of the major selling point of Scala. The code was readable, but the authors had this strange idea that every little piece of logic should be part of its own "Actor". An actor is pretty much equivalent to a class, and each one of them had their own little file. With this many classes, with very specialized purposes, you ended up with 90 character identifier names. To understand what would be a single function in a normal code base, you would have to dive through half a dozen files through several repositories to piece together the logic. Generally, at the end, you find that most of the code is about passing around a value, and there is this one file where there is actual logic applied to the value. It wasn't that awful. The thing was that it was impossible to change anything: no documentation, no test, no bug tracking, not even any PR process, laconic commit messages, no Wiki pages. And the actor framework made it very difficult to add tests. But later developers did manage it pretty well, they drew fences around the old services, with an HTTP API to communicate with it. And all later additions were part of different services that were very cleanly and consistently designed.
  • senorrib 2y ago
    It’s refreshing to hear that last paragraph. Honestly, sounds like the difference between hobby programming and professional engineering.
  • IIsi50MHz 2y ago
    > To understand what would be a single function in a normal code base, you would have to dive through half a dozen files through several repositories to piece together the logic. I've experienced something like this, where the other devs preferred to breakecode down into the smallest units possible. I often saw a half-page procedure changed into multiple function calls, each with their own boilerplate and error handling (as required by the style-guide), such that you could not view the entire logic on one screen. Every procedure name described the intent, they said, so you know what the parent did without having to worry about details like how it did it. I, meanwhile, trying to track down what the actual implementation was, would hold as many subfunctions as I could in windows with a tiny font, and hold the rest in my head…just to find out stuff like: "Oh, so it's trying to use system feature foo.x, but I know foo.x is broken current release for a certain edgecase. Which case happens to be what our customer does all the time…"
• trustno2 2y ago
  You ship your org chart. If you have a messy app, you have a messy organisation.
• Snacklive 2y ago
  This hits close to home. At my current job we have a similar experience, We are building an Android app and the codebase is probably the same age as me (25yo). It started as a Windows Phone app that was later ported over to Android, you can easily find files and segments of the codebase that were auto generated by some code converter from C# to Java. In the codebase itself you can see the evolution of code styles, older views and the core of the system is written in a very old and very stateful manner while newer parts of the application use modern architecture practices, we have two local databases that load configuration into the app for our different clients and their requirements, an global state is loaded at all times to check what is the correct business logic to follow. Im a junior, few years into Android Programming and while sometimes its frustrating having to deal with some nasty bug because a random variable is updated for reasons only god knows, i think the experience its giving me its something im going to appreciate years down the road.
  • kelsey98765431 2y ago
    > Windows Phone - Wikipedia > 1 week ago - It was first launched in October 2010 with Windows Phone 7. 25 years ago phones still had cords. Infact, C# itself is only just barely 19 or 20 years old at best. I know I felt like everything that came before me was ancient when I was young, but a decade is a long time in tech so i just felt the need to point out that if you had a codebase for a phone from 25 years ago if it was not written directly in assembly, it would have been running a very slim java version called Java ME (Micro Edition) for feature phones which was all the rage before android, or if you were unlucky you would be dealing with BREW which was a SDK for c/cpp development on feature handsets. https://en.wikipedia.org/wiki/Java_Platform,_Micro_Edition https://en.wikipedia.org/wiki/Java_Platform,_Micro_Edition https://en.wikipedia.org/wiki/Binary_Runtime_Environment_for_Wireless https://en.wikipedia.org/wiki/Binary_Runtime_Environment_for... C# Would have been beyond a luxury for a mobile application 25 years ago.
• EGreg 2y ago
  There were a couple times I was convinced that basic methods were as good or better than highfalutin ones, or got me thinking about solutions that hew closer to the basic method. In 2014 or so, someone at a company told me that they just don’t do branches, and merge everything into trunk. I agree that long-lived branches are bad, but no branches at all? Sure enough, this was OK — meaning branches could be kept and synced by a few people outside the main repo, but forks are never pushed to the repo. Using table columns for data, as long as you’re building for an actual specific business vertical and not just a general-purpose framework. You never know when you’ll want to put an index on the data, and databases already have built-in mechanisms to select a subset of the fields, etc. You avoid all kinds of joins. If you need to extend the table, just start a new table with the same synthetic ID. I would say, in tables where you don’t want to even have global locks (eg sharded tables), just don’t have an autoincrementing ID. Instead, try a random ID or a UUID, and insert it, then use it across tables. In short, what was described here is actually good design.
• EGreg 2y ago
  I just wrote an answer regarding data denormalization: https://stackoverflow.com/a/78831591/467460 https://stackoverflow.com/a/78831591/467460 It is from the point of view of an app developer, not a DBA, but should be relevant to most people on HN.
• mystified5016 2y ago
  In my very first programming job, I got hired on right at the end of a major refactor that had been going on for the better part of a year. I was a hacker kid who started programming less than a year prior. My first task at this company was to merge and resolve all conflicts from the refactored branch back into mainline. No one checked what I did. As long as it compiled on my machine and passed the CI test (just one), it got merged and released into production. Shockingly, this project was notorious for regressions. We were on a weekly(!) update cycle, and we constantly had bugs reappear that had been solved months prior. This was 2010 or so, and we were using SVN because the project lead didn't trust or understand git. He also didn't understand SVN. The solution to our constant regressions was pretty simple. Instead of merging branches into trunk, we would delete the master branch every week and create a new one out of all of the finished develop branches. Surprising absolutely nobody apart from that project manager, this plan was a spectacular failure. He also stole code out of my personal git repos from before I worked there, and bragged about how smart he was for stealing my code. So, y'know, just a general idiot and asshat.
• wyclif 2y ago
  Jimmy Miller the drummer and record producer? Anyway, that's who I thought of when I first started reading this.
  • CRConrad 2y ago
    TBF, it doesn't feel like the rarest of names.
• rk06 2y ago
  In my first job, I was part of an offshore ops team who maintained lot of code that no one (Onshore and offshore) wants to maintain. But like all ops. Code, it was business critical. This included a project called IefParser, its job was to parse incoming data files and put the data into databases. It was a serious project. Like really serious. The software input format came with a full manual with a section highlighting the changes from previous versions. I have not seen that level of detail before or since. And It was also very old. Written about a year or two after I was born.and never rewritten after. So the software side of things were less serious and more hacky. Core code was written in C. Database used was oracle. And code was driven by Perl batch script triggered via cron job. And all that ran on IBM AIX (unix). Yes,not windows or Linux. It ran on unix. The sheer difference in software requirements which were meticulously and the software was mind boggling. Some fun facts: - c code could not be compiled on windows . You need to login to dev server via putty and run makefile to do it. - Perl code was not checked in to repository. Perl code also was different for each environment. - unix version has a vi editor which for some reason didn’t tell if you were in edit mode or command mode. WTF! Yes, other editor didn’t exist. Apparently I was only one who bothered to learn vi in India. As no one else in India could reliably edit files - cron job schedule was also no checked in. After a “great learning opportunity “, I decided indeed to check that in - I once had the horror of fixing bugs in Perl and cron job. Apparently global variables are the state of art in Perl. - cron job always failed on New Year’s Day because the batch script uses MMDD formatted folder for temp file storage, and was unable to understand 0101 is greater than 1231. I volunteered to track down the issue, and fix it for good. M3 shot it down saying, “we don’t to take risks on such a mission critical service”
• deterministic 2y ago
  I loved reading this article. It made me laugh and cry and rage scream all at the same time. It is a true miracle that most things work most of the time. Don't look behind the curtains fowks! It might give you nightmares.
• m463 2y ago
  "When you can't tie a knot, tie a lot"
• lg2229897 2y ago
  [dead]