16 ms·

Mistakes engineers make in large established codebases

• ozim 2y ago
  Problem with consistency is that people miss forest for the trees. So lots of nitpicking on irrelevant stuff - keep files under 50 lines - that is silly consistency of little minds. Author of the post fortunately writes from experience perspective with architectural examples so I ca write that it is good article.
• kbruce 2y ago
  One common mistake: not playing Factorio and understanding how to scale, maintain, and debug a large factory with friends >> (this is a half-joke... iykyk)
• valzam 2y ago
  During my last job I have formulated what for me are the 2 unquestionable metrics to care about when trying to build long-term maintanable systems: - Consistency (fully agree with the article here) - Control Control to me means that you have to work extremely hard to lose the ability to change the parts you care about. For example: - Do not leak libraries and frameworks far into your business logic. At some point you want to introduce a new capabilty but say the class/type you re-used from a library makes it really awkward. Now you are faced with a huge refactor. The more logic, the purer and simpler the code should be. Ideally stdlib only. - Do not build magic, globally shared test harnesses. Helpers yes, but if you give up control over the environment a test runs is / setting up fixtures, test data etc. you will run into a world of pain due to dependencies between tests and especially the test data. - Do not let libraries dictate your application architecture. E.g. I always separate the web framework layer (controllers, views etc.) from the service and data layers. - Consistency plays a major part here. If you introduce 3 libraries to do the same thing you have basically given up control over that dependencies and refactors in the future will be much harder.
  • vimax 2y ago
    I lump those together into what I call Dependency-Driven Development. Manage and reduce dependencies first, both external dependencies and internal dependencies within code, and better code will follow. It's not always 100%, but in general the fewer the dependencies the better the code.
• deleted 2y ago
  [deleted]
• IvyMike 2y ago
  The "The cardinal mistake is inconsistency" is 100% true. We used to call the guiding philosophy of working in these codebases "When in Rome".
  • pryelluw 2y ago
    I have this bad codebase at work. Really bad. One of the things I’ve been working on for the past two years is making it consistent. I’m almost at the point where interfaces can be left alone and internals rewrites in a consistent style. People often ask why I hardly ever have any prod issues (zero so far this year). This is part of the reason. Having consistent codebases that are written in a specific style and implement things in similar manner. Some codebases make me feel like I’m reading a book in multiple languages …
    • cyco130 2y ago
      > zero so far this year I saw what you did there.
      • onemoresoop 2y ago
        Maybe that’s not even that bad if number of issues went down from multiple a day to none in a couple of days.
    • bizzletk 2y ago
      > People often ask why I hardly ever have any prod issues (zero so far this year). It also helps that we're still only in January!
      • bballer 2y ago
        Bahh thanks for the chuckle. The man is 7/7 as of today!
    • onemoresoop 2y ago
      > Some codebases make me feel like I’m reading a book in multiple languages … In most cases the codebase does consist of muliple languges.
• protonbob 2y ago
  I don't have a real critique because I don't have that many years in a codebase the size of OP (just 2). But I struggle with the advice to not try and make a clean section of the code base that doesn't depend on the rest of the application. Isn't part of good engineering trying to reduce your dependencies, even on yourself? In a latter part of the post, OP says to be careful tweaking existing code, because it can have unforeseen consequences. Isn't this the problem that having deep vertical slices of functionality tries to solve? High cohesion in that related code is grouped together, and low coupling in that you can add new code to your feature or modify it without worrying about breaking everyone else's code. Does this high cohesion and low coupling just not really work at the scale that OP is talking about?
  • gleenn 2y ago
    It's one thing to reduce dependency and another to have reduced consistency. If you have 10 web routes and 1 behaves differently, it doesn't matter if the code is cross coupled or not, it matters if it behaves similarly. Does it return the same status codes on error? Does it always return JSON with error messages inside? Do you auth the same way? The implementation can be wholly separate but end users will notice because logic on their side now has to special-case your 11th endpoint because you returned HTTP 20x instead of 40x on error. Or when you realize that you want to refactor the code to DRY it (Don't Repeat Yourself), now you can't reduce all the duplication because you have bespoke parts.
  • mbivert 2y ago
    I think the gist of it is humility: as a newcomer, you don't really know what's out there and why, and there are often good reasons for things to be how they are. Not always, but often enough for avoiding being too original to be favored. This doesn't imply relinquishing on "good engineering habits" either. Now, once you have a deeper understanding of the codebase, you'll know when and why to break away from existing patterns, but in the beginning phase, it's a good habit to start by learning carefully how things are designed and why.
  • Salgat 2y ago
    Consistency makes code predictable and reduces mental overhead. It doesn't mean you have to write it poorly like the rest of the codebase, but it does mean using the same general practices as the rest of the codebase. Think of it like using knockoff legos vs the real thing. They both work interchangeably which makes it easy to use them together, but you'd prefer to use the nicer lego pieces as much as possible in your builds because the material is higher quality, tighter tolerances, just overall works better even if it's the same shape as the knockoff pieces.
• hoten 2y ago
  I love how the first example is "use the common interfaces for new code". If only! That assumes there _is_ a common interface for doing a common task, and things aren't just a copy-paste of similar code and tweaked to fit the use case. So the only tweak I'd make here, is that if you are tempted to copy a bit of code that is already in 100 places, but with maybe 1% of a change - please, for the love of god, make a common function and parameterize out the differences. Pick a dozen or so instances throughout the codebase and replace it with your new function, validating the abstraction. So begins the slow work of improving an old code base created by undisciplined hands. Oh, and make sure you have regression tests. The stupider the better. For a given input, snapshot the output. If that changes, audit the change. If the program only has user input, consider capturing it and playing it back, and if the program has no data as output, consider snapshotting the frames that have been rendered.
  • edudobay 2y ago
    Yes, this is the counterpoint I'd make to "resist the urge to make every corner of the codebase nicer than the rest of it": in an inconsistent codebase, maybe we should prioritize making it consistent where possible, and reducing unnecessary duplication is one way to reduce future change costs.
  • ggregoryarms 2y ago
    At some points, new improvement and occasionally ingenuity need to find a healthy way back into the workflow. Moreso early on, but consistently over time as well. If we just create copies of copies forever, products degrade slowly over time. This is a problem in a few different spheres, to put it lightly. The main rule is a good one, but the article overfocuses on it.
• adamc 2y ago
  I really liked this: "as a general rule, large established codebases produce 90% of the value." People see the ugliness -- because solving real problems, especially if business practices are involved, is often very messy -- but that's where the value is.
  • bdangubic 2y ago
    I also find amusing that “legacy” more often than not gets used in negative conotation. I hear “legacy” and I think “bunch of people wrote some AWESOME shit that lasted so long that now other people get to view it as ‘legacy’”
    • tpoacher 2y ago
      There's a good chance that's not what people mean by this term though. It's probably used in the (now) classic sense as defined by M. Feathers in his "Working with legacy code" book. Code that is old but otherwise awesome, maintainable (or even actively maintained) and easy / a joy to work with are rarely referred to as "legacy code".
      • bdangubic 2y ago
        hmmm almost 3 decades in the industry and have very seldom (some exceptions) heard “legacy” for code that is old but awesome.
      • jimbokun 2y ago
        It doesn’t seem awesome at first glance because it takes longer to get up to speed on a large, old code base than a small, young one. But you will quickly learn how awesome the old code base is if you attempt to rewrite it, and realize everything the old code base takes into account.
    • forty 2y ago
      Measuring value by what stays the longer is tempting, but sometimes it's just that the mess is such that no one can touch it :)
• nitwit005 2y ago
  Except, the old stuff will be effectively untestable, and they'll demand near perfect coverage for your changes. Also, they're will be four incomplete refactorings, and people will insist on it matching the latest refactoring attempt. Which, will then turn out to be impossible, as it's too unfinished.
• crabbone 2y ago
  OP has some particular type of project in mind, where what they say probably makes sense. Not all large codebases are like that. For example, it could be a lot of individual small projects all sitting on some common framework. Just as an example: I've seen a catering business that had an associated Web site service which worked as follows. There was a small framework that dealt with billing and navigation etc. issues, and a Web site that was developed per customer (couple hundreds shops). These individual sites constituted the bulk of the project, but outside of the calls to the framework shared nothing between them, were developed by different teams, added and removed based on customer wishes etc. So, consistency wasn't a requirement in this scheme. Similar things happen with gaming portals, where the division is between some underlying (and relatively small) framework and a bunch of games that are provided through it, which are often developed by teams that don't have to talk to each other. But, to the user, it's still a single product.
• mjr00 2y ago
  > The other reason is that you cannot split up a large established codebase without first understanding it. I have seen large codebases successfully split up, but I have never seen that done by a team that wasn’t already fluent at shipping features inside the large codebase. You simply cannot redesign any non-trivial project (i.e. a project that makes real money) from first-principles. This resonates. At one former company, there was a clear divide between the people working on the "legacy monolith" in PHP and the "scalable microservices" in Scala/Go. One new Scala team was tasked with extracting permissions management from the monolith into a separate service. Was estimated to take 6-9 months. 18 months later, project was canned without delivering anything. The team was starting from scratch and had no experience working with the current monolith permissions model and could not get it successfully integrated. Every time an integration was attempted they found a new edge case that was totally incompatible with the nice, "clean" model they had created with the new service.
  • cratermoon 2y ago
    I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Not only had it gone into production, it made maintenance far easier.
    • deleted 2y ago
      [deleted]
    • mjr00 2y ago
      > Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.
• ge96 2y ago
  I'm just thinking about this time at a previous job, I was reviewing a PR and they decided to just find/replace every variable and switch from snake to camel case. I was like "why are you guys doing this, not part of the job". There was some back and forward on that. This is a place where PRs weren't about reviews but just a process to follow, ask someone to approve/not expect feedback. edit: job = ticket task
  • teeray 2y ago
    For stuff like this, the best thing is to have a style guide. If you disagree with something in the style guide (tabs/spaces, casing, etc.), you make a PR to the style guide and hash it out there. Also, those PRs need updates everywhere to make the codebase agree with the change and updates to relevant linters enforce it. Outside of that, the style guide is law.
  • dboreham 2y ago
    It's also literally not part of the job.
  • Cthulhu_ 2y ago
    What was the established code style (...if any) in that project? Anyway it doesn't sound like that was a very mature project or developers, not when the reviewer decide to just edit code instead of provide a review.
    • ge96 2y ago
      the old/existing code was all underscore, they wanted to use camelcase instead. it's a dumb thing to be argue about I know but it made the code review harder when instead of 10s of line diffs there's almost a hundred granted easy to see just changing casing
      • willseth 2y ago
        I just insist that style only changes go in a separate commit.
• gwbas1c 2y ago
  One thing I did was implement a code formatter, and enforce it in CI. "dotnet format" can do wonders, and solved most serious inconsistency issues.
  • Cthulhu_ 2y ago
    Honestly Go's approach to code formatting and it being taken over by other parties has saved so much trivial debates. I remember spending stupid amounts of review comments on stupid shit like formatting, trailing commas, semicolons, spacing, etc that should have been spent on more important things. How come automatic, standardized, non-ide bound automatic formatting has only been a thing in the past decade or so? I do recall Checkstyle for Java ages ago but I forgot if it did any formatting of its own.
    • duskwuff 2y ago
      > How come automatic, standardized, non-ide bound automatic formatting has only been a thing in the past decade or so? A lot of it boils down to "because the people writing code parsers/lexers weren't thinking about usability". Writing a C formatter, for example, depends on having a parser that doesn't behave like a compiler by inlining all your include files and stripping out comments. For a long time, writing parsers/lexers was the domain of compiler developers, and they weren't interested in features which weren't strictly required by the compiler. Another effect of those improvements, incidentally, has been higher quality syntax errors.
  • dgunay 2y ago
    This is good practice, but I don't think it's the kind of inconsistency the author is talking about. There are forms of inconsistency that an auto formatter can't fix. An example: old code deciding that an unhandled error is a 400 status code, but in newer code it causes a 500 status code (real problem I'm dealing with at work).
• mrkeen 2y ago
  There was only one mistake that the article felt like giving a header to: "The cardinal mistake is inconsistency" The instinct to keep doing things the wrong way because they were done the wrong way previously is strong enough across the industry without this article. I love to > take advantage of future improvements. However, newer and better ways of doing things are almost invariably inconsistent with the established way of doing things. They are dutifully rejected during code review. My current example of me being inconsistent with our current, large, established database: Every "unit test" we have hits an actual database (just like https://youtu.be/G08FxxwPjXE?t=2238 https://youtu.be/G08FxxwPjXE?t=2238). And I'm not having it. For the module I'm currently writing, I'm sticking the reads behind a goddamn interface so that I can have actual unit tests that will run without me spinning up and waiting for a database.
  • jrockway 2y ago
    I like keeping things consistent even if the consistent way is "wrong". One thing that bugged me about the large codebase I most recently worked on is that we used a custom assert library for tests. The Go team says this about them: https://go.dev/wiki/TestComments#assert-libraries https://go.dev/wiki/TestComments#assert-libraries , and having learned Go at Google, I would never have been allowed to check in code like that. But this place wasn't Google and there were tens of thousands of lines of these tests, so I told new developers to keep doing things the "wrong" way. This didn't cause many problems, even if failing tests failing too soon is pretty annoying. Most of the time the tests pass, and the yes/no signal is valuable even if you can debug more by simply `t.Errorf(...)` and continuing. As for starting databases during tests, it's saved me a lot of trouble over the years. One time, we used sqlite for tests and Postgres for production. We had some code that inserted like `insert into foo (some_bool) values ('t')` and did a query like `select * from foo where some_bool='true'`. This query never matched rows in the tests, because t != true in SQLite, but t == true in Postgres. After that, I found it easier to just run the real database that's going to be used in production for tests. The only thing that behaves identically to production is the exact code you're running in production. Over here, I have code that uses a hermetic Postgres binary (and chain of shared libraries because Postgres hates static linking) that starts up a fresh Postgres instance for each test. It takes on the order of a millisecond to start up: https://github.com/jrockway/monorepo/blob/main/internal/testpostgres/testpostgres.go https://github.com/jrockway/monorepo/blob/main/internal/test.... The biggest problem I've had with using the "real" database in tests is low throughput because of fsync (which `perf` showed me when I finally looked into it). Fortunately, you can just disable fsync, and boy is it fast even with 64 tests running in parallel. One thing that's been slow in the past is applying 50 migrations to an empty database before every test. When you have one migration, it's fast, but it's one of those things that starts to slow down as your app gets big. My solution is to have a `go generate` type thing that applies the migrations to an empty database and pg_dumps resulting database to a file that you check in (and a test to make sure you remembered to do this). This has two benefits; one, tests just apply a single SQL file to create the test database, and two, you get a diff over the entire schema of your database for the code reviewer to look at during code reviews. I've found it incredibly useful (but don't do it for my personal projects because I've been lazy and it's not slow yet). Overall, my take on testing is that I like an integration test more than a unit test. I'd prefer people spend time on exercising a realistic small part of the codebase than to spend time on mocks and true isolation. This is where a lot of bugs lie. Of course, if you are writing some "smart" code and not just "glue" code, you're going to be writing a lot of unit tests. Neither replaces the other, but if you can spend 30 seconds writing a test that does actual database queries or 2 weeks mocking out the database so the test can be a unit test instead of an integration test, I'd tell you to just write the integration test. Then you know the real code works.
• o_nate 2y ago
  A big part of this advice boils down to the old adage: "Don't remove a fence if you don't know why it was put there." In other words, when making changes, make sure you preserve every behavior of the old code, even things that seem unnecessary or counter-intuitive.
  • Aloha 2y ago
    Chesterton's Fence
• gwbas1c 2y ago
  Unit tests, exhaustive regression tests, and automated tests are the best way to prevent regressions. Time spent writing good unit tests today allows you to make riskier changes tomorrow; good unit tests de-risk refactors.
  • notyourwork 2y ago
    Unit tests cover the single functionality but ignore the system as a whole. Most regressions I've seen in industry are because of a lack of understanding how the system components interact with one another. Therefore, I see unit tests as one pillar but also suspect that without good quality integration or end-to-end testing you won't be able to realize the riskier re-factors you describe. Perhaps you consider these part of your regression testing and if so, I agree.
    • magicalhippo 2y ago
      The way I like to view it is that Lego bricks might pass unit tests aka QC with zero issue, but you can still easily build an unreliable mess with them.
  • grumpy_coder 2y ago
    I see an over-reliance on automated tests recently. Often suggesting just passing the CI tests is enough to approve a change. In ancient code it's just as important to limit the blast radius of your changes, and have a good reason for making them. Not changing something is the ultimate way to prevent a regression.
• peterldowns 2y ago
  I agree that consistency is important — but what about when the existing codebase is already inconsistent? Even worse, what if the existing codebase is both inconsistent and the "right way to do things" is undocumented? That's much closer to what I've experienced when joining companies with lots of existing code. In this scenario, I've found that the only productive way forward is to do the best job you can, in your own isolated code, and share loudly and frequently why you're doing things your new different way. Write your code to be re-used and shared. Write docs for it. Explain why it's the correct approach. Ask for feedback from the wider engineering org (although don't block on it if they're not directly involved with your work.) You'll quickly find out if other engineers agree that your approach is better. If it's actually better, others will start following your lead. If it's not, you'll be able to adjust. Of course, when working in the existing code, try to be as locally consistent as possible with the surrounding code, even if it's terrible. I like to think of this as "getting in and out" as quickly as possible. If you encounter particularly sticky/unhelpful/reticent team members, it can help to remind them that (a) the existing code is worse than what you're writing, (b) there is no documented pattern that you're breaking, (c) your work is an experiment and you will later revise it. Often asking them to simply document the convention that you are supposedly breaking is enough to get them to go away, since they won't bother to spend the effort.
  • deleted 2y ago
    [deleted]
  • mgfist 2y ago
    I rarely see large 10m+ LOC codebases with any sort of strong consistency. There are always flavors of implementations and patterns all over the place. Hell, it's common to see some functionality implemented multiple times in different places
    • peterldowns 2y ago
      And it's fine, right? Honestly I think people need to realize that part of being a good engineer is being able to deal with inconsistency. Maybe submodule A and submodule B do network requests slightly differently but if both ways are reasonable, working, and making the company money, it's probably not worth delaying product improvements in order to make things "more consistent." On the other hand if no one in your company cares about consistency, at some point everything becomes so awful you basically won't be able to retain engineers or hire new ones, so this is a place where careful judgement is needed.
• hnthrow90348765 2y ago
  To be fair, the previous engineers got paid to write the legacy mess and were employed for a long time if there's a lot of it. Where is the incentive to go the extra mile here? Do you eventually put up with enough legacy mess, pay your dues, then graduate to the clean and modern code bases? Because I don't see a compelling reason you should accept a job or stay in a code base that's a legacy mess and take on this extra burden.
  • __MatrixMan__ 2y ago
    Would you be willing to put that take on your LinkedIn profile? If not... well that's why.
    • hnthrow90348765 2y ago
      Remembering a controversial take over the code produced/improved is evidence to me that the matter is not settled that we should be spending the extra effort to align our practices with the article. The incentives are not there.
      • __MatrixMan__ 2y ago
        Of course it's not settled. It's awful (the code, not the article). It was written at a time when they didn't know better. Now they do, but they need somebody to maintain it anyway. > Do you eventually put up with enough legacy mess, pay your dues, then graduate to the clean and modern code bases? Yeah, that's called retirement. The point of the article isn't that whatever you're conforming to in the legacy codebase is worth preserving. The point is that whatever hell it is, it'll be a worse hell if you make it an inconsistent one.
        • hnthrow90348765 2y ago
          If they want to pay more to maintain legacy messes, then I'm fine with more rules. That shows the business wants it done right. They don't though, so I can't agree with putting in more work for no extra compensation. I know this counter argument sounds crabby, but going along with existing conventions on a legacy code base might be a lot of work for someone who's only familiar with more recent practices. It's not something you can passively do. Plus having to adopt these older patterns won't help your resume, which is an opportunity cost we are absorbing for free (and shouldn't have to)
• Hansome 2y ago
  [flagged]
• jongjong 2y ago
  I'm now working on a codebase which is quite large (13 micro-services required to run the main product); all containerized to run on Kubernetes. The learning curve was quite steep but luckily, I was familiar with most of the tech so that made it easier (I guess that's why they hired me). The project has been around for over 10 years so it has a lot of legacy code and different repos have different code styles, engine versions and compatibility requirements. The biggest challenge is that it used to be maintained by a large team and now there are just 2 developers. Also, the dev environment isn't fully automated so it takes like 20 minutes just to launch all the services locally for development. The pace of work means that automating this hasn't been a priority. It's a weird experience working on such project because I know for a fact that it would be possible to create the entire project from scratch using only 1 to 3 services max and we would get much better performance, reliability, maintainability etc... But the company wouldn't be willing to foot the cost of a refactor so we have to move at steady snail's pace. The slow pace is because of the point mentioned in the article; the systems are all intertwined and you need to understand how they integrate with one another in order to make any change. It's very common that something works locally but doesn't work when deployed to staging because things are complicated on the infrastructure side with firewall rules, integration with third-party services, build process, etc... Also, because there are so many repos with different coding styles and build requirements, it's hard to keep track of everything because some bug fixes or features I implement touch on like 4 different repos at the same time and because deployment isn't fully automated, it creates a lot of room for error... Common issues include forgetting to push one's changes or forgetting to make a PR on one of the repos. Or sometimes the PR for one of the repos was merged but not deployed... Or there was a config or build issue with one of the repos that was missed because it contained some code which did not meet the compatibility requirements of that repo...
  • jimbokun 2y ago
    If you can’t change, test and deploy each service independently from the others, you don’t really have separate services. You just have a Ball of Mud held together with HTTP calls.
    • jongjong 2y ago
• 0xbadcafebee 2y ago
  "Coding defensively" is perhaps the understatement of the year. Good software architecture is, in my opinion, the single most powerful tool you have to keep things from becoming an unmanageable mess. If I could give one "defensive coding" tip, it would be for seniors doing the design to put in road blocks and make examples that prevent components from falling for common traps (interdependency, highly variant state, complex conditions, backwards-incompatibility, tight coupling, large scope, inter-dependent models, etc) so that humans don't have to remember to avoid those things. Make a list of things your team should never do and make them have a conversation with a senior if they want to do it anyway. Road blocks are good when they're blocking the way to the bad. Starting with good design leads to continuing to follow good design. Not starting with good design leads to years of pain. Spend a lot more time on the design than you think you should.
• pablobaz 2y ago
  In my experience with very large codebases, a common problem is devs trying to improve random things. This is well intentioned. But in a large old codebase finding things to improve is trivial - there are thousands of them. Finding and judging which things to improve that will actually have a real positive impact is the real skill. The terminal case of this is developers who in the midst of another task try improve one little bit but pulling on that thread leads to them attempting bigger and bigger fixes that are never completed. Knowing what to fix and when to stop is invaluable.
  • jimbokun 2y ago
    Which can lead to trying to rewrite Netscape Navigator from scratch and killing the company: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
  • Kinrany 2y ago
    Do you think boyscouting, "leave it better than you found it" is misguided as well?
    • bogdan 2y ago
      I always took it as "leave it better than you found it" across the files that I've been working on (with some freedom as long I'm on schedule). My focus is to address the ticket I'm working on. Larger improvements and refactorings get ticketed separately (and yes, we do allocate time for them). In other words, I don't think it's misguided.
    • bricestacey 2y ago
      I do not believe in "boyscouting". I think if you want to leave it better, make a ticket and do it later. Tacking it on to your already planned work is outside the scope of your original intent. This will impact your team's ability to understand and review your changes. Your codebase is unlikely to be optimized for your whimsy. Worse though is when a reviewer suggests boyscouting. I've seen too many needless errors after someone happened to "fix a tiny little thing" and then fail to deliver their original task and further distract others trying to resolve the mistake. I believe clear intention and communication are paramount. If I want to make something better, I prefer to file a ticket and do it with intention.
• deleted 2y ago
  [deleted]
• forty 2y ago
  I have been working on a code base that is now 14 year old for many years (almost since the beginning), and is now well over 1M LoC of typescript (for Nodejs) - we are only 20-30 engineers working on it, rather than the 100-1000 suggested on the article. And I can say I couldn't agree more with the article. If you have to work on such projects, there are two things to keep in mind: consistency and integration tests.
  • jamesfinlayson 2y ago
    > integration tests Yes. I remember working in a 700,000+ line PHP code base that around 30% unit test coverage and an unknown percentage of e2e test coverage. I kept my changes very localised because it was a minefield. Also, the unit tests didn't do teardown so adding a new unit test required you to slot it in with assertions accounting for the state of all tests run so far.
• maxwellg 2y ago
  I never really understood putting consistency on a pedestal. It's certainly nice when everything operates exactly the same way - but consistency for consistency's sake is awful to work in too. If a team realizes that logging library B is better than library A, and but NEVER switches from A to B because of consistency concerns, then in two years they'll still all be using inferior tools and writing worse code. Similarly, if a team DOES decide to switch from A to B, they probably shouldn't spend months rewriting all previous code to use the new tool. It's ok for multiple established patterns to live in the same codebase, so long as everyone has an understanding of what the "correct" pattern should be for all new code.
  • jakefromstatecs 2y ago
    The consistency that they're referring to specifically is to do with consistency in the way that certain features or functionality is implemented. To make your example match, it would be more so that there are two teams A and B, Team A already created a framework and integration for logging across the entire application. Team B comes along and doesn't realize that this framework exists, and also invents their own framework and integration for logging. This is the type of consistency that the author points to, because Team B could have looked at other code already referencing and depending on the logging framework from Team A and they would have avoided the need to create their own.
  • lilyball 2y ago
    "Consistency for consistency's sake" is usually a misinterpretation of "Consistency because there are reasons for the way things are already done and you don't understand those reasons well enough to diverge". If you understand the current system completely, then you can understand when to diverge from the system (though this is usually better expressed as "when to improve the system" rather than doing something completely new). If you don't understand the current system, then you can't possibly ensure that you haven't missed something in your shiny new way of doing things.
  • ted_bunny 2y ago
    It's about minimizing cognitive load.
• lr4444lr 2y ago
  Single-digit million lines of code (~5M, let’s say) Somewhere between 100 and 1000 engineers working on the same codebase The first working version of the codebase is at least ten years old All of these things, or any of them? In any event, though I agree with him about the importance of consistency, I think he's way off base about why and where it's important. You might as well never improve anything with the mentality he brings here.
• trgn 2y ago
  Great article, has much wisdom. If you're contributing to a large code base, you need to be knee deep. It's against your developer instinct, but it's constant pruning and polishing. It is the only way.
• PaulDavisThe1st 2y ago
  > You can’t practice it beforehand (no, open source does not give you the same experience). This is ridiculous. Even if you want to ignore the kernel, there are plenty of "large established codebases" in the open source world that are at least 20 years old. Firefox, various *office projects, hell, even my own cross-platform DAW Ardour is now 25 years old and is represented by 1.3M lines of code at this point in time. You absolutely can practice it on open source. What you can't practice dealing with is the corporate BS that will typically surround such codebases. Which is not to say that the large established codebases in the open source world are BS free, but it's different BS.
  • francisofascii 2y ago
    Agreed. If you didn’t work on a codebase of 1000 engineers, how else would you practice.
  • sfink 2y ago
    Personally, I think that part is just incomplete. I think that there is an attitude that because you have jumped into several older open source projects and successfully made meaningful changes, that you have mastered the art of diving into legacy codebases and can handle anything. Even if those projects were not all that large. People sometimes think that because in theory thousands of people could be looking at the code, that they can treat the code as if it has been vetted over time by thousands of people. But I agree that there are absolutely eligible open source codebases that could be used to practice beforehand. I'd better; I work on Firefox. It is not a small thing to dive into, but people successfully do, and get solid experience from it.
• jongjong 2y ago
  I once worked on a large project in the past where it took 3 days to rename a field in an HTTP response because of how many services and tests were affected. Just getting that through QA was a huge challenge. Working in a large dev team, focusing on a small feature and having a separate product manager and QA team makes it easier to handle the scale though. Development is very slow but predictable. In my case, the company had low expectations and management knew it would take several months to implement a simple form inside a modal with a couple of tabs and a submit button. They hired contractors (myself included), paying top dollar to do this; for them, the ability to move at a snail's pace was worth it if it provided a strong guarantee that the project would eventually get done. I guess companies above a certain size have a certain expectation of project failure or cancellation so they're not too fussed about timelines or costs. It's shocking coming from a startup environment where the failure tolerance is 0 and there is huge pressure to deliver on time.
  • AndyNemmity 2y ago
    Only 3 days? That's incredible. Getting a PR reviewed in 3 days is an achievement!
    • jongjong 2y ago
      Not sure if you're serious. I can't remember working for any company that took more than a day to review a PR. I think this company took about 1 day to provide QA feedback and I was thinking that it's so slow. In startup land, I got my code reviewed by the CTO within a few hours. It was rare if it required a whole day like if he was too busy. In my current company, the other dev usually reviews and merges my code to staging within an hour. Crazy thing is we don't even write tests. A large project with no tests.
      • AndyNemmity 2y ago
        I was definitely serious.
      • tmnvdb 2y ago
        In my current job I did a PR in the first week of joining. It was reviewed after exactly 2 years. I had to rewrite the whole PR because of the affected lines had changed. Of course I did not remember at all what is was about. Some PRs are faster but some are slower as well.
• ram_rar 2y ago
  >The other reason is that you cannot split up a large established codebase without first understanding it. I have seen large codebases successfully split up, but I have never seen that done by a team that wasn’t already fluent at shipping features inside the large codebase I cannot resonate with this. Having worked with multiple large code bases 5M+, splitting the codebase is usually a reflection of org structure and bifurcation of domain within eng orgs. While it may seem convoluted at first, its certainly doable and gets easier as you progress along. Also, code migrations of this magnitude is usually carried out by core platform oriented teams, that rarely ship customer-facing features.
• sirmarksalot 2y ago
  Consistency is often helpful, but you also need to be wary of cargo culting. For example, you see a server back end that uses an ORM model and you figure you'll implement your new feature using the same patterns you see there. Then a month later the author of the original code you cribbed comes by and asks you, "just out of curiosity, why did you feel the need to create five new database tables for your feature?" I know, that's a pretty specific "hypothetical," but that experience taught me that copying for the sake of consistency only works if you actually understand what it is you're copying. And I was also lucky that the senior engineer was nice about it.
• LAC-Tech 2y ago
  Debates about the technical merits aside...the alternative to not allowing people to build their own nice little corner of a legacy codebase is not a bunch of devs building a consistent codebase. It's devs not wanting to touch the codebase at all. Working on an old shitty codebase is one thing. Being told you have to add to the shit is soul crushing.
• lifeisstillgood 2y ago
  All these are part of a different mistake - lack of common culture amount the coders. It’s really really hard, and often antithetical to the politics of the org, but having a common area (email, actual physical meetings) between the “tech leads” - so that’s one in 8 to one in twenty devs, is vital. Sharing code, ideas, good and bad etc is possible - but it requires deliberate effort
• _fizz_buzz_ 2y ago
  > no, open source does not give you the same experience Why not? There are open source projects that are many years old with millions lines of code and many developers.
  • ZeWaka 2y ago
    yeah, this doesn't make sense. The project I work on has had a thousand+ contributors of extremely varied skill levels, and is over 15 years old. Many lines of code, but I'm not going to count them because that's a terrible metric. This fits all of the criteria outlined in the article. Sure, it might not apply to portfolio project #32 but there's plenty of open source repositories out there that are huge legacy codebases.
• lmm 2y ago
  I've worked in codebases like this and disagree. Consistency isn't the most important, making your little corner of the codebase nicer than the rest of it is fine, actually, and dependencies are great - especially as they're the easiest way to delete code (the article is right about the importance of that). What's sometimes called the "lava layer anti-pattern" is actually a perfectly good way of working, that tends to result in better systems than trying to maintain consistency. As Wall says, the three cardinal virtues of a programmer are laziness, impatience, and hubris; if you don't believe you can make this system better then why would you even be working on it? Also if the system was actually capable of maintaining consistency then it would never have got that large in the first place. No-one's actual business problem takes 5M lines of code to describe, those 5M lines are mostly copy-paste "patterns" and repeated attempts to reimplement the same thing.
  • jimbokun 2y ago
    Pulling in lots of dependencies will eventually grind progress on features to a halt as you spend more and more time patching and deploying vulnerabilities. The probability of seeing new vulnerabilities I believe is pretty much linear in the number of dependencies you have.
    • abc-1 2y ago
      As opposed to your in-house code which is vulnerability free? The issue isn’t vulnerability's, it’s dependency hell where all your packages are constantly fighting each other for specific versions. Although some languages handle this better than others.
      • jimbokun 2y ago
        In house code could very well have many fewer vulnerabilities, as you only write exactly the functionality you need, vs pulling a large dependency and only using a small percentage of the API.
        • lmm 2y ago
• tonymet 2y ago
  Or you can help migrate them to Karenina Microservices: each service can be dysfunctional in its own way.
• dang 2y ago
  Small prior discussion: Mistakes engineers make in large established codebases - https://news.ycombinator.com/item?id=42570490 https://news.ycombinator.com/item?id=42570490 - Jan 2025 (3 comments)
• hnanon98791 2y ago
  > Single-digit million lines of code (~5M, let’s say) > Somewhere between 100 and 1000 engineers working on the same codebase > The first working version of the codebase is at least ten years old > The cardinal mistake is inconsistency Funny enough, the author notes the problem on why consistency is impossible in such a project and the proceeds to call it the cardinal mistake. You cannot be consistent in a project of that size and scope. Full stop. Half those engineers will statistically be below average and constantly dragging the codebase towards their skill level each time they make a change. Technology changes a lot in ten years, people like to use new language features and frameworks. And the final nail in the coffin: the limits of human cognition. To be consistent you must keep the standards in working memory. Do you think this is possible when the entire project is over a million LOC? Don't be silly. There's a reason why big projects will always be big balls of mud. Embrace it. http://www.laputan.org/mud/ http://www.laputan.org/mud/
  • noodletheworld 2y ago
    > people like to use new language features and frameworks. Have of the point of this article is that people need to suck it up and not use new frameworks sometimes... There are times for coding in a way you, personally, find pleasing; and there are times when: > So you should know how to work in the “legacy mess” because that’s what your company actually does. Good engineering or not, it’s your job. A quote from the 'big ball of mud': > Sometimes it’s just easier to throw a system away, and start over. It is easier, but it's also a) not your decision and b) enormously disruptive and expensive. How do you tell if you're in the 'naive and enthusiastic but misguided' camp, or in the 'understand the costs and benefits and it's worth a rewrite' camp? Maybe the takeaway from the OP's post really should be this one: > If you work at a big tech company and don’t think this is true, maybe you’re right, but I’ll only take that opinion seriously if you’re deeply familiar with the large established codebase you think isn’t providing value. ^ because this is the heart of it. If you don't understand, or haven't bothered to understand, or haven't spent the time understanding what is already there, then you are not qualified to make large scale decisions about changing it.
    • Tknl 2y ago
• dav 2y ago
  I have three maxims that basically power all my decisions as an engineer: 1. The three C’s: Clarity always, Consistency with determination, Concision when prudent. 2. Keep the pain in the right place. 3. Fight entropy! So in the context of the main example in this article, I would say you can try to improve clarity by e.g. wrapping the existing auth code in something that looks nicer in the context of your new endpoint but try very hard to stay consistent for all the great reasons the article gives.
  • ctxc 2y ago
    It's got a nice ring to it :)
• kazinator 2y ago
  This is reminiscent of the stupid things people have tried to fix email. If I just do this simple thing in my mail client. ... or server ... mail security and spam and whatever else will be solved.
• luisgvv 2y ago
  My only advice is "if it ain't broke don't fix it". And if you're going to improve something, make sure it's something small and local, ideally further from the "core logic" of the business.
• rf15 2y ago
  > Single-digit million lines of code (~5M, let’s say) as someone working on a 60M codebase, we have very different understandings of the word "large". My team is leaning more towards "understand the existing code, but also try to write maintainable and readable code". Everything looks like a mess built by a thousand different minds, some of them better and a lot of them worse, so keeping consistency would just drag the project deeper into hell.
• Animats 2y ago
  > Single-digit million lines of code (~5M, let’s say) > Somewhere between 100 and 1000 engineers working on the same codebase > The first working version of the codebase is at least ten years old That's 5,000 to 50,000 lines of code per engineer. Not understaffed. A worse problem is when you have that much code, but fewer people. Too few people for there to be someone who understands each part, and the original authors are long gone. Doing anything requires reverse engineering something. Learning the code base is time-consuming. It may be a year before someone new is productive. Such a job may be a bad career move. You can spend a decade learning a one-off system, gaining skills useless in any other environment. Then it's hard to change jobs. Your resume has none of the current buzzwords. This helps the employer to keep salaries down.
  • atmavatar 2y ago
    > A worse problem is when you have that much code, but fewer people. Too few people for there to be someone who understands each part, and the original authors are long gone. Maybe. I spent most of my career at a small mom and pop shop where we had single-digit MLOC spanning 20-25 years but only 15-20 engineers working at any given time. This wasn't a problem, though, because turn-over was extremely low (the range in engineer count was mostly due to internships), so many of the original code owners were still around, and we spent some effort to spread out code ownership such that virtually all parts were well understood by at least 2 people at any given moment. If anything, I rather shudder at the thought of working somewhere that only has ~5M lines of code split up amongst 100 (and especially 1000) engineers over a span of 10 years. I can't imagine producing only 5-50 KLOC over that time, even despite often engaging in behind-the-scenes competition with colleagues over who could produce pull requests with the least (and especially negative) net LOC. > Your resume has none of the current buzzwords. That's one of my bigger pet peeves about software development, actually. While you probably didn't mean it this way, over the years, I encountered a number of people who'd consistently attempt to insert fad technologies for the sake of doing so, rather than because they actually provided any kind of concrete benefit. Quite the contrary: they often complicated code without any benefit whatsoever. My more experienced colleagues snidely referred to it as resume-driven development. I can't hate people doing this too much, though, because our industry incentivizes job hopping.
  • 2y ago
• riwsky 2y ago
  Sounds like common law—one of the biggest, oldest "codebases" there is. To quote Wikipedia: > Common law is deeply rooted in stare decisis ("to stand by things decided"), where courts follow precedents established by previous decisions.[5] When a similar case has been resolved, courts typically align their reasoning with the precedent set in that decision.[5] However, in a "case of first impression" with no precedent or clear legislative guidance, judges are empowered to resolve the issue and establish new precedent.
• jheriko 2y ago
  there is a balance to be had here. oftentimes people make their own corner of the code because they are afraid, or their superiors are, of the scope of work which is actually about 3 hours of consistent work with good discipline and not the 17 years they imagine. millions of lines of code itself is a code smell. some of the absolute worst code i have to work with comes from industry standard crapware that is just filled with lots of do nothing bug factories. you gotta get rid of them if you want to make it more stable and more reliable. however... i often see the problem, and its not "don't do this obvious strategy to improve qol" its "don't use that bullshit you read a HN article about last week" i suspect this is one of those.
• Jean-Papoulos 2y ago
  I'm all for consistency. But imagine having a codebase where most operations point back to an in-memory dataset representation of a database table, and everytime you change your position in this dataset (the only "correct" way of accessing that table's data), it updates the UI accordingly. New feature where you compare 2 records ? Too bad, the UI is going to show them both then go back to the first one in a epileptic spasm. Sometimes, things are just that bad enough that keeping it consistent would mean producing things that will make clients call saying it's a bug. "No sorry, it's a feature actually".
• BlackFly 2y ago
  > Because it protects you from nasty surprises, it slows down the codebase’s progression into a mess, and it allows you to take advantage of future improvements. The codebase is already a nasty surprise for people coming in from the outside with experience or people that are aware of current best practices or outside cultures, therefore, the codebase is already a mess and you cannot take advantage of future improvements without a big bang since that would be inconsistent. How to keep your code evolving in time and constantly feeling like it is something you want to maintain and add features to is difficult. But constantly rewriting the world when you discover a newer slight improvement will grind your development to a halt quickly. Never implementing that slight improvement incrementally will also slowly rot your feelings and your desire to maintain the code. Absolute consistency is the opposite of evolution: never allowing experimentation; no failed experiments mean no successes either. Sure, too much experimentation is equally disastrous, but abstinence is the other extreme and is not moderation.
• praptak 2y ago
  Another post from the same author puts this in an interesting context: https://www.seangoedecke.com/glue-work-considered-harmful/ https://www.seangoedecke.com/glue-work-considered-harmful/ (follow up: https://www.seangoedecke.com/cynicism/ https://www.seangoedecke.com/cynicism/) Keeping the code base tidy is glue work, so you should only do enough of it to ship features. So maybe these are not "mistakes" but rather tactical choices made by politically smart engineers focused on shipping features.
  • lmm 2y ago
    Following management instructions against your better judgement is hardly uncynical - unless it's coming from a place of such naivety that you actually think you're wrong and they're right, and I don't think it is. My experience is the opposite of the author's: in terms of their revealed preferences, line workers care far more about the company and its customers than managers and executives do, precisely because it's far easier for the latter to fail upwards than the former.
• KronisLV 2y ago
  > If they use some specific set of helpers, you should also use that helper (even if it’s ugly, hard to integrate with, or seems like overkill for your use case). You must resist the urge to make your little corner of the codebase nicer than the rest of it. This reads like an admission of established/legacy codebases somewhat sucking to work with, in addition to there being a ceiling for how quickly you can iterate, if you do care about consistency. I don't think that the article is wrong, merely felt like pointing that out - building a new service/codebase that doesn't rely on 10 years old practices or code touched by dozens of developers will often be far more pleasant, especially when the established solution doesn't always have the best DX (like docs that tell you about the 10 abstraction layers needed to get data from an incoming API call through the database and back to the user, and enough tests). Plus, the more you couple things, the harder it will be to actually change anything, if you don't have enough of the aforementioned test coverage - if I change how auth/DB logic/business rules are processed due to the need for some refactoring to enable new functionality, it might either go well or break in hundreds of places, or worse yet, break in just a few untested places that aren't obvious yet, but might start misbehaving and lead to greater problems down the road. That coupling will turn your hair gray.
  • olau 2y ago
    But if you see yourself as trying to make the world a better place, you'll accept that because the large code base is actually doing a lot of good out there. The article discusses this at the end.
    • KronisLV 2y ago
      Sure, I can accept that, I'm not saying they're all bad, just that they have very obvious drawbacks, same as projects that pick untested technologies and get burned when they cease to exist after a year. > Large codebases are worth working in because they usually pay your salary Though remember that greenfield projects might also pay your salary and be better for your employability and enjoyment of your profession in some cases. They might be the minority in the job market, though.
  • noisy_boy 2y ago
• quick_brown_fox 2y ago
  I mostly agree, however experienced a different challenge exactly for the very reason of consistency: I used to work within the Chromium codebase (at the order of 10s of million LOC) and the parts I worked in were generally in line with Google's style guide, i.e. consistent and of decent quality. The challenge was to identify legacy patterns that shouldn't be imitated or cargo-culted for the sake of consistency. In practice that meant having an up to date knowledge of coding standards in order to not perpetuate anti-patterns in the name of consistency.
• sema4hacker 2y ago
  I once did some contract work for a development group at Apple in the late 90's working on a product not yet released. It was the first time I was exposed to a large codebase that could be updated by any of a large number of programmers at any time. While investigating a bug, it was scary to constantly see large, complicated routines headed with one-line comments from 20 or 30 people logging the changes they had made. There would be no consistent style of code, no consistent quality, no consistent formatting, no real sense of ownership, a real free-for-all. The system not only immediately projected a sense of hopelessness, but also indicated that any attempts at improvement would quickly be clobbered by future sloppy changes.
• cannibalXxx 2y ago
  [dead]
• ReflectedImage 2y ago
  "as a general rule, large established codebases produce 90% of the value." This is only until your new upstart competitor comes along, rewrites your codebase from scratch and runs you out of the market with higher development velocity (more features).
  • zild3d 2y ago
    Not really, a startup can certainly disrupt the old big cos, but as its growing and taking on more large enterprise customers and scaling up teams, by the time its "producing 90% of the value" you're a few short years from finding yourself with a large, complex, and legacy codebase.
  • rmk 2y ago
    > rewrites your codebase from scratch This almost never happens. It takes a long time and huge amounts of money to come up to parity, and in the meantime, the legacy org is earning money on the thing you're trying to rewrite. It's more often the case that the technology landscape shifts dramatically helping a niche player (who has successfully saturated the niche) become mainstream or more feasible. Take, for example, Intel. Their CISC designs and higher power consumption is now being challenged by relatively simpler RISC and lower power designs. Or Nvidia with its GPUs. In both cases, it's the major shifts that have hurt Intel. No one can outcompete Intel in making server CPUs of old, if they are starting from scratch. Take another example, this time, of a successful competitor (of sorts). Oracle vs Postgres. Same deal, except that Postgres is the successor of Ingres (which doesn't exist anymore), and was developed at Berkeley and was open-source (i.e., it relied upon the free contributions of a large number of developers). I doubt that another proprietary database has successfully challenged Oracle. Ask any Oracle DB user, and you will likely get the answer that other databases are a joke compared to what it offers.
• physicsguy 2y ago
  A big problem I come across is also half-assing the improvements. Taks as an example - for some reason you need to update an internal auth middleware library, or a queue library - say there is a bug, or a flaw in design that means it doesn't behave as expected in some circumstances. All of your services use it. So someone comes along, patches it, makes the upgrade process difficult / non-trivial, patches the one service they're working on, and then leaves every other caller alone. Maybe they make people aware, maybe they write a ticket saying "update other services", but they don't push to roll out the fixed version in the things they have a responsibility for.
• cess11 2y ago
  Very good advice. An implication from being reluctant to introducing dependencies is that you should remove dependencies if you can. Perhaps different parts of the system is using different PDF-generation libraries, or some clever person introduced Drools at some point but you might as well convert those rules to plain old Java. Tooling is important too. IDE:s are great, but one should also use standalone static analysis, grepping tools like ripgrep and ast-grep, robust deterministic code generation, things like that.
• fedeb95 2y ago
  this should be tattooed to every newly hired CTO arm.
• perdomon 2y ago
  https://jimmyhmiller.github.io/ugliest-beautiful-codebase https://jimmyhmiller.github.io/ugliest-beautiful-codebase Heres an alternate take of what a lack of consistency and a tendency to build microservices makes for an enjoyable work environment (and an admittedly ugly codebase)
• dalton_zk 2y ago
  Interesting point of view
• t43562 2y ago
  Big codebases develop badly/well because of the established company culture. Culture tends to flow from the top. If it's very expedient at the top then the attitude to code will be too. You get stuck in the "can't do anything better because I cannot upgrade from g++-4.3 because there's no time or money to fix anything, we just work on features. Work grinds to a near halt because the difficulties imposed by tech debt. The people above don't care because they feel they're flogging a nearly-dead horse anyhow or they're just inappropriately secure about its market position. Your efforts to do more than minor improvements are going to be a waste. Even in permissive environments one has to be practical - it's better to have a small improvement that is applied consistently everywhere than a big one which affects only one corner. It has to materially help more than just you personally otherwise it's a pain in the backside for others to understand and work with when they come to do so. IMO this is where you need some level of consensus - perhaps not rigid convention following but at least getting other people who will support you. 2 people are wildly more powerful and convincing than 1. The senior programmers are both good and bad - they do know more than you and they're not always wrong and yet if you're proposing some huge change then you very likely haven't thought it out fully. You probably know how great it is in one situation but not what all the other implications are. Perhaps nobody does. The compiler upgrade is fine except that on windows it will force the retirement of win2k as a supported platform .... and you have no idea if there's that 1 customer that pays millions of dollars to have support on that ancient platform. So INFORMATION is your friend in this case and you need to have it to convince people. In the Internet world I suppose the equivalent question is about IE5 support or whatever. You have to introduce ideas gradually so people can get used to them and perhaps even accept defeat for a while until people have thought more about it. It does happen that people eventually forget who an idea came from and you need to resist the urge to remind them it was you. This almost never does you a favour. It's sad but it reduces the political threat that they feel from you and lets them accept it. One has to remember that the idea might not ultimately have come from you either - you might have read it in a book perhaps or online. At the end, if your idea cannot be applied in some case or people try to use it and have trouble, are you going to help them out of the problem? This is another issue. Once you introduce change be prepared to support it. In other words, I have no good answers - I've really revolutionised an aspect of one big system (an operating system) which promptly got cancelled after we built the final batch of products on it :-D. In other cases I've only been able to make improvements here and there, in areas where others didn't care too much. The culture from the top has a huge influence that you cannot really counter fully - only within your own team sometimes or your own department and you have to be very careful about confronting it head on. So this is why startups work of course - because they allow change to happen :-)
• edanm 2y ago
  Fantastic article. One small point - consistency is a pretty good rule in small codebases too, for similar reasons. Less critical, maybe, but if your small codebase has a standard way of handling e.g. Auth, then you don't want to implement auth differently, for similar reasons (unified testing, there might be specialized code in the auth that handles edge cases you're not aware of, etc.)
• daitangio 2y ago
  I agree that consistency is important, and also this is the real problem. There is no perfect architecture. Needs evolve. So consistency is a force, but architecture evolution (pushed by new features, for example) is an opposite force. Balancing the two is not easy, and often if you do not have time, you are forced to drop your strong principles. Let me do a simple example. Imagine a Struts2 GUI. One day your boss ask you to do upgrade it to fancy AJAX. It is possible, for sure, but it can require a lot of effort, and finding the right solution is not easy,
• z3t4 2y ago
  If you have no idea what the usage is you are already doomed. However tests helps, whenever there is a regression you should write a test so that the same thing wont regress again. Writing code is not like in real life where herd mentally usually saves your life. Go ahead and improve the code, what helps is tests... but also at least logging errors, and throwing errors. Tests and errors go hand in hand. Errors are not your enemy, errors helps you improve the program.
• Kon5ole 2y ago
  Sometimes the right approach is to keep the consistency. Other times, that approach is either impossible or catastrophic. IMO software development is so diverse and complex that universal truths are very very rare. But to us programmers, anything that promises to simplify the neverending complexity is very tempting. We want to believe! So we're often the equivalent of Mike Tyson reading a book by Tiger Woods as we look down a half-pipe for the first time. We've won before and read books by other winners, now we're surely ready for anything! Which leads to relational data stored in couchDB, datalayers reimplemented as microservices, simple static sites hosted in kubernetes clusters, spending more time rewriting tests than new features, and so on. IMO, most advice in software development should be presented as "here's a thing that might work sometimes".
  • hhsuey 2y ago
    The same is true for stock picking.
• tsarchitect 2y ago
  OP has identified a universal norm: "Law of Large Established Codebases (LLEC)" states that "Single-digit million lines of code, Somewhere between 100 and 1000 engineers, first working version of the codebase is at least ten years old" tend to naturally dissipate, increasing the entropy of the system, inconsistency being one of characteristics. OP also states that in order to 'successfully' split a LEC you need to first understand it. He doesn't define what 'understanding the codebase' means but if you're 'fluent' enough you can be successful. My team is very fluent in successfully deploying our microfrontend without 'understanding' the monstrolith of the application. I would even go out and make the law a bit more general: any codebase will be both in a consistent and inconsistent state. If you use a framework, library, or go vanilla, the consistency would be the boilerplate, autogenerated code, and conventional patterns of the framework/library/programming language. But inconsistency naturally crops up because not all libraries follow the same patterns, not all devs understand the conventional patterns, and frameworks don't cover all use cases (entropy increases after all). Point being, being consistent is how we 'fight' against entropy, and inconsistency is a manifestation of increasing entropy. But there is nothing that states that all 'consistent' methods are the same, just that consistency exists and can be identified but not that the identified consistency is the same 'consistency'. And taking a snapshot of the whole you will always find consistent & inconsistent coexisting
• oglop 2y ago
  Coders aren’t engineers, they are just bit bureaucrats. Everything in this post isn’t engineering, it’s autism.
• roxyrox 2y ago
  Have you ever tried Source-graph ? To handle such consistency issues. (we are trying to do the same at Anyshift for Terraform code) For me the issue is only be exacerbated by gen AI and the era of "big code" thats ahead
• dktoao 2y ago
  This is good advice but only it has been followed from the beginning and consistently throughout the development of the original code. It is applicable to large organizations with lots of resources who hire professional developers and have a lot of people who are familiar with the code that are active in code reviews and have some minimum form of documentation / agreement on what the logic flow in the code should look like (the article does not claim otherwise). But I would implore those who work at the 80% of other companies that this advice is nearly useless and YMMV trying to follow it. The one thing that I think is universally good advice is to try and aggressively remove code whenever possible.
  • jowdones 2y ago
    [dead]
• zzbzq 2y ago
  Wrong, wrong. Opposite of everything he said. All his examples are backwards. The article is basically inversing the Single Responsibility Principle. First of all, consistency does not matter at all, ever. THat's his main thesis so it's already wrong. Furthermore, all his examples are backwards. If you didn't know the existence of "bot" users, you probably don't want your new auth mechanism to support them. Otherwise, the "nasty surprise" is the inverse of what he said: not that you find you don't support bot users, but you find out that you do. Build stuff that does exactly what you want it to do, nothing more. This means doing the opposite of what he said. Do not re-use legacy code with overloaded meanings.
  • pavel_lishin 2y ago
    > First of all, consistency does not matter at all, ever. THat's his main thesis so it's already wrong. Can you say more about this? Because I strongly disagree with your assertion.
    • veverkap 2y ago
      > Build stuff that does exactly what you want it to do, nothing more This is also confusing to me. In a multi-million line codebase, it's extremely difficult to find an actual place where you have zero side effects with ANYTHING you write.
  • fixprix 2y ago
    Wrong. If code is written consistently everywhere, that allows any dev to dive in anywhere to get work done. Which is what you often have to do in large code bases to make cross functional updates. Code bases where devs pick a different framework or library for every little thing are a nightmare to maintain. Agreed on standards is what gets your team out of the weeds to work on a higher and more productive level.
• nadis 2y ago
  A little buried since the overarching focus is on consistency but I found these two paragraphs from the blog post really relevant points as well: "You need to develop a good sense of how the service is used in practice (i.e. by users). Which endpoints are hit the most often? Which endpoints are the most crucial (i.e. are used by paying customers and cannot gracefully degrade)? What latency guarantees must the service obey, and what code gets run in the hot paths? One common large-codebase mistake is to make a “tiny tweak” that is unexpectedly in the hot path for a crucial flow, and thus causes a big problem. You can’t rely on your ability to test the code in development like you can in a small project. Any large project accumulates state over time (for instance, how many kinds of user do you think GMail supports?) At a certain point, you can’t test every combination of states, even with automation. Instead, you have to test the crucial paths, code defensively, and rely on slow rollouts and monitoring to catch problems."