20 ms·

In software development it's pretty important to know when to build "on top" of something else, and when to start from scratch. Lots of developers will find it

In software development it's pretty important to know when to build "on top" of something else, and when to start from scratch. Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist. They'll cleverly manipulate and convince the boss, against the better discretion of their elder developers, that they can do it, and if they're one of the better developers, the boss won't want to risk losing them so they'll agree to the escapade. Then said escapade turns into a shambles, as predicted by the elder devs, and the developer who created the mess simply quits and moves to some other job, in search of more fun and greener pastures. Any developer with decades of experience has probably seen this same pattern multiple times.

wheybags 2y ago
This is a sentiment that I've seen expressed in comment sections many times. I've been programming professionally now for 10 years, and it just doesn't resonate with my experience. Problems with build systems for external dependencies, package managers, and underfeatured / overcomplicated / buggy third party dependencies have been by far the worse issue in my career, compared to problems with homebrewed systems. I'm not saying you're wrong, I don't doubt that many people have the opposite experience. It just makes me feel a bit alien when I read comments like this.
- marcosdumay 2y ago
  I've been there, on both sides, with homebrew ideas pushed from up and down, some that worked nicely, and some that were complete disasters... And I agree with you. The problems with third party dependencies are way worse than any in-house complete disaster. But that happens almost certainly because everybody is severely biased into adding dependencies. Make people biased into NIH again, and the homebrew systems will become the largest problems again.
  - lukan 2y ago
    "because everybody is severely biased into adding dependencies" When I make a request to chatGPT to show me a example of something with javascript and node - it always brings me a solution with a external libary needed. So I have to add "without external dependencies" - then it presents me a nice and clean solution without all the garbage I don't need. So apparently adding yet another libary seems normal for many people, otherwise this behavior would not replicate in an LLM.
    - pca006132 2y ago
      This depends on languages. For c/c++ without a package manager, people hesitate to add external dependencies (and especially when high performance is needed, e.g. game engines).
      - lukan 2y ago
        It surely does, but I think in most languages it is a bad idea, to add a external dependency, where it isn't needed. Like in the cases I mentioned, it was just standard stuff, already covered by the browser standard libaries. So when I just need a simple connection to a web socket - I don't need that external baggage. But for high performance graphics, yeah, I won't write all the WebGL code by hand, but use Pixi (with the option of writing my own shader where needed).
    - OtomotO 2y ago
      It simply depends on what you need. I am gladly writing my own left-pad. I am gladly using something like three.js if I need it. The problem are the extreme stances. Don't add any dependency is as stupid as pulling in dependencies for every second line.
      - wizzwizz4 2y ago
        Please don't write your own left-pad. It's built into the standard library, under the name (String.prototype.)padStart.
- quantadev 2y ago
  This kind of thing admittedly isn't as pervasive in the last decade as it was the two before, so if you've been a dev only since 2014 years you may not have seen it. The old people like me will get it tho.
- dylan604 2y ago
  Not targeting you, but the industry in general. In every other industry I've been in outside of software dev, 10 years is not considered elder. You're just now becoming not a greenhorn. You're just now getting your sea legs. It's amazing what additional experience happens after year 10. To that effect, Rust (2015) is 9 years old, Go and Node are 15 years old. While Python (1991) is 33 years old. Just putting things in a different perspective
  - Cthulhu_ 2y ago
    It just feels different in software development because things have moved very fast, I'd say especially when github rose to prominence. The amount of software developers on the market has also increased exponentially since then, so the amount of (relatively) junior developers is much higher than those of 15, 20+ years of experience.
    - jltsiren 2y ago
      The number of software developers has maybe doubled in the last 20 years. The number of senior developers has "always" been low, because the field suffers from unusually high attrition. Many people find that software is not for them, many switch fields after losing their jobs in an economic downturn, some move to management, and some make too much money to bother continuing until retirement age.
      - mrkeen 2y ago
        Has there actually been attrition? Exponential growth is enough to explain "many more juniors than seniors" at any time in the past, present or future. Also for attrition to be the cause, you'd need a lot more seniors dropping off than juniors.
        jltsiren 2y ago
        The exponential growth has been something like 3-4%/year, or 2x in 20 years. Though it's hard to find useful statistics that take different job titles and changing nature of the industry properly into account. If you had asked me in 2010, I would have said that the median software developer lasts 5-10 years in the industry. A lot of people left the field after the dot-com bubble burst. The same happened again in a smaller scale in the late 2000s, at least in countries where the financial crisis was a real-world event (and not just something you heard about in the news). But now there has been ~15 years of sustained growth, and the ratio of mid-career developers to juniors may even be higher than usual.
- RangerScience 2y ago
  I've seen both, although rarely for either. The worse trash fires were the homebrewed systems, but maybe that's because I could dig in and see how bad they were. But I'd actually agree with you - as bad as those were, I'd rather them than a shitty 3rd party something. At least I can theoretically do something about the in-house one, and, all the ones I've seen were smaller in scope than any SaaS product.
- tazjin 2y ago
  It's people trying to generalise some rule over the wrong thing. The right thing is that, in both directions, how the project goes is simply a skill question. You have unskilled, sloppy developers? The homebrew project AND the third-party integration will turn out a mess.
- kaba0 2y ago
  Well, foreign projects communicating with each other is always ground for a mess, but this is not an either-or question. Also, your mileage may vary based on the niche you are working on - in case of, say java, the initial setup of the build system may not be "fun", but it will just work from then on.
- jesse__ 2y ago
  Strong agree here. I tend to try as hard as I can to write as much as I can in house so that when shit hits the fan, I have a great chance of being able to do something about it. Shelling out to an AST parsing library that happens to be slow? Well, shit, that sucks. Guess your compilers just slow now.
- cmrdporcupine 2y ago
  The argument is not between NIH and external deps. The argument is over needless complexity and brittle unreliable bits (which can come through either channel) vs keeping things simple. In my experience, younger developers will push both (in-house and external) directions at once, actually. Building out complex edifices with sharp corners over a maze of transitive dependencies that few understand. It's the same thing: A fantasy that a framework will solve the problem, combined with a fantasy that they can develop said framework. It's an urge we all suffer from but some of us have learned the hard way to be careful about. (And others who are great at self-promotion have been rewarded for it by naive investors and managers.) Finding simple solutions takes humility and time.
- eitally 2y ago
  I think this depends a lot on whether you're already using high level languages and lots of external libraries vs doing lower level programming using something like C/C++. I managed a large dev team in a Microsoft shop and it would never have occurred to anyone to ever create their own compiler. Even the most experienced programmers would have just continued to brute force things atop .Net's compiler until it eventually "worked". The result, combined with esoteric and poorly understood business requirements, was fragile spaghetti code few could parse for bugs or updates, but it was still several layers above the compiler. This attitude is by far the most common among "enterprise developers", and one of the big differences between people building things from preexisting building blocks vs -- as witnessed from my 8 years at Google later -- people who think they're smart enough to build everything from the ground up, and do so, using primitive blocks and custom compilers created by similarly hubristic engineers who came before them. Ymmv, but this has been my experience over the past 25 years.
  - neonsunset 2y ago
    To be fair expression trees offer nice capability to write your mini-dsl, then map it to expressions and then compile it. It’s just an uncommon attitude to most enterprise teams, it has less to do with the language and more to do with the part of the industry. I wish more teams knew the tools they already have at their disposal.
- MobiusHorizons 2y ago
  Thanks for saying this, I feel this way all the time even though I know it’s against the prevailing wisdom. My experience is that in the pursuit of not reinventing the wheel, I am frequently told to use a dependency that doesn’t allow us to solve the whole problem, or prevents us from making making the user experience fast or cannot be made to understand our data model. It’s all well and good to use a tool that exists, but using the wrong tool just because it exists is madness. Even worse is when dependencies are deprecated or our use cases become unsupported. Honestly I would prefer to just build everything above the database layer in house, that way we at least know what we can and can’t deliver, and have some chance of fixing things when they break.
  - PartiallyTyped 2y ago
    I am practically having this conversation at work. There's a sister team with a great tool for benchmarking what they are working on, but it is not convenient for our needs, and I am told to "just do the plumbing to make it work for our needs". Reality is that there are far, far easier ways to achieve what we need than doing all that plumbing, adding more layers of abstraction on top of what is a side project of an adjacent team.
  - solatic 2y ago
    When dependencies don't deliver 100% of what you want, you should be able to work with upstream to get what you want added. If upstream won't cooperate with you, then fork. It's still usually better to start from a battle-hardened codebase than it is from complete scratch.
  - anal_reactor 2y ago
    The problem with being smarter than average programmer is that your insights will rarely ever be considered, even if they're correct, because they're new and controversial. That's because, from the perspective of an average programmer, a bad programmer who doesn't know what they're doing, and a programmer using techniques so advanced that they cannot be understood, are effectively indistinguishable, which means that average team will treat both geniuses and morons in the same way. I feel like the collapse of the tech bro coincided with the masses going to programming, which changed the culture from promoting innovation and development, into simply following whatever best practices someone had already written, turning programming from a creative job into yet another repetitive office job. This is also, in my opinion, the true reason why salaries collapsed. Most business don't need creative specialists, they need code monkeys, and most people aren't creative specialists, they're code monkeys. So why would the salary worthy of a creative specialist even be talked about over here?
- rzwitserloot 2y ago
  Yes. And one of those vaunted differences between 'senior' and 'medior' is knowing the difference. Because I can confirm what you said: Both experiences are real. Brewing it up yourself can blow up in your face. Reaching for external deps that solve the problem can blow up in your face. Knowing which choice to make is _tricky_ and is hard to confirm. It doesn't sit well with programmers; either solution will _work_ (you can't write a unit test that 'fails' if you made the wrong choice here), and even if you're willing to accept highly suspect, Goodhart's law-susceptible metrics such as LOC, you still can't get anywhere because it's trading off more code you have to write and maintain without help from a larger community against having fewer lines _in total_ as part of the system. I do not know of any way to do it right other than to apply a ton of experience. And it's really hard to keep yourself honest. Even if you're willing to wait 5 years and then spend some time looking back, how do you really know? Anybody with a bunch of experience has seen enough homebrew stuff asplode in their face to be able to paint a picture with how utterly badly that choice could go. If you chose the 'build it on external deps' route you can easily tell yourself you did it right by painting a terrible picture of how it would have gone if you made the other choice. But the reverse is just as true. I think I'm really good at it. But, writing about it here, I don't have any real basis to make that claim. I look around at other dev shops that make products of similar complexity and it feels like they need 10x to 100x more resources, have more downtime, and have far larger dev teams. But no doubt bias is creeping in there too, and no 2 software products are 100% comparable in this sense. I naturally trust homebrewers more because they tend to understand complex technical things better. Someone who can just glue libraries together is lost when I ask them to fire up a debugger and figure out why some interaction is not working. A hopeless NIH sufferer needs to be 'supervised' and their choices about what to write needs to be questioned, but, that's doable with supervision. "Just git gud and be technically proficient" not so much. But then maybe that's bias too - that leads to a codebase that is easier navigated when you're familiar with debuggers and reading code to understand it. Reaching for third party deps a lot leads to a codebase that is easier navigated when you're familiar with docs and tutorials. These are self fulfilling prophecies.
  - vanitywords 2y ago
    Been in software 25 years and was board design for telecom before that. Vaunted is a vanity word. I get what you mean based upon experience and I still prefer custom systems that eschew layers of syntax sugar. EE was way harder than managing a code base. The syntax patterns are of a finite set of values. I won’t re-roll encryption libs but there’s a lot of “tooling” packages that just add syntax sugar to parse and cart around that came about in prior eras of sneakerware software that are baked into to development habits that no longer make sense. I for one am excited about using ML to streamline the code base that comprised my preferred Linux system yet still builds to the usual runtime system. There’s a lot of duplication in code that models can help remove and a few tools can help unpack into machine state. EE brain informs me there’s no “code” in a running system. Just electricity. There’s way too much syntax sugar in the software ecosystem that’s just for parsing/marshaling/transpiling between syntax sugars. Bleh. It’s a big dumb monolith of glyph art that needs to be whacked back like a prairie that needs a controlled fire.
- segfaltnh 2y ago
  I've spent most of my career in the infrastructure space and I agree with this so much. These days prevailing wisdom is just to use 20 off the shelf open source components and spend your entire day debugging YAML integrations. I think we've lost our minds a bit because of this prevailing wisdom that building a simple wheel that does the 10% of this you actually need is somehow self-indulgence or negligent or both.
chii 2y ago
> against the better discretion of their elder developers why are the junior calling the shots over the elder developers?
- quantadev 2y ago
  Lots of times it's just ordinary office politics, or the boss likes one person more than another, or isn't "technical" enough to know when he's being manipulated. Because often managers aren't developers themselves, so they don't know which developer is telling them the best advice, when two developers disagree.
  - taeric 2y ago
    And to be fair, often times it is against the senior's judgement, but if off the critical path can be a decent gamble. The hubris of junior engineers accomplishes a lot.
    - quantadev 2y ago
      Yes, it's true great developers can 'reinvent' things and do a great job of it, but the problem is that every line of code in a project is an efficiency drag forever moving forward. It always has to be maintained, updated, and managed by someone. Developers should be measured by how many lines of great code they can delete, not how many lines of great code they create (<-- but don't take this literally of course, it's just making a point)
      - taeric 2y ago
        Right. My "to be fair" is largely similar to "devil's advocate." The caveat of, "off critical path" is a heavy lift, too. My view was to give projects a form of risk budget. If possible, do things same way as last time. Any deviation is a risk. Can have rewards, sure. But if there was a known way to do it already, be budgeted to pivot back.
- dylan604 2y ago
PittleyDunkin 2y ago
> Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist. This is true; but after enough years in the industry you learn to correlate success with laziness. This is well-discussed and arguably obvious but on an emotional level it takes a long time to fully sink in. We were all once developers with outsized ambitions and awareness we can flee to greener pastures.
- quantadev 2y ago
  I've said in the past "The best developers are the laziest ones". We don't want to do a bit of unnecessary work at all. But at the same time since I spend almost ever waking hour of my spare time coding, the word lazy still isn't quite accurate in every way either.
  - mech422 2y ago
    Larry Wall would agree :-D
    - gmac 2y ago
      https://thethreevirtues.com/ https://thethreevirtues.com/
      - mech422 2y ago
        Yeah ... the Camel book was awesome. I remember being amazed how fast it got you writing really useful perl scripts.
  - zimpenfish 2y ago
    > the word lazy still isn't quite accurate in every way either. I take it in its form from "lazy loading" - "don't do it until absolutely necessary" (which kinda devolves into YAGNI.)
rectang 2y ago
> even when there are better things that already exist That's a "big if". Lots of times what's there is a nightmarish tangle of technical debt left by previous greenfield devs. The dev who gets to maintain and evolve this dreck is the sucker, scapegoated for ever slower development. Canonical example: on-call AWS engineers working hellish overtime to close tickets on one of AWS's many terribad fragile codebases.
revskill 2y ago
Is he a wheel manufacture ?
fmbb 2y ago
I have only seen this problem in elder devs. Some people simply seem to believe they are selected by god to hand out their frameworks for poor juniors to be forced to work inside. Sometimes they are just a founding engineer, they were the only devs in a startup, or one of five. These senior devs often quit or are fired and leave the rest of the developers with their “good ideas”. I have never seen a junior taking on something new that is inherently huge and complex. I have seen them go overboard with refactoring, because someone tricked them into thinking the Boy Scout rule is good, or that DRY is important, or that they need to think ahead and abstract/generalize for the future. Inevitably that is something they were “taught” by senior colleagues or teachers. A corollary to this is the pandemic of phobia for NIH. A lot of developers really seem to prefer janky, undermaintained third party libraries with huge APIs over a quick home made hack to solve exactly the problem your team has, and you can maintain and test and just know everything there is to know about. Building your own stuff is good. It is the business we are in.
- zimpenfish 2y ago
  > I have only seen this problem in elder devs. To drop an anecdatum into the fray, I am an older dev and I tend to see this from younger devs because, frankly, the elder devs are too tired and semi-burnt out from trying to stop the younger devs creating nonsensically pure houses of glass built on the Wise Words of The Bloggers using Technique du Jour where everything takes 4x as long and results in the most fragile and complex sugar-spun castles which collapse into unmaintainable slop after the first contact with enemy (customer) fire. (I may or may not be bitter about this from previous jobs)
  - FunnyLookinHat 2y ago
    Yeah, the fatigue is real. It takes much more energy to stop a bad idea / framework from taking root in a large org than to start one.
- brabel 2y ago
  I think what you're calling "elder devs" is actually the "intermediate" devs. They're not junior in any sense if they're capable/allowed to create these huge balls of mud we're referring to. And the elders normally have seen way too much to fall into that trap... and definitely don't quit often like you're describing at all (my experience is that the younger you are, the more often you change jobs - which is good for you as it's been shown this is the best way to get a good paycheck, but bad for employers, of course). They're tired of that constant churn and have had more than enough time to find a place where they're comfortable. The OP is likely talking about those as well, but from the perspective of someone who probably is truly senior and has been doing this at least since the 90's... basically, they're talking about the devs who know just enough to be dangerous (some will enter this stage from around 3 years of experience to 10, others may stay there from 5y to 20y - so it's difficult to group them together in a neat group) just like you're doing, but to you, they look senior as well.
KronisLV 2y ago
> In software development it's pretty important to know when to build "on top" of something else, and when to start from scratch. Building some brownfield CRUD for a run of the mill org? Starting from scratch will almost always go horribly, just pick whatever enterprise'y solution fits the task at hand and be done with it. Working for one of the big orgs on something interesting, and have the backing needed for being able to throw person-years at a problem until it crumbles under the collective engineering effort? Building from scratch might be a good choice sometimes. Personal learning projects, side projects and the like? If you won't have to maintain it long term or at least don't think you'll have significant amounts of time or effort you can spare for that, then from scratch is okay (your own game engine to learn about the internals? your own implementation of something S3 compatible? maybe your own CMS for the hell of it?), otherwise consider treating it as a brownfield project (e.g. if you want to make and finish a game, or just store some files, or maybe just run a blog where the focus is on the content not how you made the thing it's running on). What's my reasoning for this? Code is typically written to solve a particular problem. In business context, that typically means finishing some Jira issues and having deliverables. In large enough open source projects that typically also means having instructions on how to run and administer the thing, proper test coverage given the larger amount of various people working on it. Thus, the bus factor becomes larger and it won't be as much of a miserable experience of code archaeology as when the dev who wrote some custom CMS for a project at work leaves and literally only some code without even proper CI/CD is left behind, no proper comments, no ADRs, no code examples that aren't coupled to the logic, no documentation or even summary of the project, maybe an empty template for a README, no decoupling between the technical bits and the business rules (or just tight coupling in general), because again, they only wanted to ship. And even if they had better intentions, there were still deadlines and they were still one person (or a small team) that can't compete with any of the large multi-year projects out there.
- quantadev 2y ago
  In the last paragraph what you said is often what happens due to bad management too. A good developer can be given a task that they barely have time to get done, and as a result the unit tests, and the documentation, and even the architecture suffers, or gets omitted. Often in shops where just cranking out new features and/or bug fixing is the goal of management, the software continues to degrade endlessly due to all the things you mentioned, because spending time in those areas isn't something the boss finds justifiable expenditure of developer time. Once all the developers who originally wrote the code have left or been fired then the deterioration in code quality can start to go down rapidly until some kind of "cleanup" effort is undertaken, where ZERO new features are created, but things are just cleaned up. In projects with millions of lines of spaghetti code sometimes this cleanup is completely impossible, because a total rewrite would be easier.
maccard 2y ago
There’s a flip side to this which is building on top of which solves your problem but aren’t actually suitable. It “works”, but often at the expense of someone else. A great example is homebrew and GitHub. Or, making a shim between something that solves 80% of your problem rather than solving the problem yourself. The mark of a 10x engineer IMO is getting the build vs buy question right consistently. My experience is that teams get it wrong often in both directions
deleted 2y ago
[deleted]
wannabe44 2y ago
Often the reverse happens. People will think some half baked toy will solve the problem and it brings it's own set of challenges. https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/ https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
iveqy 2y ago
I'm working with SBoM, one fun side effect is that you can scan SBoM's for vulnerabilities. Suddenly hackers, your customers and your competitors starts do to this and you need to make sure your third party dependencies are updated. This reveals the cost of dependencies (that often are ignored). I hope that we in the future will have a more nuanced discussion on when it's okay to add a dependency and when you should write from scratch.
Evidlo 2y ago
I'm in that boat except the ages are reversed and the older guys are constantly trying to build things from scratch. They just refuse to spend any time looking for off-the-shelf solutions and only build on what they know, so we waste time and end up with a crappy result. Example: For a parallel data processing pipeline they wanted to build a REST interface for submitting "jobs" to a cluster which would parallelize with MPI, instead of just using xarray+dask. Another example: They wanted to store tabular data product metadata Postgres with URIs pointing to NetCDF files on disk, instead of just putting everything inside NetCDF.
chmod775 2y ago
> boss won't want to risk losing them so they'll agree to the escapade. That's precisely it. A motivated engineer is almost always going to outperform a bored engineer/one who quits. Morale is miles more important than chasing after efficiency. "Boss" here is likely making the correct decision after weighing the ups and downs.
faizshah 2y ago
On the flip side there are many engineers who are so afraid to build anything that seems even a little bit difficult you end up with a million dependencies for things that could have been 100 lines of code. A common thing to hear in frontend these days is “you’re just rebuilding X” “you mean you want to rebuild X” where X is some trivial flavor of the month library. All of the famous tools, databases and frameworks you see today was built by someone who said they can do it better and then they built a community around it.
lerax 2y ago
I like to build from scratch, but I prefer maintain something on top of something. Less code to maintain since I can delegate part of my code to the technology built on top of it. In that matters, for my professional work have a healthy long term life, I usually select on-top development style, and sometimes, if I have really good reasons to do, from scratch is the option. Of course, for learning and personal projects, from scratch is always a very fun choice!
entropyie 2y ago
+1 on this. Number one source of bugs at a recent job was a homebrew TLS / HTTP load balancer. First chance I got I replaced it with nginx and bugs shot down immediately. With tools like apache, nginx, haproxy and caddy available, it was pure madness to reinvent that wheel... But the dev wanted open source CV padding...

wheybags 2y ago

This is a sentiment that I've seen expressed in comment sections many times. I've been programming professionally now for 10 years, and it just doesn't resonate with my experience. Problems with build systems for external dependencies, package managers, and underfeatured / overcomplicated / buggy third party dependencies have been by far the worse issue in my career, compared to problems with homebrewed systems. I'm not saying you're wrong, I don't doubt that many people have the opposite experience. It just makes me feel a bit alien when I read comments like this.

marcosdumay 2y ago
I've been there, on both sides, with homebrew ideas pushed from up and down, some that worked nicely, and some that were complete disasters... And I agree with you. The problems with third party dependencies are way worse than any in-house complete disaster. But that happens almost certainly because everybody is severely biased into adding dependencies. Make people biased into NIH again, and the homebrew systems will become the largest problems again.
- lukan 2y ago
  "because everybody is severely biased into adding dependencies" When I make a request to chatGPT to show me a example of something with javascript and node - it always brings me a solution with a external libary needed. So I have to add "without external dependencies" - then it presents me a nice and clean solution without all the garbage I don't need. So apparently adding yet another libary seems normal for many people, otherwise this behavior would not replicate in an LLM.
  - pca006132 2y ago
    This depends on languages. For c/c++ without a package manager, people hesitate to add external dependencies (and especially when high performance is needed, e.g. game engines).
    - lukan 2y ago
      It surely does, but I think in most languages it is a bad idea, to add a external dependency, where it isn't needed. Like in the cases I mentioned, it was just standard stuff, already covered by the browser standard libaries. So when I just need a simple connection to a web socket - I don't need that external baggage. But for high performance graphics, yeah, I won't write all the WebGL code by hand, but use Pixi (with the option of writing my own shader where needed).
  - OtomotO 2y ago
    It simply depends on what you need. I am gladly writing my own left-pad. I am gladly using something like three.js if I need it. The problem are the extreme stances. Don't add any dependency is as stupid as pulling in dependencies for every second line.
    - wizzwizz4 2y ago
      Please don't write your own left-pad. It's built into the standard library, under the name (String.prototype.)padStart.
quantadev 2y ago
This kind of thing admittedly isn't as pervasive in the last decade as it was the two before, so if you've been a dev only since 2014 years you may not have seen it. The old people like me will get it tho.
dylan604 2y ago
Not targeting you, but the industry in general. In every other industry I've been in outside of software dev, 10 years is not considered elder. You're just now becoming not a greenhorn. You're just now getting your sea legs. It's amazing what additional experience happens after year 10. To that effect, Rust (2015) is 9 years old, Go and Node are 15 years old. While Python (1991) is 33 years old. Just putting things in a different perspective
- Cthulhu_ 2y ago
  It just feels different in software development because things have moved very fast, I'd say especially when github rose to prominence. The amount of software developers on the market has also increased exponentially since then, so the amount of (relatively) junior developers is much higher than those of 15, 20+ years of experience.
  - jltsiren 2y ago
    The number of software developers has maybe doubled in the last 20 years. The number of senior developers has "always" been low, because the field suffers from unusually high attrition. Many people find that software is not for them, many switch fields after losing their jobs in an economic downturn, some move to management, and some make too much money to bother continuing until retirement age.
    - mrkeen 2y ago
      Has there actually been attrition? Exponential growth is enough to explain "many more juniors than seniors" at any time in the past, present or future. Also for attrition to be the cause, you'd need a lot more seniors dropping off than juniors.
      - jltsiren 2y ago
        The exponential growth has been something like 3-4%/year, or 2x in 20 years. Though it's hard to find useful statistics that take different job titles and changing nature of the industry properly into account. If you had asked me in 2010, I would have said that the median software developer lasts 5-10 years in the industry. A lot of people left the field after the dot-com bubble burst. The same happened again in a smaller scale in the late 2000s, at least in countries where the financial crisis was a real-world event (and not just something you heard about in the news). But now there has been ~15 years of sustained growth, and the ratio of mid-career developers to juniors may even be higher than usual.
RangerScience 2y ago
I've seen both, although rarely for either. The worse trash fires were the homebrewed systems, but maybe that's because I could dig in and see how bad they were. But I'd actually agree with you - as bad as those were, I'd rather them than a shitty 3rd party something. At least I can theoretically do something about the in-house one, and, all the ones I've seen were smaller in scope than any SaaS product.
tazjin 2y ago
It's people trying to generalise some rule over the wrong thing. The right thing is that, in both directions, how the project goes is simply a skill question. You have unskilled, sloppy developers? The homebrew project AND the third-party integration will turn out a mess.
kaba0 2y ago
Well, foreign projects communicating with each other is always ground for a mess, but this is not an either-or question. Also, your mileage may vary based on the niche you are working on - in case of, say java, the initial setup of the build system may not be "fun", but it will just work from then on.
jesse__ 2y ago
Strong agree here. I tend to try as hard as I can to write as much as I can in house so that when shit hits the fan, I have a great chance of being able to do something about it. Shelling out to an AST parsing library that happens to be slow? Well, shit, that sucks. Guess your compilers just slow now.
cmrdporcupine 2y ago
The argument is not between NIH and external deps. The argument is over needless complexity and brittle unreliable bits (which can come through either channel) vs keeping things simple. In my experience, younger developers will push both (in-house and external) directions at once, actually. Building out complex edifices with sharp corners over a maze of transitive dependencies that few understand. It's the same thing: A fantasy that a framework will solve the problem, combined with a fantasy that they can develop said framework. It's an urge we all suffer from but some of us have learned the hard way to be careful about. (And others who are great at self-promotion have been rewarded for it by naive investors and managers.) Finding simple solutions takes humility and time.
eitally 2y ago
I think this depends a lot on whether you're already using high level languages and lots of external libraries vs doing lower level programming using something like C/C++. I managed a large dev team in a Microsoft shop and it would never have occurred to anyone to ever create their own compiler. Even the most experienced programmers would have just continued to brute force things atop .Net's compiler until it eventually "worked". The result, combined with esoteric and poorly understood business requirements, was fragile spaghetti code few could parse for bugs or updates, but it was still several layers above the compiler. This attitude is by far the most common among "enterprise developers", and one of the big differences between people building things from preexisting building blocks vs -- as witnessed from my 8 years at Google later -- people who think they're smart enough to build everything from the ground up, and do so, using primitive blocks and custom compilers created by similarly hubristic engineers who came before them. Ymmv, but this has been my experience over the past 25 years.
- neonsunset 2y ago
  To be fair expression trees offer nice capability to write your mini-dsl, then map it to expressions and then compile it. It’s just an uncommon attitude to most enterprise teams, it has less to do with the language and more to do with the part of the industry. I wish more teams knew the tools they already have at their disposal.
MobiusHorizons 2y ago
Thanks for saying this, I feel this way all the time even though I know it’s against the prevailing wisdom. My experience is that in the pursuit of not reinventing the wheel, I am frequently told to use a dependency that doesn’t allow us to solve the whole problem, or prevents us from making making the user experience fast or cannot be made to understand our data model. It’s all well and good to use a tool that exists, but using the wrong tool just because it exists is madness. Even worse is when dependencies are deprecated or our use cases become unsupported. Honestly I would prefer to just build everything above the database layer in house, that way we at least know what we can and can’t deliver, and have some chance of fixing things when they break.
- PartiallyTyped 2y ago
  I am practically having this conversation at work. There's a sister team with a great tool for benchmarking what they are working on, but it is not convenient for our needs, and I am told to "just do the plumbing to make it work for our needs". Reality is that there are far, far easier ways to achieve what we need than doing all that plumbing, adding more layers of abstraction on top of what is a side project of an adjacent team.
- solatic 2y ago
  When dependencies don't deliver 100% of what you want, you should be able to work with upstream to get what you want added. If upstream won't cooperate with you, then fork. It's still usually better to start from a battle-hardened codebase than it is from complete scratch.
- anal_reactor 2y ago
  The problem with being smarter than average programmer is that your insights will rarely ever be considered, even if they're correct, because they're new and controversial. That's because, from the perspective of an average programmer, a bad programmer who doesn't know what they're doing, and a programmer using techniques so advanced that they cannot be understood, are effectively indistinguishable, which means that average team will treat both geniuses and morons in the same way. I feel like the collapse of the tech bro coincided with the masses going to programming, which changed the culture from promoting innovation and development, into simply following whatever best practices someone had already written, turning programming from a creative job into yet another repetitive office job. This is also, in my opinion, the true reason why salaries collapsed. Most business don't need creative specialists, they need code monkeys, and most people aren't creative specialists, they're code monkeys. So why would the salary worthy of a creative specialist even be talked about over here?
rzwitserloot 2y ago
Yes. And one of those vaunted differences between 'senior' and 'medior' is knowing the difference. Because I can confirm what you said: Both experiences are real. Brewing it up yourself can blow up in your face. Reaching for external deps that solve the problem can blow up in your face. Knowing which choice to make is _tricky_ and is hard to confirm. It doesn't sit well with programmers; either solution will _work_ (you can't write a unit test that 'fails' if you made the wrong choice here), and even if you're willing to accept highly suspect, Goodhart's law-susceptible metrics such as LOC, you still can't get anywhere because it's trading off more code you have to write and maintain without help from a larger community against having fewer lines _in total_ as part of the system. I do not know of any way to do it right other than to apply a ton of experience. And it's really hard to keep yourself honest. Even if you're willing to wait 5 years and then spend some time looking back, how do you really know? Anybody with a bunch of experience has seen enough homebrew stuff asplode in their face to be able to paint a picture with how utterly badly that choice could go. If you chose the 'build it on external deps' route you can easily tell yourself you did it right by painting a terrible picture of how it would have gone if you made the other choice. But the reverse is just as true. I think I'm really good at it. But, writing about it here, I don't have any real basis to make that claim. I look around at other dev shops that make products of similar complexity and it feels like they need 10x to 100x more resources, have more downtime, and have far larger dev teams. But no doubt bias is creeping in there too, and no 2 software products are 100% comparable in this sense. I naturally trust homebrewers more because they tend to understand complex technical things better. Someone who can just glue libraries together is lost when I ask them to fire up a debugger and figure out why some interaction is not working. A hopeless NIH sufferer needs to be 'supervised' and their choices about what to write needs to be questioned, but, that's doable with supervision. "Just git gud and be technically proficient" not so much. But then maybe that's bias too - that leads to a codebase that is easier navigated when you're familiar with debuggers and reading code to understand it. Reaching for third party deps a lot leads to a codebase that is easier navigated when you're familiar with docs and tutorials. These are self fulfilling prophecies.
- vanitywords 2y ago
  Been in software 25 years and was board design for telecom before that. Vaunted is a vanity word. I get what you mean based upon experience and I still prefer custom systems that eschew layers of syntax sugar. EE was way harder than managing a code base. The syntax patterns are of a finite set of values. I won’t re-roll encryption libs but there’s a lot of “tooling” packages that just add syntax sugar to parse and cart around that came about in prior eras of sneakerware software that are baked into to development habits that no longer make sense. I for one am excited about using ML to streamline the code base that comprised my preferred Linux system yet still builds to the usual runtime system. There’s a lot of duplication in code that models can help remove and a few tools can help unpack into machine state. EE brain informs me there’s no “code” in a running system. Just electricity. There’s way too much syntax sugar in the software ecosystem that’s just for parsing/marshaling/transpiling between syntax sugars. Bleh. It’s a big dumb monolith of glyph art that needs to be whacked back like a prairie that needs a controlled fire.
segfaltnh 2y ago
I've spent most of my career in the infrastructure space and I agree with this so much. These days prevailing wisdom is just to use 20 off the shelf open source components and spend your entire day debugging YAML integrations. I think we've lost our minds a bit because of this prevailing wisdom that building a simple wheel that does the 10% of this you actually need is somehow self-indulgence or negligent or both.

chii 2y ago

> against the better discretion of their elder developers why are the junior calling the shots over the elder developers?

quantadev 2y ago
Lots of times it's just ordinary office politics, or the boss likes one person more than another, or isn't "technical" enough to know when he's being manipulated. Because often managers aren't developers themselves, so they don't know which developer is telling them the best advice, when two developers disagree.
- taeric 2y ago
  And to be fair, often times it is against the senior's judgement, but if off the critical path can be a decent gamble. The hubris of junior engineers accomplishes a lot.
  - quantadev 2y ago
    Yes, it's true great developers can 'reinvent' things and do a great job of it, but the problem is that every line of code in a project is an efficiency drag forever moving forward. It always has to be maintained, updated, and managed by someone. Developers should be measured by how many lines of great code they can delete, not how many lines of great code they create (<-- but don't take this literally of course, it's just making a point)
    - taeric 2y ago
      Right. My "to be fair" is largely similar to "devil's advocate." The caveat of, "off critical path" is a heavy lift, too. My view was to give projects a form of risk budget. If possible, do things same way as last time. Any deviation is a risk. Can have rewards, sure. But if there was a known way to do it already, be budgeted to pivot back.
dylan604 2y ago

PittleyDunkin 2y ago

> Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist. This is true; but after enough years in the industry you learn to correlate success with laziness. This is well-discussed and arguably obvious but on an emotional level it takes a long time to fully sink in. We were all once developers with outsized ambitions and awareness we can flee to greener pastures.

quantadev 2y ago
I've said in the past "The best developers are the laziest ones". We don't want to do a bit of unnecessary work at all. But at the same time since I spend almost ever waking hour of my spare time coding, the word lazy still isn't quite accurate in every way either.
- mech422 2y ago
  Larry Wall would agree :-D
  - gmac 2y ago
    https://thethreevirtues.com/ https://thethreevirtues.com/
    - mech422 2y ago
      Yeah ... the Camel book was awesome. I remember being amazed how fast it got you writing really useful perl scripts.
- zimpenfish 2y ago
  > the word lazy still isn't quite accurate in every way either. I take it in its form from "lazy loading" - "don't do it until absolutely necessary" (which kinda devolves into YAGNI.)

rectang 2y ago

> even when there are better things that already exist That's a "big if". Lots of times what's there is a nightmarish tangle of technical debt left by previous greenfield devs. The dev who gets to maintain and evolve this dreck is the sucker, scapegoated for ever slower development. Canonical example: on-call AWS engineers working hellish overtime to close tickets on one of AWS's many terribad fragile codebases.

revskill 2y ago

Is he a wheel manufacture ?

fmbb 2y ago

I have only seen this problem in elder devs. Some people simply seem to believe they are selected by god to hand out their frameworks for poor juniors to be forced to work inside. Sometimes they are just a founding engineer, they were the only devs in a startup, or one of five. These senior devs often quit or are fired and leave the rest of the developers with their “good ideas”. I have never seen a junior taking on something new that is inherently huge and complex. I have seen them go overboard with refactoring, because someone tricked them into thinking the Boy Scout rule is good, or that DRY is important, or that they need to think ahead and abstract/generalize for the future. Inevitably that is something they were “taught” by senior colleagues or teachers. A corollary to this is the pandemic of phobia for NIH. A lot of developers really seem to prefer janky, undermaintained third party libraries with huge APIs over a quick home made hack to solve exactly the problem your team has, and you can maintain and test and just know everything there is to know about. Building your own stuff is good. It is the business we are in.

zimpenfish 2y ago
> I have only seen this problem in elder devs. To drop an anecdatum into the fray, I am an older dev and I tend to see this from younger devs because, frankly, the elder devs are too tired and semi-burnt out from trying to stop the younger devs creating nonsensically pure houses of glass built on the Wise Words of The Bloggers using Technique du Jour where everything takes 4x as long and results in the most fragile and complex sugar-spun castles which collapse into unmaintainable slop after the first contact with enemy (customer) fire. (I may or may not be bitter about this from previous jobs)
- FunnyLookinHat 2y ago
  Yeah, the fatigue is real. It takes much more energy to stop a bad idea / framework from taking root in a large org than to start one.
brabel 2y ago
I think what you're calling "elder devs" is actually the "intermediate" devs. They're not junior in any sense if they're capable/allowed to create these huge balls of mud we're referring to. And the elders normally have seen way too much to fall into that trap... and definitely don't quit often like you're describing at all (my experience is that the younger you are, the more often you change jobs - which is good for you as it's been shown this is the best way to get a good paycheck, but bad for employers, of course). They're tired of that constant churn and have had more than enough time to find a place where they're comfortable. The OP is likely talking about those as well, but from the perspective of someone who probably is truly senior and has been doing this at least since the 90's... basically, they're talking about the devs who know just enough to be dangerous (some will enter this stage from around 3 years of experience to 10, others may stay there from 5y to 20y - so it's difficult to group them together in a neat group) just like you're doing, but to you, they look senior as well.

KronisLV 2y ago

> In software development it's pretty important to know when to build "on top" of something else, and when to start from scratch. Building some brownfield CRUD for a run of the mill org? Starting from scratch will almost always go horribly, just pick whatever enterprise'y solution fits the task at hand and be done with it. Working for one of the big orgs on something interesting, and have the backing needed for being able to throw person-years at a problem until it crumbles under the collective engineering effort? Building from scratch might be a good choice sometimes. Personal learning projects, side projects and the like? If you won't have to maintain it long term or at least don't think you'll have significant amounts of time or effort you can spare for that, then from scratch is okay (your own game engine to learn about the internals? your own implementation of something S3 compatible? maybe your own CMS for the hell of it?), otherwise consider treating it as a brownfield project (e.g. if you want to make and finish a game, or just store some files, or maybe just run a blog where the focus is on the content not how you made the thing it's running on). What's my reasoning for this? Code is typically written to solve a particular problem. In business context, that typically means finishing some Jira issues and having deliverables. In large enough open source projects that typically also means having instructions on how to run and administer the thing, proper test coverage given the larger amount of various people working on it. Thus, the bus factor becomes larger and it won't be as much of a miserable experience of code archaeology as when the dev who wrote some custom CMS for a project at work leaves and literally only some code without even proper CI/CD is left behind, no proper comments, no ADRs, no code examples that aren't coupled to the logic, no documentation or even summary of the project, maybe an empty template for a README, no decoupling between the technical bits and the business rules (or just tight coupling in general), because again, they only wanted to ship. And even if they had better intentions, there were still deadlines and they were still one person (or a small team) that can't compete with any of the large multi-year projects out there.

quantadev 2y ago
In the last paragraph what you said is often what happens due to bad management too. A good developer can be given a task that they barely have time to get done, and as a result the unit tests, and the documentation, and even the architecture suffers, or gets omitted. Often in shops where just cranking out new features and/or bug fixing is the goal of management, the software continues to degrade endlessly due to all the things you mentioned, because spending time in those areas isn't something the boss finds justifiable expenditure of developer time. Once all the developers who originally wrote the code have left or been fired then the deterioration in code quality can start to go down rapidly until some kind of "cleanup" effort is undertaken, where ZERO new features are created, but things are just cleaned up. In projects with millions of lines of spaghetti code sometimes this cleanup is completely impossible, because a total rewrite would be easier.

maccard 2y ago

There’s a flip side to this which is building on top of which solves your problem but aren’t actually suitable. It “works”, but often at the expense of someone else. A great example is homebrew and GitHub. Or, making a shim between something that solves 80% of your problem rather than solving the problem yourself. The mark of a 10x engineer IMO is getting the build vs buy question right consistently. My experience is that teams get it wrong often in both directions

deleted 2y ago

[deleted]

wannabe44 2y ago

Often the reverse happens. People will think some half baked toy will solve the problem and it brings it's own set of challenges. https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/ https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...

iveqy 2y ago

I'm working with SBoM, one fun side effect is that you can scan SBoM's for vulnerabilities. Suddenly hackers, your customers and your competitors starts do to this and you need to make sure your third party dependencies are updated. This reveals the cost of dependencies (that often are ignored). I hope that we in the future will have a more nuanced discussion on when it's okay to add a dependency and when you should write from scratch.

Evidlo 2y ago

I'm in that boat except the ages are reversed and the older guys are constantly trying to build things from scratch. They just refuse to spend any time looking for off-the-shelf solutions and only build on what they know, so we waste time and end up with a crappy result. Example: For a parallel data processing pipeline they wanted to build a REST interface for submitting "jobs" to a cluster which would parallelize with MPI, instead of just using xarray+dask. Another example: They wanted to store tabular data product metadata Postgres with URIs pointing to NetCDF files on disk, instead of just putting everything inside NetCDF.

chmod775 2y ago

> boss won't want to risk losing them so they'll agree to the escapade. That's precisely it. A motivated engineer is almost always going to outperform a bored engineer/one who quits. Morale is miles more important than chasing after efficiency. "Boss" here is likely making the correct decision after weighing the ups and downs.

faizshah 2y ago

On the flip side there are many engineers who are so afraid to build anything that seems even a little bit difficult you end up with a million dependencies for things that could have been 100 lines of code. A common thing to hear in frontend these days is “you’re just rebuilding X” “you mean you want to rebuild X” where X is some trivial flavor of the month library. All of the famous tools, databases and frameworks you see today was built by someone who said they can do it better and then they built a community around it.

lerax 2y ago

I like to build from scratch, but I prefer maintain something on top of something. Less code to maintain since I can delegate part of my code to the technology built on top of it. In that matters, for my professional work have a healthy long term life, I usually select on-top development style, and sometimes, if I have really good reasons to do, from scratch is the option. Of course, for learning and personal projects, from scratch is always a very fun choice!

entropyie 2y ago

+1 on this. Number one source of bugs at a recent job was a homebrew TLS / HTTP load balancer. First chance I got I replaced it with nginx and bugs shot down immediately. With tools like apache, nginx, haproxy and caddy available, it was pure madness to reinvent that wheel... But the dev wanted open source CV padding...