9 ms·

Ask HN: Why did back-end development explode in complexity?

• amalgamated_inc 4y ago
  I think it's artificial, and the reasons are ego inflation and job security. Nobody wants to admit that all their abstractions and knowledge are made-up and net-negative. But they largely are.
• quaintdev 4y ago
  I think the backend development is still simple it's the deployment that has grown overly complex with Kubernetes/Docker and CI/CD pipelines
  • veb 4y ago
    hard agree. then all the choices of cloud servers, learning those, getting up to speed and actually being productive. it's good there's DevOps, but... yeah good luck explaining to pretty much any business without one that they need 'em. > "Why can't you do it?"
• birdymcbird 4y ago
  worked at two big tech companies. there is perverse incentive to grow your organization. gets you to director or VP. if you engineer it goes to your promotion packet to get you promoted. pattern i saw is teams do not build customer feature but work on “efficiency” or “cost saving” project. Ok, so you build something that other teams should use because it will save them time. Sounds very good on paper. so you do these things to entrench your team DEEP into the tech stack. now people depend on you you use it as reason to get even more head count. ok now the engineers got their promotion but they struggle to support these teams when things not working. So they leave. they got promo, why stay? managers now use all this ammo to ask for even more hc. it is self serving cycle. Now..development harder. harder to push new features. new people from college come. first they have imposter syndrome and think this is all great and they just not smart to get it. within few year they get it..see processes and tools are idiotic. they learn to play the same game. circle of life continue
  • Nextgrid 4y ago
    This is also encouraged by vendors. Old-school "boring" solutions (that you can't rent-seek over) are shunned in favour of proprietary reinventions of the wheel that do the same but poorly or more expensive. This in turn incentivises "engineers" to do resume-driven-development and implement those suboptimal products/services at their own company, and the cycle repeats.
• readonthegoapp 4y ago
  i think a couple of reasons: - 1. Nodejs - 2. Nodejs
  • Raed667 4y ago
    I have seen much more unnecessary complexity from backend developers shoehorning GoLang microservices, kafka and k8s... Than from those who create Express.js apps
• draw_down 4y ago
  [dead]
• potamic 4y ago
  Oh you sweet child, you haven't seen complexity until you have deployed J2EE services generated using WSDL, running in an OSGi container, interfaced with EJBs packaged using Ant and deployed into GlassFish, integrated with IIS running on a Windows Server. Try finding a starter kit for that.
  • croo 4y ago
    Right? We already over the explosion. Backend development is not complex, it's mature. With spring boot + initializer + bean based configuration, REST interfaces everywhere, maven and gradle are both great what's not to like?
    • isbvhodnvemrwvn 4y ago
      Can you tell me exactly what kind of beans you have in your simple spring boot app, and how request processing works step by step?
  • dham 4y ago
    I did Java EE, Glassfish, JSF/IceFaces for my first job and at the same time I could have ftp'd a PHP/Perl script on a free host and called it a day (had I known better). I think Devops has a played a huge role in why things have gotten more complicated. They want Kubernetes to be the solution to all problems. Heroku has fallen out of fashion for literally no reason. They won't even consider ElasticBeanstalk or App runner for even the smallest of projects. Companies would rather hire a 200k plus benefits person instead of paying a few hundred dollars more a month on hosted solutions. I've honestly never seen anything like it.
    • halfmatthalfcat 4y ago
      You're oversimplifying the problem and/or not viewing the problem for the organization. Maybe for an individual developer it's easier to create a Dyno or a Beanstalk deployment, but it's not necessarily easier for the organization. IMO, Kubernetes solves a lot of issues for both camps but most of the Kubernetes detractors I've met are only hostile to it because of ignorance, not any reasonable criticisms. "It's different, it's hard or it's something I don't know" aren't valid excuses.
• mr90210 4y ago
  Try Frontend development!
• mathverse 4y ago
  Backend development? Have you seen what SRE/DevOps people have to deal with? People literally use hundreds of different tools for no reason at all.
  • nodra 4y ago
    > People literally use hundreds of different tools for no reason at all. Slightly Disagree. Resume driven development (RDD) is a thing.
• SkyPuncher 4y ago
  BE is simultaneously easier than ever and harder than ever. The "new" complexity in the BE is stuff that has always been there. However, in the past, the infrastructure required so much manual work that it was handled completely independently from the BE. Now, I can write a BE, configure a Dockerfile, and deploy to a server nearly anywhere in the world without leaving my chair.
• jmfldn 4y ago
  Follow up question. Did backend development explode in complexity? What's your source?
• fxtentacle 4y ago
  I don't think it has. We just have a fashion trend where people love to brag about managing 100 micro-services using Kubernetes. But there's still companies out there where the entire back-end is a single binary on a single beefy server, potentially with a hot spare server running alongside it.
  • Nextgrid 4y ago
    Would you happen to know where I can find such companies?
    • SpeedilyDamage 4y ago
      What, so you could DDoS them? Heh...
    • deleted 4y ago
      [deleted]
    • chinabot 4y ago
      Most government entities, big banks and companies using mainframes and AS/400 systems
      • vba616 4y ago
        There are mainframes & COBOL still around, but speaking as someone who's never worked in the sort of place that uses the trendy things mentioned on HN, I would say Oracle and things built on Oracle are my concept of "normal". I don't know how decision makers think, but I get a general sense that Oracle is the safe "you can't be fired for choosing..." choice. I don't think these places have just one server, though, far from it.
• jmclnx 4y ago
  sarbanes oxley, other reporting requirements. As systems become faster, companies need to save more data. Where I work, we need to be able to reproduce finance reports created years ago using the original data. Doing that makes things far more complex. When I started out decades ago, systems were far less complex. Note, the IRS is our best friend. Without them, most of us would be doing something else (if we are very lucky).
• javajosh 4y ago
  The scope of back-end development has grown considerably. In fact, it's precisely "one level of indirection" bigger: "commit is the new save", "push is the new commit", and "repo is the new module". Plus, "full-stack development" now includes containers and orchestration! Which is, of course, insane: distributed systems are, in general, not well understood. Expecting people to understand front-end, back-end, and then just tacking on an academically challenging area is beyond foolish. I strongly believe that we're now at the limit of human cognitive ability (as evidenced by the tight, and tightening, employment market) and we human programmers are painting ourselves into a complexity corner and we will weep when our employers happily hire the only minds capable of dealing with mechanical complexity, artificial ones.
  • pluc 4y ago
    I have never met a full-stack developer that is comfortable with the full stack. I am also pretty sure full-stack was coined by recruiters and marketing folks to create a technical middle class.
    • javajosh 4y ago
      Hmm. Yes. https://dev.to/cubiclebuddha/i-m-sorry-but-this-full-stack-meme-makes-me-really-mad-sad-35hi https://dev.to/cubiclebuddha/i-m-sorry-but-this-full-stack-m...
    • chinabot 4y ago
      I dont even have a clue what a full-stack even means these days, its just a resume wank-word somewhere between "cloud" and "framework"
      • hnrodey 4y ago
        "Full stack developer" is supposed to mean someone who will just figure shit out irrespective of where the bug/enhancement resides. I say this, half tongue-in-cheek.
      • jacquesm 4y ago
• smsm42 4y ago
  I think a lot of things actually became easier lately. Deployment? Docker, k8s and cheap virtualization made it much easier than before. Exchanging data and interoperation? Finally people are starting to use comprehensible things like REST and GRPC, instead of trying to invent some byzantine "Enterprise Solution". C/Assembly is no longer the only option if you want performance. Sure, there are more tools around now, but I think the tools themselves are getting somewhat better and more sane.
• SpeedilyDamage 4y ago
  I really don't think this is a productive question or will produce productive answers that aren't tribal and arbitrary. Some folks will blame "the system" or "the man", others will claim it's not hard and you just need to RTFM, still others will explain how much harder it was "back in their day", and some folks will question the question (like me). But nobody is going to drop some single piece of information or knowledge that'll "answer" the question; "In 2004 Linus Torvalds declared that all backend systems must contain three additional technologies than the currently mandated two." If only it were that simple. Things are hard until they aren't, complex until they're simple, convoluted until they're straightforward as can be. The line is different for everyone, so nobody knows what your line is, which means nobody's going to answer the question in a way that will satisfy you, and if they do, it'll only be helpful to a few folks (your folks) and possibly actively harmful to others. Maybe if you had a specific question that illuminated where you're coming from, folks could provide better answers...
  • nwienert 4y ago
    Lovely answer, should be cross posted to the frontend thread :)
    • SpeedilyDamage 4y ago
      I see your edit, fair enough! I tend to agree with you; stuff that felt impossible for me to understand at one point is now second nature to me, and over time I've learned how to do more. UI and BE, as well as orchestration and infra. AWS used to scared me, and now I maintain a dozen microservices (and I may slightly regret breaking everything up lmao) for my startup in AWS by myself because Terraform is pretty easy to pick up and at this point AWS makes a lot of sense (hello Dunning-Kruger, my oldest colleague). Frontend terrified me because I didn't know how to compose UI cleanly (not just JS), but then I learned React and now feel confident I can build, maintain, and deploy complex frontends because honestly it wasn't that hard to grok. The single strongest piece of advice I've been given, the single idea that gets me through like... 99% of work related strife is this: How you feel now is not permanent and will not last forever. I guess the more poetic way people say this is, "This too shall pass"[0] but I just think of Gandalf so... [0] https://en.wikipedia.org/wiki/This_too_shall_pass https://en.wikipedia.org/wiki/This_too_shall_pass
• skatanski 4y ago
  Maybe it's a general comment to both back-end and front-end. To me it seems that down in the trenches coding is getting more complex, even though frameworks hide more and more complexity. Languages and frameworks gain features and they don't necessarily shed old ones. I've been working in C# for near 15 years. Each C# and .net release brings more to the table. Often these are features, which simplify things, at the same time the old ways still work. Googling often returns N solutions, each from a different era. I imagine it can be confusing for newcomers to even start. As for the growing complexity in the world of backend, I think, when needed, the best tool against it, is pragmatism.
• orangesite 4y ago
  Best case complexity is not discouraged. Worst case complexity is incentivized in more ways than this comment has room for. We live in a society.
• pacifika 4y ago
  Because managing Linux servers is now less complex than orchestrating containers. But people didn’t want to do the former.
• jayar95 4y ago
  The products we're building are more complex than they used to be, but I don't know if its true that backend development has inherently become more complex than it was in the past. I've worked on everything from dinosaurs older than me to new greenfield projects using all the latest tech. I think over-complexity emerges in many ways, notably: too much abstraction, ignorance, dogma, and just straight up carelessness. I think the abstraction problem is the most common and severe though. In enterprise software, I've spent an entire day in the past combing through several layers of services and repositories just to get the full picture of how a response object is being created for a single controller method. Let me give another example: think about the great debate over ORM vs No-ORM. It's fundamentally a battle about the balances and costs of abstraction in software development. On one hand, you have a (hopefully) simple interface for defining a db's schema, but do I really want to obfuscate the developer from the implementation details of that `findBy` method and trust that the ORM will be 1) used correctly and 2) generate a performant, sensible query? What about security handling? Should I have my developers concerned about preventing potential SQL injections, or just let the ORM take care of that too? The answer depends on a lot of things. Anyway, I believe for every abstraction: - something significantly useful should be added (this seems like a truism as I write this, but i've seen a startling amount of useless abstractions) - the cost should be carefully calculated - understand where complexity is being added and subtracted I believe thinking about abstraction in this way would've cut down a lot of the complexity I've seen in softwares past.
• yamtaddle 4y ago
  To respond to OP's edit: frontend catches more shit because it's largely gotten worse at the same time as getting more complex. This isn't (only) outsiders looking in leveling that criticism—it's people who worked on GUI software in the 90s looking at the state of things and wondering why it's both constantly changing and, in many ways, worse than it was back then. Slower, worse tools producing slower, worse software, while constantly demanding that frontend developers run on a churn-treadmill for—it sure seems—no good reason. Backend suffers from some similar issues, but not to the same degree. Frontend seems to be subject not just to the effects of the apparently-eternal fat/thin client cycle, as the backend also is, but to some other pressures that lead it to be constantly changing but not really getting better, and even getting worse over time.
  • marcosdumay 4y ago
    > frontend catches more shit because it's largely gotten worse at the same time as getting more complex. The "microservices by fad" based architecture usually comes with a huge decrease in performance and an even large one in reliability. The NoSQL for no reason fad came with horrible performance and some very bad data losses. Every once in a while we get some data leakage story here, certainly many are due to complexity (but nobody can tell which ones). Frontend is not alone suffering for the complexity.
    • yamtaddle 4y ago
      > The "microservices by fad" based architecture usually comes with a huge decrease in performance and an even large one in reliability. Good point, that one was a big mess. > The NoSQL for no reason fad came with horrible performance and some very bad data losses. 100% agreed. My perception at the time, though, was that it was largely driven by frontend folks who'd come to the backend via NodeJS buying (especially) MongoDB's marketing hype. There was a real "finally, all those barriers the backend folks had for no reason whatsoever are gone and this is easy now—better, even, you dinosaur, have you seen the marketing, it makes it so clear that this is the future—so I can be full-stack without having to learn much!" vibe to the whole thing. But maybe that's just what I happened to see and that wasn't really the main cause of that going as far as it did.
• satvikpendem 4y ago
  It is almost as if complexity is increasing regardless of the software field one is in. Maybe it's due to the fact that engineers exist to build complexity, stacking layers is what we do day to day.
• 1270018080 4y ago
  The two biggest wrenches that get thrown into a backend system and blow up complexity in my experience: - 100x in growth, aka scaling. - Needing to set up reporting data pipelines. Metrics, auditing, 3rd party uses. Does the frontend need to worry about either of these? Maybe for scaling, they need to do some CDN work, or is that still the backend developer's job?
  • chinabot 4y ago
    Scaling is over-rated and over-sold to most businesses who generally have predictable, stable or no growth over their lifetimes and then see their new system running at 1% capacity for its lifetime.
    • 1270018080 4y ago
      I agree. But I've worked at startups and it's been a real issue that comes up when the business really starts to grow.
• amarant 4y ago
  Because development jobs pays too damn well, and are too easy to get. Someone famous once said something along the lines of "any idiot can make a thing more complicated, more complex, but it takes the stroke of genius to go in the opposite direction" Basically, backend dev is so complicated because it's done by fools. When it's not done by fools, it's either not complex or has some very difficult requirements, perhaps sometimes both. Edit to add, I say this as a foolish backend Dev myself, no offence meant to my own kin, of course!
• JanneVee 4y ago
  Because everyone is prentending that they need to be infinetly scalable like the FAANGs, exacerbated by Cloud providers marketing spiel. In reality nobody knows how to build distributed systems but everybody needs to build them as resume fodder, because they might one day apply for a job at a FAANG... sigh...I sometimes pine back to the days where I saturated the NICs of two quadcore servers with C++ std::unordered_map and FastCGI. Especially the days when I ask what problems the service mesh solves which isn't solvable with internal DNS and loadbalancers in our use case.
• riddleronroof 4y ago
  Funny. I am a backend engineer and think so about front end.
• kyriakos 4y ago
  Microservices and services in general where they weren't needed in the first place. Not saying they don't have their use but most companies just ride the hype train and start using technologies for the sake of using them. A couple of years later they end up with an unmanageable spaghetti of tightly coupled services.
• Shinmon 4y ago
  I think complexity is conserved. You can move complexity around in an application but in the end someone has to deal with it. E.g. the whole microservices vs monoliths debate. While a single microservice is usually relatively simple, you have 10s, 100s or even 1000s of microservices and the entire system design becomes crucial. With a monolith you don't have to worry as much about distributed services and inter-service communication but designing a robust monolith is still a complex process.
• sunshinerag 4y ago
  "Enterprise" Software development
• Caitlynbranson 4y ago
  [dead]