5 ms·

Why is that? It's because we don't learn from the past and keep thinking we're the smartest and have to reinvent everything with every new generation of develop

Why is that? It's because we don't learn from the past and keep thinking we're the smartest and have to reinvent everything with every new generation of developers. I've worked at a company where elasticseatch, postgres, message queues, custom microservice platforms, custom cicd platforms were deployed for a load that could fit in ram of one machine. With maybe 10 customers per minute. And you're wrong about accounting, new laws and regulations are being passed constantly and one has to keep up with them all the time

pm90 6y ago
Its important to distinguish b/w anecdotal experience and industry trends. A lot of great tech was invented to solve very real problems that groups of people faced. Once it gets hyped though, everyone wants it in their stack because they've read about it in blog posts or want to give a conference talk about or <INANE_REASON_X>. Elasticsearch, postgres, message queues etc. all were invented by people when they faced real problems. As such, I don't view them as reinventions, but as simply inventions. Software engineers tackle a problem space thats truly ginormous and diverse. It seems quite natural that this situation would spawn a multitude of tools to manage that complexity, get a competitive edge, or just build a product that just wasn't possible before.
- mstipetic 6y ago
  You're just proving my point. As a profession we base our decisions on blog posts and potential fun. I'm all for these technologies, I'm against the fact that regularly as a profession we order multi million dollar digging equipment to dig a small hole. I've literally seen a bored group of devops start building their own cicd solution and give up after a few months of (highly paid) work Not to mention that all these powerful technologies are super complicated and developers don't take the proper time to learn how to operate it correctly. Just think of the Hadoop craze 5 or so years ago when people were throwing it at few gig datasets. Or even now what's happening with "data engineering" where we build massive pipelines that could be handled by a competent python script. It's crazy and we don't as a profession have any success criteria on our deliverables and we've just accepted that "things are hard" while getting paid more than most professions can dream about
  - pm90 6y ago
    It’s becoming pretty clear that you’ve never seen the power of these tools in action. I don’t think I proved your point at all. I very explicitly showed the opposite. I’ve worked at companies that ingest petabytes of data, processing it would just not be possible without hadoop and sister tech (chiefly hdfs). Hundreds of developers working on a common codebase that would not have been possible without modern ci/cd solutions. Data pipelines whose reliability would be 0 without a modern rpc system like thrift to ensure system interfaces are well defined and consistent. There are very real and practical use cases for these technologies. Just because you haven’t experienced them doesn’t mean that they’re overkill. Just because enterprise engineers like to overengineer using these tools does not mean that everyone does. There are very real scenarios where these tools have helped smaller companies and startups to punch beyond their weight.
    - mstipetic 6y ago
      Mate I'm with you, they're extremely powerful tools and it's basically magic what a few good engineers can build with modern tooling. When you say enterprise engineers that's 80+ percent of our profession. My gripe is only with lack of standards and misusing those tools without understanding them properly, which happens all the time
      - pm90 6y ago
        Understood. Apologies for the Curt tone in my earlier comment. Cranky before morning coffee.
- Silhouette 6y ago
  A lot of new tools were indeed invented to solve real problems, albeit not necessarily real problems shared by as many people as subsequently adopt those tools. But there is also a huge amount of reinvention because, as mstipetic was saying, a lot of developers just don't study their subject any more. It's so superficial, rarely learning lessons from history. In particular, that's how we get the modern world of web development, with new tools and frameworks every few months that do require retraining but don't contribute many new ideas. This is an industry where the dominant programming language for the front end literally had no standard concept of modules until relatively recently. To this day, almost all of the main tools like building, testing and static analysis usually have at least one extra layer of transpilation complexity on top, and usually it has to be configured separately for each tool. The tools themselves are often relatively basic, with routine (in the rest of the programming world) optimisations like tree shaking regarded as some sort of incredible advance. The same goes for the software architecture patterns used by the big front-end libraries and frameworks. And setting all of this up has become so complicated that there are now tools to scaffold the project and set up the other tools that orchestrate the other tools, which I really wish was a joke but is literally true. Unfortunately, I think the real root cause of the rot is money. There is so much cash now in certain parts of the industry that it is undeniably a successful career strategy for developers to go job-hopping and collecting buzzwords working at those places, even if they are never really learning much of substance or building much of actual, lasting value. It turns out that huge numbers of people are willing to pay, directly or indirectly, for junk. And so writing junk and not caring is an effective business model. The silver lining is that it's a huge industry and there is still room for quality as well, so those who do make the effort to build better software can at least make a living doing it, even if the rewards are more in satisfaction than greater financial compensation.

pm90 6y ago

Its important to distinguish b/w anecdotal experience and industry trends. A lot of great tech was invented to solve very real problems that groups of people faced. Once it gets hyped though, everyone wants it in their stack because they've read about it in blog posts or want to give a conference talk about or <INANE_REASON_X>. Elasticsearch, postgres, message queues etc. all were invented by people when they faced real problems. As such, I don't view them as reinventions, but as simply inventions. Software engineers tackle a problem space thats truly ginormous and diverse. It seems quite natural that this situation would spawn a multitude of tools to manage that complexity, get a competitive edge, or just build a product that just wasn't possible before.

mstipetic 6y ago
You're just proving my point. As a profession we base our decisions on blog posts and potential fun. I'm all for these technologies, I'm against the fact that regularly as a profession we order multi million dollar digging equipment to dig a small hole. I've literally seen a bored group of devops start building their own cicd solution and give up after a few months of (highly paid) work Not to mention that all these powerful technologies are super complicated and developers don't take the proper time to learn how to operate it correctly. Just think of the Hadoop craze 5 or so years ago when people were throwing it at few gig datasets. Or even now what's happening with "data engineering" where we build massive pipelines that could be handled by a competent python script. It's crazy and we don't as a profession have any success criteria on our deliverables and we've just accepted that "things are hard" while getting paid more than most professions can dream about
- pm90 6y ago
  It’s becoming pretty clear that you’ve never seen the power of these tools in action. I don’t think I proved your point at all. I very explicitly showed the opposite. I’ve worked at companies that ingest petabytes of data, processing it would just not be possible without hadoop and sister tech (chiefly hdfs). Hundreds of developers working on a common codebase that would not have been possible without modern ci/cd solutions. Data pipelines whose reliability would be 0 without a modern rpc system like thrift to ensure system interfaces are well defined and consistent. There are very real and practical use cases for these technologies. Just because you haven’t experienced them doesn’t mean that they’re overkill. Just because enterprise engineers like to overengineer using these tools does not mean that everyone does. There are very real scenarios where these tools have helped smaller companies and startups to punch beyond their weight.
  - mstipetic 6y ago
    Mate I'm with you, they're extremely powerful tools and it's basically magic what a few good engineers can build with modern tooling. When you say enterprise engineers that's 80+ percent of our profession. My gripe is only with lack of standards and misusing those tools without understanding them properly, which happens all the time
    - pm90 6y ago
      Understood. Apologies for the Curt tone in my earlier comment. Cranky before morning coffee.
Silhouette 6y ago
A lot of new tools were indeed invented to solve real problems, albeit not necessarily real problems shared by as many people as subsequently adopt those tools. But there is also a huge amount of reinvention because, as mstipetic was saying, a lot of developers just don't study their subject any more. It's so superficial, rarely learning lessons from history. In particular, that's how we get the modern world of web development, with new tools and frameworks every few months that do require retraining but don't contribute many new ideas. This is an industry where the dominant programming language for the front end literally had no standard concept of modules until relatively recently. To this day, almost all of the main tools like building, testing and static analysis usually have at least one extra layer of transpilation complexity on top, and usually it has to be configured separately for each tool. The tools themselves are often relatively basic, with routine (in the rest of the programming world) optimisations like tree shaking regarded as some sort of incredible advance. The same goes for the software architecture patterns used by the big front-end libraries and frameworks. And setting all of this up has become so complicated that there are now tools to scaffold the project and set up the other tools that orchestrate the other tools, which I really wish was a joke but is literally true. Unfortunately, I think the real root cause of the rot is money. There is so much cash now in certain parts of the industry that it is undeniably a successful career strategy for developers to go job-hopping and collecting buzzwords working at those places, even if they are never really learning much of substance or building much of actual, lasting value. It turns out that huge numbers of people are willing to pay, directly or indirectly, for junk. And so writing junk and not caring is an effective business model. The silver lining is that it's a huge industry and there is still room for quality as well, so those who do make the effort to build better software can at least make a living doing it, even if the rewards are more in satisfaction than greater financial compensation.