4 ms·

Software engineering is the perfect example of the "blind scientists and the elephant" problem. It is a very complex field, with a number of related but distinc

Software engineering is the perfect example of the "blind scientists and the elephant" problem. It is a very complex field, with a number of related but distinct disciplines and activities required to make it work; it's impossible to be an expert in everything, so we tend to specialise: we have back-end engineers, front-end engineers, data engineers, SRE experts, devops specialists, database experts, data scientists and so on. Additionally, the software we are building varies wildly in terms of complexity, dependencies, external requirements etc; and finally, the scale of that software and the teams building it can vary from one person to literally thousands. Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped". The reality is that various technologies and patterns -- microservices, monoliths, Kubernetes, Heroku, AWS, whatever -- are tools that enable us to solve certain problems in software development. And different teams have different problems and need different solutions, and each needs to carefully weigh their options and adopt the solutions that work the best for them. Yes, choosing the wrong solutions can be expensive and might take a long time to fix, but that can happen to everyone and actually shows how important it is to understand what is actually needed. And it's completely pointless to berate someone for their choices unless you have a very detailed insight into their particular needs.

alexandercrohde 7y ago
> Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped". It's my experience the opposite is true. The blindness is people overestimating their needs (or resume-padding) and using specialized, overcomplicated tools meant for traffic in the billions (e.g. cassandra, kafka, mapreduce) for 20-person startups that haven't hit rapid growth (most of which never do).
- BerislavLopac 7y ago
  I'm afraid you might be falling into the exact trap I have described. Realistically, how many such cases have you seen? And of those you have, how many did actually implement such a complex solution and ran it for a long time without either closing down or transforming to something more suited to their needs?
  - alexandercrohde 7y ago
    I kind of suspect you may be the one lacking experience. I've worked at at least 8 different tech companies, mostly startups in SF or NY. The vast majority used overcomplicated technologies that didn't fit the needs of project (most frequently microservices and no-sql). Off the top of my head I can't think of a single time such mistakes got corrected. More often than not things would continue to be even more poorly designed with the addition of new unnecessary technology. In short -- I'm annoyed about this stuff because I've seen it first hand and had to struggle with it for numerous years. Your weird theory that people are inventing hypothetical situations to be angry about... well I think you're the one inventing hypotheticals here...
    - BerislavLopac 7y ago
      > mostly startups in SF or NY This explains a lot. Fair point, this kind of approach is likely very common in this very small (except financially) corner of the software industry. > Your weird theory that people are inventing hypothetical situations to be angry about Uh, can you point where I said or implied that? That doesn't have much in common with the point I was trying to get across, but I can believe that I failed in my intent.
    - RonanTheGrey 7y ago
      In 24 years I've only rarely seen scenarios that actually require something at the level of complexity that k8s represents. I worked on Bing some years ago, and it would definitely have benefited but MS rolled their own solution (which has since been replaced by I don't know what). I've seen k8s USED many times where it was wholly and completely unnecessary and being pushed by juniors who wanted to go apply to Google in a year or two. I am currently running a service that receives 3000 rpm spike and averaging 500k requests a day. On a single server behind cloudflare deployed straight from Github. We have a version of the service also running on ElasticBeanstalk with a single server. Neither experiences downtime. People severely overestimate their needs. Google, Facebook, Microsoft, Amazon? Are serving literally billions of requests per minute. They have a need for that level of complexity. Most of us here... do not.
deleted 7y ago
[deleted]

alexandercrohde 7y ago

> Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped". It's my experience the opposite is true. The blindness is people overestimating their needs (or resume-padding) and using specialized, overcomplicated tools meant for traffic in the billions (e.g. cassandra, kafka, mapreduce) for 20-person startups that haven't hit rapid growth (most of which never do).

BerislavLopac 7y ago
I'm afraid you might be falling into the exact trap I have described. Realistically, how many such cases have you seen? And of those you have, how many did actually implement such a complex solution and ran it for a long time without either closing down or transforming to something more suited to their needs?
- alexandercrohde 7y ago
  I kind of suspect you may be the one lacking experience. I've worked at at least 8 different tech companies, mostly startups in SF or NY. The vast majority used overcomplicated technologies that didn't fit the needs of project (most frequently microservices and no-sql). Off the top of my head I can't think of a single time such mistakes got corrected. More often than not things would continue to be even more poorly designed with the addition of new unnecessary technology. In short -- I'm annoyed about this stuff because I've seen it first hand and had to struggle with it for numerous years. Your weird theory that people are inventing hypothetical situations to be angry about... well I think you're the one inventing hypotheticals here...
  - BerislavLopac 7y ago
    > mostly startups in SF or NY This explains a lot. Fair point, this kind of approach is likely very common in this very small (except financially) corner of the software industry. > Your weird theory that people are inventing hypothetical situations to be angry about Uh, can you point where I said or implied that? That doesn't have much in common with the point I was trying to get across, but I can believe that I failed in my intent.
  - RonanTheGrey 7y ago
    In 24 years I've only rarely seen scenarios that actually require something at the level of complexity that k8s represents. I worked on Bing some years ago, and it would definitely have benefited but MS rolled their own solution (which has since been replaced by I don't know what). I've seen k8s USED many times where it was wholly and completely unnecessary and being pushed by juniors who wanted to go apply to Google in a year or two. I am currently running a service that receives 3000 rpm spike and averaging 500k requests a day. On a single server behind cloudflare deployed straight from Github. We have a version of the service also running on ElasticBeanstalk with a single server. Neither experiences downtime. People severely overestimate their needs. Google, Facebook, Microsoft, Amazon? Are serving literally billions of requests per minute. They have a need for that level of complexity. Most of us here... do not.

deleted 7y ago

[deleted]