3 ms·

I've always been hesitant about adopting frameworks, and I still favor Flask over Django. Yet, I spent a couple of years in a Go-centric department where the se

I've always been hesitant about adopting frameworks, and I still favor Flask over Django. Yet, I spent a couple of years in a Go-centric department where the senior members were so against using frameworks that we ended up building everything ourselves from scratch. This included creating our own load-balancer, event store, and aggregation engine. When I attempted to point out that we were investing substantial time solving problems that have already been resolved, I was invariably met with references to "frameworks" and "complexity". However, in my perspective, our reluctance to utilize frameworks led to an inflated team size, slower development cycles, and a lengthy onboarding process for new members. I believe that if we had chosen to use industry-standard tools for our backend & infra (e.g. Kafka, Kubernetes, Apache Beam/Spark streams), we might have had to deal with a higher degree of complexity, but it would have been manageable. I would much prefer to diagnose issues with a Kubernetes deployment than to debug obscure panic-inducing code written by a colleague, which is questionably designed and lacks readily available support. One of the main advantages of using a framework is standardization: thanks to tools like git, we don't need to familiarize ourselves with every company's unique version control tool. Similarly, if everyone agrees on using Kubernetes and Docker, we can eliminate the need to learn each company's specific deployment process. This would allow us to dedicate more time to providing actual business value, rather than contemplating how our code should be transitioned into production. As for the department - it just got shutdown due to not making enough money and costing too much.

whstl 3y ago
There is no silver bullet that solves this problem. The thing I've seen the most in this industry is the opposite of that: teams that would mostly use third-party tools and frameworks, but the result was effectively the same, with complexity exploding and becoming so unbearable that the teams inflated to compensate, with productivity grinding to a halt. It happens with third-party tools as often as it does with NIH. Like you say, Kafka, Kubernetes and Spark would add a higher-degree of complexity. They aren't really that simple to manage in real world production environments. That's the problem here: the complexity. It doesn't matter where it comes from, it will bite you in the ass. One extreme of that is off-the-shelf enterprise software, that often requires a team of consultants to integrate. I've seen a few disaster, one software specifically started with a 500.000 price, but things became so complex the project ended up costing 4x that. There is no panacea against complexity. It costs money and takes time. The main problem here seems to be the "questionably designed and lacks readily available support" part, which is something universally bad, even when using third-party software. You can design your infrastructure badly. You can make questionable design using third-party tools just as much as you can with your own code. And support is also very often not readily available. "Similarly, if everyone agrees on using Kubernetes and Docker" I'm totally sympathetic to things becoming standards, and I don't have an axe to grind with those two tools, but the reason for the pushback against tools like Kubernetes and Docker is grounded on reality. Not only they introduce complexity by themselves, they are often just band-aids for accidental complexity introduced by teams and by third-party things like programming languages and frameworks.
- afiodorov 3y ago
  There isn't a one-size-fits-all solution to this issue. Before joining the department, I was part of a start-up where, with a considerably smaller team, we were able to process even larger amounts of data. We could introduce a new feature in a single day - a feat unthinkable in the department I just described. This speed was not achieved at the expense of stability but was a direct result of our choice of tools. Addressing the aspect of "questionable design and lack of readily available support", I believe only a handful of individuals are capable of developing something on the scale of Docker or Kubernetes, let alone doing it effectively. Assuming you're fortunate enough to hire such individuals, would you genuinely want them dedicating their cognitive resources to these types of problems? Even if they are highly skilled, can their solution truly compete with industry-standard frameworks developed by a team of equally competent individuals? And even if such an individual could design a superior solution, they are now responsible for maintaining it and training juniors on a system they likely have little motivation to understand. Moreover, the 'Not Invented Here' syndrome can lead to a cascade of more of the same. For instance, we needed to orchestrate several processes. Had we adopted Kubernetes, I would have suggested utilizing Airflow, which could have been implemented in just a few days. However, we chose to develop a custom pipeline runner, which took several months to complete. I understand that the no-frameworks sentiment has its roots – perhaps from Java's insistence on using frameworks. However, it seems that the industry swings from one extreme to the other. Golang emerged as a reaction to a world overrun with frameworks, but it appears to have veered too far in the other direction.
  - whstl 3y ago
    Dare I suggest that it is possible that the nimble startup was moving quickly not necessarily because of tool choice, but because of the small team size and because the appropriate amount of complexity was introduced? And conversely the inflated team was slow exactly because it was inflated and complexity was introduced? This effect happens with or without bringing third-party or in-house tooling/frameworks into the equation, and is well known and well covered since Mythical Man Month. And no, I don't see the point of rebuilding Kubernetes and Docker in-house and most critics also don't. What I'm saying that teams will be better of if they work towards not needing Docker or Kubernetes (or any NIH replacement) at all. But that requires challenging the assumption that software needs "something" like Docker or Kubernetes. Sure, if there's a tool for X then by all means use it, but my point is more that not doing anything in-house can often lead to the same issues.

whstl 3y ago

There is no silver bullet that solves this problem. The thing I've seen the most in this industry is the opposite of that: teams that would mostly use third-party tools and frameworks, but the result was effectively the same, with complexity exploding and becoming so unbearable that the teams inflated to compensate, with productivity grinding to a halt. It happens with third-party tools as often as it does with NIH. Like you say, Kafka, Kubernetes and Spark would add a higher-degree of complexity. They aren't really that simple to manage in real world production environments. That's the problem here: the complexity. It doesn't matter where it comes from, it will bite you in the ass. One extreme of that is off-the-shelf enterprise software, that often requires a team of consultants to integrate. I've seen a few disaster, one software specifically started with a 500.000 price, but things became so complex the project ended up costing 4x that. There is no panacea against complexity. It costs money and takes time. The main problem here seems to be the "questionably designed and lacks readily available support" part, which is something universally bad, even when using third-party software. You can design your infrastructure badly. You can make questionable design using third-party tools just as much as you can with your own code. And support is also very often not readily available. "Similarly, if everyone agrees on using Kubernetes and Docker" I'm totally sympathetic to things becoming standards, and I don't have an axe to grind with those two tools, but the reason for the pushback against tools like Kubernetes and Docker is grounded on reality. Not only they introduce complexity by themselves, they are often just band-aids for accidental complexity introduced by teams and by third-party things like programming languages and frameworks.

afiodorov 3y ago
There isn't a one-size-fits-all solution to this issue. Before joining the department, I was part of a start-up where, with a considerably smaller team, we were able to process even larger amounts of data. We could introduce a new feature in a single day - a feat unthinkable in the department I just described. This speed was not achieved at the expense of stability but was a direct result of our choice of tools. Addressing the aspect of "questionable design and lack of readily available support", I believe only a handful of individuals are capable of developing something on the scale of Docker or Kubernetes, let alone doing it effectively. Assuming you're fortunate enough to hire such individuals, would you genuinely want them dedicating their cognitive resources to these types of problems? Even if they are highly skilled, can their solution truly compete with industry-standard frameworks developed by a team of equally competent individuals? And even if such an individual could design a superior solution, they are now responsible for maintaining it and training juniors on a system they likely have little motivation to understand. Moreover, the 'Not Invented Here' syndrome can lead to a cascade of more of the same. For instance, we needed to orchestrate several processes. Had we adopted Kubernetes, I would have suggested utilizing Airflow, which could have been implemented in just a few days. However, we chose to develop a custom pipeline runner, which took several months to complete. I understand that the no-frameworks sentiment has its roots – perhaps from Java's insistence on using frameworks. However, it seems that the industry swings from one extreme to the other. Golang emerged as a reaction to a world overrun with frameworks, but it appears to have veered too far in the other direction.
- whstl 3y ago
  Dare I suggest that it is possible that the nimble startup was moving quickly not necessarily because of tool choice, but because of the small team size and because the appropriate amount of complexity was introduced? And conversely the inflated team was slow exactly because it was inflated and complexity was introduced? This effect happens with or without bringing third-party or in-house tooling/frameworks into the equation, and is well known and well covered since Mythical Man Month. And no, I don't see the point of rebuilding Kubernetes and Docker in-house and most critics also don't. What I'm saying that teams will be better of if they work towards not needing Docker or Kubernetes (or any NIH replacement) at all. But that requires challenging the assumption that software needs "something" like Docker or Kubernetes. Sure, if there's a tool for X then by all means use it, but my point is more that not doing anything in-house can often lead to the same issues.