3 ms·

I have inherited overengineered unwieldy architectures from programmers who had solved much harder technical problems than making our product successful. Havin

I have inherited overengineered unwieldy architectures from programmers who had solved much harder technical problems than making our product successful. Having to live with this kind of complexity slows you down in ways that you can't even really quantify (because it's difficult to run the required experiments while you're living the situation). A list of issues it causes in web development (for example): 1. Makes it difficult to experiment with new features or different features in a product because every change is now 10 times as involved as it needs to be. 2. Makes it difficult to hire programmers with less experience because you are working with 3 or 4 complicated technologies (caches, job queues, etc.) when a single database server would do. 3. Makes it difficult to debug problems because you have to identify bugs across $(n \choose 2)$ microservice boundaries (where $n$ is the number of microservices in your architecture), each boundary involves a different set of components to enable communication (REST APIs, GraphQL APIs, Redis, RabbitMQ, Elasticsearch, etc.) each of which perhaps made sense locally but which certainly do not make sense as part of the whole system. Apologies if I triggered anyone with this comment.

morelisp 5y ago
More than overengineering, misplaced engineering. We have a lot of stuff that "scales" and would not solve a real problem if it didn't "scale", but there is also an equally good solution not in the domain of the person who was tasked with writing it. So for example, we were distributing a "large" (~100e6) multi-dimensional lookup table to ~30 servers to do some transformation job. The engineer who did it didn't like to think about data formats, so they used JSON. They didn't like to think about cache friendliness, so it was just maps of maps of maps. They didn't look at the actual dataset contents - which was some mostly opaque-to-us customer catalog SKUs - so didn't notice the massive redundancy in it. Implementing their algorithm as a straight line process, yeah, it did really take several hours to run and it was too long and probably required more memory than we wanted to commit. But the solution wasn't "farm it out to a Hadoop cluster", it was "translate the dataset into some DFA-like things and then run some optimization passes on them" which yeah, was slow to "compile" - like 2min - but once compiled could chew through the rest of the job in another 10min, with sub-1G memory. Very big / indirect "everything looks like a nail" situation, combined with lots of people who want to rush for the Big Data / ML / whatever stuff and not fundamentals. And the reasons don't seem to be uniform either. I've seen it from careerists and architecture astronauts, but also bootcamp grads who just weren't properly educated, at startups often early non-technical or technically-adjacent employees trying to help out but don't realize the mess they're making, and sometimes people who just don't want to think very hard.
- zomglings 5y ago
  In the case of your JSON lookup table, as long as the damage was to the tune of < $50 and you don't need to run those jobs frequently, I wouldn't spend too much time on it. I would ask for documentation and put a few notes about what to do if you started needing to run those jobs on a frequent basis - because it can be difficult to predict whether or not that one-off script that someone hacked together is going to become a foundational pillar of your product.
  - fragmede 5y ago
    I've worked at places where the AWS bill was obfuscated/otherwise opaque, so an implementing engineer would have no idea if a job cost $50 or $5,000. (Without a lot of research that's outside the scope of their job.) What tools are you using to display costs to internal developers?
    - zomglings 5y ago
      My business is very small and there are only 5 of us writing code. I just screen share the AWS bill periodically and we make sure that things aren't insane. We will be switching over to our own hardware in data centers towards the middle of next year. At that point, the whole team will need to be more aware of our constraints in terms of access to hardware.
  - morelisp 5y ago
    I'm talking about a daily service that's a core business process, not a one-off BI analysis. Way, way more than $50. It's cute - and another example of not counting right, I guess - how many people think everyone is on AWS or some other elastic compute system. If we want to run a job, we have to find computers for it.
    - zomglings 5y ago
      Definitely if it's a daily service, it's totally worth spending the time on automation. I did make the assumption about AWS/equivalent, didn't realize how much of an unconscious assumption it had become. Sorry about that.
blueplanet200 5y ago
>caches, job queues, etc. This is over engineered? Yikes.
- morelisp 5y ago
  Complex (and usually somehow broken) caches where optimization would suffice is probably the #1 overengineering outcome I see. And at all levels - from caching HTTP proxies all the way down to functions that take five arguments to avoid re-summing a dozen numbers.
- zomglings 5y ago
  Absolutely, if used in a situation that doesn't call for caches or job queues or horizontally scalable NoSQL databases or whatever.
chousuke 5y ago
It's easy to fall into the trap of thinking you need to make things complicated because it's the "correct" way to do something. I once rewrote an internal tool that used AJAX requests to fetch data asynchronously for "performance"; the asynchronous fetches generally took 1-2 seconds because it had to use a super-slow API. The whole thing was a really weird mess of javascript, perl and Python and each component seemed to be tacked on to work around some issue (usually performance) with the other parts of the system. My version ended up being a single file of terrible go code (it was my first go project) and a few templates. Everything is rendered server-side and directly does read-only queries to the source database, bypassing the horrendous API. Of course, not using the API is "wrong" and liable to break at any time, but to this day the database schema has never changed such that that would have required me to rewrite the few bits of SQL I used, and the entire thing is fast enough that all user actions are effectively instant, even though the code is fully synchronous and does several database queries. Turns out databases are fast when you query them directly. Writing that tool made me realize that I prefer writing "badly" engineered tools optimized to be disposable, so that I can scrap and rewrite them really quickly when I need to. Instead of thinking "what do I do if I need to extend this later?" I try to ask myself "How easily can I delete this code?" I can better focus my effort on writing code that's reasonably performant and does not fail basic security when I don't really need to worry about whether the architecture is "good" or not, because as soon as I realize that it isn't, the bad part gets deleted and rewritten. Of course, in larger systems you don't often have the luxury of doing full rewrites, but even then the principle is applicable to individual functions, modules, or whatever subcomponents your system has.

morelisp 5y ago

More than overengineering, misplaced engineering. We have a lot of stuff that "scales" and would not solve a real problem if it didn't "scale", but there is also an equally good solution not in the domain of the person who was tasked with writing it. So for example, we were distributing a "large" (~100e6) multi-dimensional lookup table to ~30 servers to do some transformation job. The engineer who did it didn't like to think about data formats, so they used JSON. They didn't like to think about cache friendliness, so it was just maps of maps of maps. They didn't look at the actual dataset contents - which was some mostly opaque-to-us customer catalog SKUs - so didn't notice the massive redundancy in it. Implementing their algorithm as a straight line process, yeah, it did really take several hours to run and it was too long and probably required more memory than we wanted to commit. But the solution wasn't "farm it out to a Hadoop cluster", it was "translate the dataset into some DFA-like things and then run some optimization passes on them" which yeah, was slow to "compile" - like 2min - but once compiled could chew through the rest of the job in another 10min, with sub-1G memory. Very big / indirect "everything looks like a nail" situation, combined with lots of people who want to rush for the Big Data / ML / whatever stuff and not fundamentals. And the reasons don't seem to be uniform either. I've seen it from careerists and architecture astronauts, but also bootcamp grads who just weren't properly educated, at startups often early non-technical or technically-adjacent employees trying to help out but don't realize the mess they're making, and sometimes people who just don't want to think very hard.

zomglings 5y ago
In the case of your JSON lookup table, as long as the damage was to the tune of < $50 and you don't need to run those jobs frequently, I wouldn't spend too much time on it. I would ask for documentation and put a few notes about what to do if you started needing to run those jobs on a frequent basis - because it can be difficult to predict whether or not that one-off script that someone hacked together is going to become a foundational pillar of your product.
- fragmede 5y ago
  I've worked at places where the AWS bill was obfuscated/otherwise opaque, so an implementing engineer would have no idea if a job cost $50 or $5,000. (Without a lot of research that's outside the scope of their job.) What tools are you using to display costs to internal developers?
  - zomglings 5y ago
    My business is very small and there are only 5 of us writing code. I just screen share the AWS bill periodically and we make sure that things aren't insane. We will be switching over to our own hardware in data centers towards the middle of next year. At that point, the whole team will need to be more aware of our constraints in terms of access to hardware.
- morelisp 5y ago
  I'm talking about a daily service that's a core business process, not a one-off BI analysis. Way, way more than $50. It's cute - and another example of not counting right, I guess - how many people think everyone is on AWS or some other elastic compute system. If we want to run a job, we have to find computers for it.
  - zomglings 5y ago
    Definitely if it's a daily service, it's totally worth spending the time on automation. I did make the assumption about AWS/equivalent, didn't realize how much of an unconscious assumption it had become. Sorry about that.

blueplanet200 5y ago

>caches, job queues, etc. This is over engineered? Yikes.

morelisp 5y ago
Complex (and usually somehow broken) caches where optimization would suffice is probably the #1 overengineering outcome I see. And at all levels - from caching HTTP proxies all the way down to functions that take five arguments to avoid re-summing a dozen numbers.
zomglings 5y ago
Absolutely, if used in a situation that doesn't call for caches or job queues or horizontally scalable NoSQL databases or whatever.

chousuke 5y ago

It's easy to fall into the trap of thinking you need to make things complicated because it's the "correct" way to do something. I once rewrote an internal tool that used AJAX requests to fetch data asynchronously for "performance"; the asynchronous fetches generally took 1-2 seconds because it had to use a super-slow API. The whole thing was a really weird mess of javascript, perl and Python and each component seemed to be tacked on to work around some issue (usually performance) with the other parts of the system. My version ended up being a single file of terrible go code (it was my first go project) and a few templates. Everything is rendered server-side and directly does read-only queries to the source database, bypassing the horrendous API. Of course, not using the API is "wrong" and liable to break at any time, but to this day the database schema has never changed such that that would have required me to rewrite the few bits of SQL I used, and the entire thing is fast enough that all user actions are effectively instant, even though the code is fully synchronous and does several database queries. Turns out databases are fast when you query them directly. Writing that tool made me realize that I prefer writing "badly" engineered tools optimized to be disposable, so that I can scrap and rewrite them really quickly when I need to. Instead of thinking "what do I do if I need to extend this later?" I try to ask myself "How easily can I delete this code?" I can better focus my effort on writing code that's reasonably performant and does not fail basic security when I don't really need to worry about whether the architecture is "good" or not, because as soon as I realize that it isn't, the bad part gets deleted and rewritten. Of course, in larger systems you don't often have the luxury of doing full rewrites, but even then the principle is applicable to individual functions, modules, or whatever subcomponents your system has.