3 ms·
More than overengineering, misplaced engineering. We have a lot of stuff that "scales" and would not solve a real problem if it didn't "scale", but there is al
by morelisp 5y ago
More than overengineering, misplaced engineering.
We have a lot of stuff that "scales" and would not solve a real problem if it didn't "scale", but there is also an equally good solution not in the domain of the person who was tasked with writing it.
So for example, we were distributing a "large" (~100e6) multi-dimensional lookup table to ~30 servers to do some transformation job. The engineer who did it didn't like to think about data formats, so they used JSON. They didn't like to think about cache friendliness, so it was just maps of maps of maps. They didn't look at the actual dataset contents - which was some mostly opaque-to-us customer catalog SKUs - so didn't notice the massive redundancy in it. Implementing their algorithm as a straight line process, yeah, it did really take several hours to run and it was too long and probably required more memory than we wanted to commit.
But the solution wasn't "farm it out to a Hadoop cluster", it was "translate the dataset into some DFA-like things and then run some optimization passes on them" which yeah, was slow to "compile" - like 2min - but once compiled could chew through the rest of the job in another 10min, with sub-1G memory.
Very big / indirect "everything looks like a nail" situation, combined with lots of people who want to rush for the Big Data / ML / whatever stuff and not fundamentals. And the reasons don't seem to be uniform either. I've seen it from careerists and architecture astronauts, but also bootcamp grads who just weren't properly educated, at startups often early non-technical or technically-adjacent employees trying to help out but don't realize the mess they're making, and sometimes people who just don't want to think very hard.
- zomglings 5y agoIn the case of your JSON lookup table, as long as the damage was to the tune of < $50 and you don't need to run those jobs frequently, I wouldn't spend too much time on it. I would ask for documentation and put a few notes about what to do if you started needing to run those jobs on a frequent basis - because it can be difficult to predict whether or not that one-off script that someone hacked together is going to become a foundational pillar of your product.
- fragmede 5y agoI've worked at places where the AWS bill was obfuscated/otherwise opaque, so an implementing engineer would have no idea if a job cost $50 or $5,000. (Without a lot of research that's outside the scope of their job.) What tools are you using to display costs to internal developers?
- zomglings 5y agoMy business is very small and there are only 5 of us writing code. I just screen share the AWS bill periodically and we make sure that things aren't insane. We will be switching over to our own hardware in data centers towards the middle of next year. At that point, the whole team will need to be more aware of our constraints in terms of access to hardware.
- morelisp 5y agoI'm talking about a daily service that's a core business process, not a one-off BI analysis. Way, way more than $50. It's cute - and another example of not counting right, I guess - how many people think everyone is on AWS or some other elastic compute system. If we want to run a job, we have to find computers for it.
- zomglings 5y agoDefinitely if it's a daily service, it's totally worth spending the time on automation. I did make the assumption about AWS/equivalent, didn't realize how much of an unconscious assumption it had become. Sorry about that.
- Dylan16807 5y agoWell, finding computers internally is usually cheaper.