2 ms·
Something I've noticed is that there's a real trend in easing developer effort by trade-off with computational effort. In some cases it's worth it, but I think
by bane 13y ago
Something I've noticed is that there's a real trend in easing developer effort by trade-off with computational effort. In some cases it's worth it, but I think the pendulum has swung too far in most.
What I mean are these efforts to build huge distributed compute clusters that do X by distributing the problem easily, but at huge overhead cost for each individual actually useful computation.
The distribution, tracking of the compute tasks, the coordination of it all and the frameworks to make it easy insert such a tremendous overhead that I suspect most of the problems that are approached this way would be better tackled with a single machine and some smart old-fashioned algorithm development and developer elbow grease.
Of course simple economics might decide if 20 machines doing something are cheaper than 1 developer working an extra 3 or 4 months so 1 machine can do the same thing resolves it. But I suspect that level of analysis just isn't done and the result is to just throw hardware at the problem.
I've had to remind my developers in the past that we're working with machines that are literally capable of doing billions of operations per second and can do something like this https://www.youtube.com/watch?v=MJfceF0syK8 https://www.youtube.com/watch?v=MJfceF0syK8 in realtime.
Performance on individual machines is way more than sufficient to handle even pretty large datasets in the TBs, but we hamstring them with layers of garbage way too often.
One unfair benchmark I've heard before is "if 1 machine can do X in k time using this distributed framework, then 100 machines can do X in q<<k time using the same framework" but it's like also true that the same 1 machine could do it w time with w within spitting distance of q.