3 ms·
At one time I worked for a company that, as a side development effort, built some pretty cool data processing software. It was built as a side effort to support
by bane 14y ago
At one time I worked for a company that, as a side development effort, built some pretty cool data processing software. It was built as a side effort to support the desktop software we were selling as the main line of business.
Since we were running on a single desktop, it never occurred to us to think about large server clusters cranking away on data day and night. We just had to make it work on a regular 'ol single machine. Since user experience is paramount in user-facing desktop software, it meant clever algorithms and performant code to get it to process the data in a timely fashion.
One day, one of our customers found themselves incredibly dissatisfied with the money and manpower that was being poured into a "big-data" data processing solution on another project, 3 FTEs and a half-dozen very expensive servers.
On a lark he asked us if our data processing tool could do what their big-data system was doing. As it turned out, functionality-wise, we were about 80% there. With a couple months worth of work we were 95% there. The big question was performance.
The customer setup a single instance of our single desktop data processing software and kicked it off. A few days later it had finished. "It's not fast enough" he said. So we spent a few months making it faster and getting rid of bottlenecks. We got it down to just a few hours. "It's not fast enough!" he said again.
So we halved the time...but that was it, it was pretty much all the performance we were ever going to get out of our code-base. We asked "how long does your big data system take to do what we're doing in a morning?", fully expecting that their multi-million dollar solution was cranking through the same data in a ten minutes to a half hour or so.
He responded, "two and a half weeks".
A year later, the other solutions provider called us up and asked us if we were interested in purchasing their big-data processing division. We politely declined.
The point is that sometimes the giant software stacks and interconnected clustering systems we build to tackle "big data" sometimes introduce so much overhead into the system that a single system, running some smart software can outperform the larger systems. We engineers sometimes get incredibly lost in the details of making cool things work to stop and ask ourselves if what we're doing is actually the best approach in the end.