4 ms·
I'm torn. On one hand, I absolutely see the logic, feel the occasional despair, and tend to agree with you, especially when it comes to economies of scale. I'l
by existencebox 8y ago
I'm torn.
On one hand, I absolutely see the logic, feel the occasional despair, and tend to agree with you, especially when it comes to economies of scale. I'll never write algos that'll detect the alpha hedge funds can. I'll never write the NLP that my own employer can leverage trivially.
On the other hand, do I really need to? In 90% of the use cases where I want to solve a problem, with some pile of hacks and heuristics I've gotten "more than good enough." And the big companies will keep investing on ways to scale up and optimize these algos, which will only benefit us tiny users too. I did both my last publication and patent using a CPU-bound model and not an ounce of deep learning, with a corpus you could fit on a thumbdrive.
I've watched a bigCo spend _months_ of some of the best engineers I know to optimize a tiny subproblem of a subproblem. (object similarity detection) Meanwhile I had to solve an isomorphism for my home camera system, threw together a prototype in a few hours with openCV and _really_ rudimentary bit-array hacks, declared it "WORKABLE" and have been using it for the last 3 years. There are some areas where what's in open source is pretty much what I'd use given any options. (Pandas, Spark, postgres) and some areas where it's not (pgadmin :P and OS UX (looking at you canonical) to name two). This isn't a one sided battle, to the strength and credit of non-big-corps.
Maybe it's the eternal rebel in me, but I'm a fan of desert kenobi, it's the start of a journey. Stick it to the man!