3 ms·

From the article: Where are things headed? There's a growing consensus that deep learning is going to be a centralizing technology rather than a decentralizin

From the article: Where are things headed? There's a growing consensus that deep learning is going to be a centralizing technology rather than a decentralizing one. We seem to be headed toward a world where the only people with enough data and compute to train truly state-of-the-art networks are a handful of large tech companies. This is terrifying, but the same conclusion that I've come to. I'm starting to feel more and more dread that this isn't how the future was supposed to be. I used to be so passionate about technology, especially about AI as the last solution in computer science. But anymore, the most likely scenario I see for myself is moving out into the desert like OB1 Kenobi. I'm just, so weary. So unbelievably weary, day by day, in ever increasing ways.

coffeemug 8y ago
Hey, I hope you don't take it the wrong way -- I'm coming from a place where I hope you start feeling better -- but what you're experiencing might be depression/mood affiliation. I.e. you feel weary and bleak, so the world seems weary and bleak. There are enormous problems for humanity to solve, but that has always been the case. From plagues and famines, to world wars, to now climate change, AI risk, and maybe technology centralization. We've solved massive problems before at unbelievable odds, and I want to think we'll do it again. And if not, what of it? What else is there to do but work tirelessly at attempting to solve them? I hope you feel better, and find help if you need it -- don't mean to presume too much. My e-mail is in my profile if you (or anyone else) needs someone to talk to.
guelo 8y ago
It seems kind of obvious in retrospect. I used to envision "the singularity" as somehow organically emerging from distributed technology and that would make it benevolent. But that was so naive. The singularity was always going to require massive investments of a scale that only monopolies or militaries can provide. That it currently looks like it will come out of ad-tech monopolies comfortable with psychological manipulation at a global scale is the most terrifying possibility of all.
existencebox 8y ago
I'm torn. On one hand, I absolutely see the logic, feel the occasional despair, and tend to agree with you, especially when it comes to economies of scale. I'll never write algos that'll detect the alpha hedge funds can. I'll never write the NLP that my own employer can leverage trivially. On the other hand, do I really need to? In 90% of the use cases where I want to solve a problem, with some pile of hacks and heuristics I've gotten "more than good enough." And the big companies will keep investing on ways to scale up and optimize these algos, which will only benefit us tiny users too. I did both my last publication and patent using a CPU-bound model and not an ounce of deep learning, with a corpus you could fit on a thumbdrive. I've watched a bigCo spend _months_ of some of the best engineers I know to optimize a tiny subproblem of a subproblem. (object similarity detection) Meanwhile I had to solve an isomorphism for my home camera system, threw together a prototype in a few hours with openCV and _really_ rudimentary bit-array hacks, declared it "WORKABLE" and have been using it for the last 3 years. There are some areas where what's in open source is pretty much what I'd use given any options. (Pandas, Spark, postgres) and some areas where it's not (pgadmin :P and OS UX (looking at you canonical) to name two). This isn't a one sided battle, to the strength and credit of non-big-corps. Maybe it's the eternal rebel in me, but I'm a fan of desert kenobi, it's the start of a journey. Stick it to the man!
patcon 8y ago
I recall this was the top-voted comment (rightfully imho) until shortly after someone suggested that the author was depressed instead of being reasonable... Now it's at the bottom, and lots of "wow, this is super-interesting" comments are above. fwiw, this quoted bit also jumped out at me as perhaps the most important note I never know what to make of the psychology of this/my community (tech, specifically), but the dynamics here on HN always provide me lots of "food for thought" to overfit :)
rstuart4133 8y ago
I took the opposite conclusion from it. After doing a back of the envelope calculation and concluding AlphaGo Zero 250MW hours of power I concluded we were going to see AI monopolies. Someone would develop the best imagine recognition, corner the market and get a stream of profits that allowed them to pour more money into training and round and round we go. As you say it was a depressing thought. If deep feature learning / embedding is really this effective it turns it upside down. We could end up with a AI being constructed from layers you buy off the shelf. Lots of parts coming from different vendors, all competing - not unlike the software stacks we use now. It might even go the MJPEG / AV1 way - large companies get the shits with paying someone for the layers and combine their resources to build a better layer than any one of them could on their own, and they can do it because they are all going do additional training on top and put it to a different use. This is impossibly speculative - but thinking you had to invest in 250MW Hours go build a decent Go machine was equally speculative and not nice. No open source group was going to come up with the next killer Go playing box if turned out that way. Now there is a possibility it may not.

coffeemug 8y ago

Hey, I hope you don't take it the wrong way -- I'm coming from a place where I hope you start feeling better -- but what you're experiencing might be depression/mood affiliation. I.e. you feel weary and bleak, so the world seems weary and bleak. There are enormous problems for humanity to solve, but that has always been the case. From plagues and famines, to world wars, to now climate change, AI risk, and maybe technology centralization. We've solved massive problems before at unbelievable odds, and I want to think we'll do it again. And if not, what of it? What else is there to do but work tirelessly at attempting to solve them? I hope you feel better, and find help if you need it -- don't mean to presume too much. My e-mail is in my profile if you (or anyone else) needs someone to talk to.

guelo 8y ago

It seems kind of obvious in retrospect. I used to envision "the singularity" as somehow organically emerging from distributed technology and that would make it benevolent. But that was so naive. The singularity was always going to require massive investments of a scale that only monopolies or militaries can provide. That it currently looks like it will come out of ad-tech monopolies comfortable with psychological manipulation at a global scale is the most terrifying possibility of all.

existencebox 8y ago

I'm torn. On one hand, I absolutely see the logic, feel the occasional despair, and tend to agree with you, especially when it comes to economies of scale. I'll never write algos that'll detect the alpha hedge funds can. I'll never write the NLP that my own employer can leverage trivially. On the other hand, do I really need to? In 90% of the use cases where I want to solve a problem, with some pile of hacks and heuristics I've gotten "more than good enough." And the big companies will keep investing on ways to scale up and optimize these algos, which will only benefit us tiny users too. I did both my last publication and patent using a CPU-bound model and not an ounce of deep learning, with a corpus you could fit on a thumbdrive. I've watched a bigCo spend _months_ of some of the best engineers I know to optimize a tiny subproblem of a subproblem. (object similarity detection) Meanwhile I had to solve an isomorphism for my home camera system, threw together a prototype in a few hours with openCV and _really_ rudimentary bit-array hacks, declared it "WORKABLE" and have been using it for the last 3 years. There are some areas where what's in open source is pretty much what I'd use given any options. (Pandas, Spark, postgres) and some areas where it's not (pgadmin :P and OS UX (looking at you canonical) to name two). This isn't a one sided battle, to the strength and credit of non-big-corps. Maybe it's the eternal rebel in me, but I'm a fan of desert kenobi, it's the start of a journey. Stick it to the man!

patcon 8y ago

I recall this was the top-voted comment (rightfully imho) until shortly after someone suggested that the author was depressed instead of being reasonable... Now it's at the bottom, and lots of "wow, this is super-interesting" comments are above. fwiw, this quoted bit also jumped out at me as perhaps the most important note I never know what to make of the psychology of this/my community (tech, specifically), but the dynamics here on HN always provide me lots of "food for thought" to overfit :)

rstuart4133 8y ago

I took the opposite conclusion from it. After doing a back of the envelope calculation and concluding AlphaGo Zero 250MW hours of power I concluded we were going to see AI monopolies. Someone would develop the best imagine recognition, corner the market and get a stream of profits that allowed them to pour more money into training and round and round we go. As you say it was a depressing thought. If deep feature learning / embedding is really this effective it turns it upside down. We could end up with a AI being constructed from layers you buy off the shelf. Lots of parts coming from different vendors, all competing - not unlike the software stacks we use now. It might even go the MJPEG / AV1 way - large companies get the shits with paying someone for the layers and combine their resources to build a better layer than any one of them could on their own, and they can do it because they are all going do additional training on top and put it to a different use. This is impossibly speculative - but thinking you had to invest in 250MW Hours go build a decent Go machine was equally speculative and not nice. No open source group was going to come up with the next killer Go playing box if turned out that way. Now there is a possibility it may not.