8 ms·

I've been wondering for a long time whether machine learning will break the long-time tech industry lifecycle we've all come to regard as some kind of immutable

I've been wondering for a long time whether machine learning will break the long-time tech industry lifecycle we've all come to regard as some kind of immutable law of nature. Traditionally, 1) we start with big, established companies in tech. 2) A few people, dissatisfied with the trammels that big companies place on their employees, leave, found a new company, and then, 3) through superior grit, gumption, execution, flexibility, and speed, steal market share from the big companies. As the new company grows, 4) it accumulates organizational scar tissue and is slowly infiltrated by incompetent careerists, and finally, 5) becomes the very company that the founders set out to create. We've been going through this cycle since the late 1950s, when the "traitorous eight" left Shockley Semiconductor Laboratory to found "Fairchild Semiconductor". Machine learning might stop this cycle by breaking step #3. Big companies have access to huge data sets that no group of human developers, no matter how talented, can match. Even slow, hidebound, and barely-competent use of these data sets can squash upstart smaller companies. It'll be like a planet, full of water and life, with a core that cools to the point where it can no longer sustain plate tectonics. Without constant replenishment, the crust seizes up, the oceans boil off, and the life dies. The tech industry will become Mars.

AndrewKemendo 10y ago
Machine learning might stop this cycle by breaking step #3. Yea that's exactly what I think is happening, and in fact that's one of the core promises of being first into AI - you use it to move faster than anyone else can because it gives you better intelligence and prediction. Even more so with AGI, which Deepmind (Alphabet company) has explictly stated they are trying to create.
thwd 10y ago
On the other hand, #3 might in this case be exactly that: Small companies creating accurate & precise modeling techniques using only a limited amount of data, while giants continue to rely on the sheer and blunt volume of it.
- trhway 10y ago
  >Small companies creating accurate & precise modeling techniques using only a limited amount of data, while giants continue to rely on the sheer and blunt volume of it. until of course sheer and blunt volume allows to build more accurate and precise models. To me this is exactly at the core of the current AI wave - the models emerging inside deep representational layers are more complex, accurate and precise way beyond whatever human analytical thinking could have built or even just come up with. My friend just couple days ago was talking about one specific multi-year effort few years ago to constructively build a practical model of sentiment analysis which i in turn contrasted with that recent result from OpenAI of "sentiment neuron" emerging from [unsupervised] learning of next character in Amazon reviews.
antognini 10y ago
You may be right for the time being, but I don't think we're permanently entering a brave new world where machine learning startups are doomed to fail to the big tech companies. It's well known that existing machine learning algorithms are extremely inefficient learners, which is why you need enormous datasets and computing resources to be competitive currently. Right now, for instance, a computer would need to see thousands of different images of a new animal to recognize it consistently, whereas a human only needs to see a single hand-drawn sketch. If someone can develop machine learning techniques that make computers generalize as flexibly as humans, that would be a huge game changer and would completely upend the advantages that the big tech companies have.
- AndrewKemendo 10y ago
  If someone can develop machine learning techniques that make computers generalize as flexibly as humans Funny you should mention that... Deepmind (acquired by Google in 2014) goal: "DeepMind Technologies' goal is to "solve intelligence" Demis has gone one to say elsewhere their goal is to create AGI Also, the data you need to get there is owned by a few companies. Also those companies hire effectively every great ML PhD on the planet. It's well known that existing machine learning algorithms are extremely inefficient learners Right, which is why the heads of research at AFGAM are the pre-eminent minds in the field (Lecun, Hinton etc...) and are actively working on this along with unsupervised learning. These companies are either funding university labs that are developing this stuff or doing it in house.
- trhway 10y ago
  >Right now, for instance, a computer would need to see thousands of different images of a new animal to recognize it consistently, whereas a human only needs to see a single hand-drawn sketch. valid question here would be - would a fresh human mind perform that well too? Or may be such performance is conditioned upon having at least several years of experience of living on planet Earth and thus having the models of animals and everything else around been built using billions of images that human eyes would have generated during those years? > It's well known that existing machine learning algorithms are extremely inefficient learners, which is why you need enormous datasets and computing resources to be competitive currently. when put into perspective, i.e. compared with the cardinality of the above mentioned set of images human eyes generate and computational performance of the brain - 40K TFlops - computers don't look that inefficient learners to me.
roymurdock 10y ago
Generally agree with OP that ML is one of the last viable frontiers for the current generation of computer tech. We are basically taking the data generated by machines invented decades ago, and using statistics to optimize a few % of productivity points more out of them every year. This optimization applies to humans to - one day you'll be required to have an OBD dongle monitoring your driving data to get car insurance, for instance. Many facets of your life will be monitored and fed into a company's individualized production function or pricing equation. When the economic cake is growing slowly, every last drop of productivity enhancement counts, and ML is doing a good job at shaving a few basis points off basic costs, such as energy (see DeepMind's work on Google's datacenters), transportation (autonomous cars), and human capital (making some simple jobs redundant). That's why everyone is piling into IoT/ML/AI right now. What ML is NOT doing is creating a new paradigm/market/computing platform. It's optimizing old markets and technologies. It's not baking new cakes. We'll see a new platform soon. The cycle of human technological progress has been chugging along steadily for a long time and has sped up exponentially in the past 200 years. I'm sure some new tech will come along soon that will allow us to bake more cakes instead of spreading icing ever-thinner on the ones we already have.
- halflings 10y ago
  Generally agree that ML is often shaving off a few % after all the low hanging fruits that can be achieved without it have been exploited, but... > What ML is NOT doing is creating a new paradigm/market/computing platform. It's optimizing old markets and technologies. It's not baking new cakes. This is simply not true: there are many products/markets (such as automatic translation, speech recognition) that are simply not possible without this progress in ML.
  - pas 9y ago
    What's the paradigm? In translate it replaces a naive translator's job. In speech recognition it gives you a new-ish interface to systems. In prediction and personalization it gives you what you likely want. (Local weather, local traffic info, local news, in the language it knows you understand, in the format you prefer, at the time it think you most likely want it, etc..). It puts some "smart" into things. But translate is just a smart dictionary, it's not a real translator, for that we need stronger AI (something like the Jeopardy playing Watson + Google Knowledge Graph / FreeBase + language translation + it should ask questions if it doesn't understand something). ML is amazing, but it's just a slow march toward more and more adaptive smarts (general intelligence) in a box (hence artificial). And there's probably a tipping point for that. When it can start to learn, or program itself, blablabla... ( https://intelligence.org/2013/04/29/intelligence-explosion-microeconomics-released/ https://intelligence.org/2013/04/29/intelligence-explosion-m... )
romaniv 10y ago
I've been wondering the same thing. It is why I don't like the fact that we need increasingly specialized types of computers to do ML. This is also why I am very interested in research that is not based on deep neural networks, which seem to be especially hungry in terms of training data and hardware power.
Eridrus 9y ago
I think this depends on what you see as the downfall of these previous companies. If you agree with Christensen's theory of disruptive innovation, the issue was that large companies would undervalue small markets that could grow to become large markets. When seen in this light, it would be surprising if ML were a disruptive technology, since it falls more into the "sustaining" category of innovation. Most of the large tech companies were built on technologies that no-one expected to become world-defining companies at the time. With all this focus on ML/AI, it would be very surprising if a small ML/AI shop became huge, rather then getting bought or replicated. Almost by definition, the big companies of the next era will come from the spaces that large companies are ignoring.
- AndrewKemendo 9y ago
  the big companies of the next era will come from the spaces that large companies are ignoring. My entire point is that they aren't ignoring any spaces, they have all future spaces accounted for by having their tentacles in literally everything possible that could be future tech. Bldinsides come from different industries, so they need to cover their bases from all directions, so that's what they all have done with their projects, products, investments in...AI, Energy, Biology, Transportation, Infrastructure, Waste Disposal, Agriculture the list goes on. The time of the non-ubiquitous firm is over. AFGAM aren't software companies, they are innovation conglomerates. I mean why do you think google changed to Alphabet?
  - Eridrus 9y ago
    Right, but this has less to do with the fundamentals of ML as a technology, rather than the less sexy explanation of better management practices.
    - AndrewKemendo 9y ago
      They are inextricably linked. If you organize your company around ML, as Google, Facebook, Amazon, Microsoft and Apple have stated they are doing, then yes it does have to do with the fundamentals of ML. Advances and implementations of ML is driving the creation and iteration of products for these companies, as well as helping them run their companies and evaluate where to go next - follow the data as it were. So it is a critical point that every one of their CEOs has outlined in clear detail over the past few years.

AndrewKemendo 10y ago

Machine learning might stop this cycle by breaking step #3. Yea that's exactly what I think is happening, and in fact that's one of the core promises of being first into AI - you use it to move faster than anyone else can because it gives you better intelligence and prediction. Even more so with AGI, which Deepmind (Alphabet company) has explictly stated they are trying to create.

thwd 10y ago

On the other hand, #3 might in this case be exactly that: Small companies creating accurate & precise modeling techniques using only a limited amount of data, while giants continue to rely on the sheer and blunt volume of it.

trhway 10y ago
>Small companies creating accurate & precise modeling techniques using only a limited amount of data, while giants continue to rely on the sheer and blunt volume of it. until of course sheer and blunt volume allows to build more accurate and precise models. To me this is exactly at the core of the current AI wave - the models emerging inside deep representational layers are more complex, accurate and precise way beyond whatever human analytical thinking could have built or even just come up with. My friend just couple days ago was talking about one specific multi-year effort few years ago to constructively build a practical model of sentiment analysis which i in turn contrasted with that recent result from OpenAI of "sentiment neuron" emerging from [unsupervised] learning of next character in Amazon reviews.

antognini 10y ago

You may be right for the time being, but I don't think we're permanently entering a brave new world where machine learning startups are doomed to fail to the big tech companies. It's well known that existing machine learning algorithms are extremely inefficient learners, which is why you need enormous datasets and computing resources to be competitive currently. Right now, for instance, a computer would need to see thousands of different images of a new animal to recognize it consistently, whereas a human only needs to see a single hand-drawn sketch. If someone can develop machine learning techniques that make computers generalize as flexibly as humans, that would be a huge game changer and would completely upend the advantages that the big tech companies have.

AndrewKemendo 10y ago
If someone can develop machine learning techniques that make computers generalize as flexibly as humans Funny you should mention that... Deepmind (acquired by Google in 2014) goal: "DeepMind Technologies' goal is to "solve intelligence" Demis has gone one to say elsewhere their goal is to create AGI Also, the data you need to get there is owned by a few companies. Also those companies hire effectively every great ML PhD on the planet. It's well known that existing machine learning algorithms are extremely inefficient learners Right, which is why the heads of research at AFGAM are the pre-eminent minds in the field (Lecun, Hinton etc...) and are actively working on this along with unsupervised learning. These companies are either funding university labs that are developing this stuff or doing it in house.
trhway 10y ago
>Right now, for instance, a computer would need to see thousands of different images of a new animal to recognize it consistently, whereas a human only needs to see a single hand-drawn sketch. valid question here would be - would a fresh human mind perform that well too? Or may be such performance is conditioned upon having at least several years of experience of living on planet Earth and thus having the models of animals and everything else around been built using billions of images that human eyes would have generated during those years? > It's well known that existing machine learning algorithms are extremely inefficient learners, which is why you need enormous datasets and computing resources to be competitive currently. when put into perspective, i.e. compared with the cardinality of the above mentioned set of images human eyes generate and computational performance of the brain - 40K TFlops - computers don't look that inefficient learners to me.

roymurdock 10y ago

Generally agree with OP that ML is one of the last viable frontiers for the current generation of computer tech. We are basically taking the data generated by machines invented decades ago, and using statistics to optimize a few % of productivity points more out of them every year. This optimization applies to humans to - one day you'll be required to have an OBD dongle monitoring your driving data to get car insurance, for instance. Many facets of your life will be monitored and fed into a company's individualized production function or pricing equation. When the economic cake is growing slowly, every last drop of productivity enhancement counts, and ML is doing a good job at shaving a few basis points off basic costs, such as energy (see DeepMind's work on Google's datacenters), transportation (autonomous cars), and human capital (making some simple jobs redundant). That's why everyone is piling into IoT/ML/AI right now. What ML is NOT doing is creating a new paradigm/market/computing platform. It's optimizing old markets and technologies. It's not baking new cakes. We'll see a new platform soon. The cycle of human technological progress has been chugging along steadily for a long time and has sped up exponentially in the past 200 years. I'm sure some new tech will come along soon that will allow us to bake more cakes instead of spreading icing ever-thinner on the ones we already have.

halflings 10y ago
Generally agree that ML is often shaving off a few % after all the low hanging fruits that can be achieved without it have been exploited, but... > What ML is NOT doing is creating a new paradigm/market/computing platform. It's optimizing old markets and technologies. It's not baking new cakes. This is simply not true: there are many products/markets (such as automatic translation, speech recognition) that are simply not possible without this progress in ML.
- pas 9y ago
  What's the paradigm? In translate it replaces a naive translator's job. In speech recognition it gives you a new-ish interface to systems. In prediction and personalization it gives you what you likely want. (Local weather, local traffic info, local news, in the language it knows you understand, in the format you prefer, at the time it think you most likely want it, etc..). It puts some "smart" into things. But translate is just a smart dictionary, it's not a real translator, for that we need stronger AI (something like the Jeopardy playing Watson + Google Knowledge Graph / FreeBase + language translation + it should ask questions if it doesn't understand something). ML is amazing, but it's just a slow march toward more and more adaptive smarts (general intelligence) in a box (hence artificial). And there's probably a tipping point for that. When it can start to learn, or program itself, blablabla... ( https://intelligence.org/2013/04/29/intelligence-explosion-microeconomics-released/ https://intelligence.org/2013/04/29/intelligence-explosion-m... )

romaniv 10y ago

I've been wondering the same thing. It is why I don't like the fact that we need increasingly specialized types of computers to do ML. This is also why I am very interested in research that is not based on deep neural networks, which seem to be especially hungry in terms of training data and hardware power.

Eridrus 9y ago

I think this depends on what you see as the downfall of these previous companies. If you agree with Christensen's theory of disruptive innovation, the issue was that large companies would undervalue small markets that could grow to become large markets. When seen in this light, it would be surprising if ML were a disruptive technology, since it falls more into the "sustaining" category of innovation. Most of the large tech companies were built on technologies that no-one expected to become world-defining companies at the time. With all this focus on ML/AI, it would be very surprising if a small ML/AI shop became huge, rather then getting bought or replicated. Almost by definition, the big companies of the next era will come from the spaces that large companies are ignoring.

AndrewKemendo 9y ago
the big companies of the next era will come from the spaces that large companies are ignoring. My entire point is that they aren't ignoring any spaces, they have all future spaces accounted for by having their tentacles in literally everything possible that could be future tech. Bldinsides come from different industries, so they need to cover their bases from all directions, so that's what they all have done with their projects, products, investments in...AI, Energy, Biology, Transportation, Infrastructure, Waste Disposal, Agriculture the list goes on. The time of the non-ubiquitous firm is over. AFGAM aren't software companies, they are innovation conglomerates. I mean why do you think google changed to Alphabet?
- Eridrus 9y ago
  Right, but this has less to do with the fundamentals of ML as a technology, rather than the less sexy explanation of better management practices.
  - AndrewKemendo 9y ago
    They are inextricably linked. If you organize your company around ML, as Google, Facebook, Amazon, Microsoft and Apple have stated they are doing, then yes it does have to do with the fundamentals of ML. Advances and implementations of ML is driving the creation and iteration of products for these companies, as well as helping them run their companies and evaluate where to go next - follow the data as it were. So it is a critical point that every one of their CEOs has outlined in clear detail over the past few years.