4 ms·

AI is just evolution of Big Data. It's not transformative, like the Internet was. The Internet was a new medium. A more advanced Hadoop will not change your li

AI is just evolution of Big Data. It's not transformative, like the Internet was. The Internet was a new medium. A more advanced Hadoop will not change your life. It will not change the way you read, listen, watch, communicate (fundamentally). The Internet did. Especially at a time when the "legacy" internet is falling apart, a new way to find information is rather a necessity than some breakthrough. The "good" AI needs to fight it out with the "bad" AI before we see if the net positive is even there. A podcast can be translated by AI into a different language, you say? The price is super-realistic fraudulent phone and video calls from your "family".

tomrod 3y ago
I'd argue this take is a bit dated, as if you're looking at ML/AI in terms of training of data. If you look at the enablement of features, AI is a new (and, currently, often flaky) medium. It would be analogous to looking at the Internet as limited to making it simpler to order from a catalog over the phone. While technically true, there is more that it can do besides. Much like we don't use copper in our walls for dialup anymore, I don't anticipate the initial architectures of LLM and MoE to last terribly long, but they do enable proving that the concept works.
- chewxy 3y ago
  Current AI/ML is in fact a reflection on training data. Change the manifold, and change the response. It's unfortunate that of course the data is the entire internet.
  - Eisenstein 3y ago
    Can you please explain what that means? I'm not sure I get it.
    - chewxy 3y ago
      GP mentioned that the current slate of transformer based AIs are not transformative in the same way the Internet was. Rather it's more of a triumph of data engineering practices. OP disagrees with GP. OP's main thesis is that AI enables a lot new applications. OP claims that GP is simply looking at it as if it were training data. I stated that current AI techniques ARE indeed just reflections of the data used in training. I agree with GP that the current "AI"s are simply not transformative in the same way the Internet was. If you change the training data for the current generation of AI, you get different behaviours. The training data forms a manifold - which you can think of as a landscape with features forming valleys and hills. What the current generation of AI does is that it tries to find a shape that fits the landscape - think of it like taking a very large sheet of cloth to cover a landscape. The stiffer the cloth, the less well the cloth fits to the landscape. The "stiffness" of the cloth is the amount of parameters that a neural network has. Modern deep nets are highly overparameterized - imagine a very soft pliable cloth - of course it fits to a landscape well. So if you have a different training data - the neural network will fit to this different landscape as well. Hence the response will be different. It's unfortunate that the training data is the entire internet for a few reasons: 1. Only the rich can train a vaguely competent AI. You're at the whims of those well-resourced enough. 2. There's no "alternate" training dataset anymore. (Though a clever thing people at OpenAI are doing are Mixture of Experts models, where you train multiple NNs using different subsets of the full training set, so you get multiple competencies)
      - Eisenstein 3y ago
        But you are specifically talking about one type of AI, which is a generative language model. There are tons of other AIs with different applications that do not need to be trained on the entire internet. You have computer vision which separates in object recognition, classification, OCR, etc; you have audio which has text-to-speech (and reverse), music generation, and all sorts of other things; machine translation; sentiment analysis (I won't list all the categories in hugging face but you get my point). These are not differentiated merely by 'training data' to my understanding, so that's why your comment didn't make sense to me. Calling all AI LLMs is like calling all of the internet the web. Of course if I am mistaken, corrections are welcome.
  - tomrod 3y ago
    In my opinion, your response is tautological and not related to the point I made that existing AI is good enough to start building applications and functionality.
    - chewxy 3y ago
      It was good enough in 2015/2016 for me to run a startup that allowed people to program in natural language. We even had paying clients though eventually none could stomach the $2000 per month for incremental/on-line training costs. The only real difference between then and now is that OpenAI's models are significantly better than my models from 2015, and they have that because well, they can afford to pile on more data. TBH, I never even considered using a large proportion of the whole internet as a training set as even remotely possible due to the sheer mind boggling costs. Even now, to go through about 10% of The Pile would cost me way too much money.
      - renegade-otter 3y ago
        And it's not really cost effective - as well as being an epicenter of culture wars. "Your AI is woke! Your AI is fascist!". THIS part of the AI sector is just a giant pyramid scheme - impress the investors so they shovel trillions your way. That's not exactly new in Silicon Valley - keep the valuation of a hot potato going up until someone is left holding the bag. AI's most useful applications are not being a generalist.
kingraoul 3y ago
There’s another way to look at this - as the beginning of the construction of the Data Mesh: https://martinfowler.com/articles/data-mesh-principles.html https://martinfowler.com/articles/data-mesh-principles.html This can be seen as a “shift left” in data capabilities. The rise of Data Products ultimately democratizes access to data.

tomrod 3y ago

I'd argue this take is a bit dated, as if you're looking at ML/AI in terms of training of data. If you look at the enablement of features, AI is a new (and, currently, often flaky) medium. It would be analogous to looking at the Internet as limited to making it simpler to order from a catalog over the phone. While technically true, there is more that it can do besides. Much like we don't use copper in our walls for dialup anymore, I don't anticipate the initial architectures of LLM and MoE to last terribly long, but they do enable proving that the concept works.

chewxy 3y ago
Current AI/ML is in fact a reflection on training data. Change the manifold, and change the response. It's unfortunate that of course the data is the entire internet.
- Eisenstein 3y ago
  Can you please explain what that means? I'm not sure I get it.
  - chewxy 3y ago
    GP mentioned that the current slate of transformer based AIs are not transformative in the same way the Internet was. Rather it's more of a triumph of data engineering practices. OP disagrees with GP. OP's main thesis is that AI enables a lot new applications. OP claims that GP is simply looking at it as if it were training data. I stated that current AI techniques ARE indeed just reflections of the data used in training. I agree with GP that the current "AI"s are simply not transformative in the same way the Internet was. If you change the training data for the current generation of AI, you get different behaviours. The training data forms a manifold - which you can think of as a landscape with features forming valleys and hills. What the current generation of AI does is that it tries to find a shape that fits the landscape - think of it like taking a very large sheet of cloth to cover a landscape. The stiffer the cloth, the less well the cloth fits to the landscape. The "stiffness" of the cloth is the amount of parameters that a neural network has. Modern deep nets are highly overparameterized - imagine a very soft pliable cloth - of course it fits to a landscape well. So if you have a different training data - the neural network will fit to this different landscape as well. Hence the response will be different. It's unfortunate that the training data is the entire internet for a few reasons: 1. Only the rich can train a vaguely competent AI. You're at the whims of those well-resourced enough. 2. There's no "alternate" training dataset anymore. (Though a clever thing people at OpenAI are doing are Mixture of Experts models, where you train multiple NNs using different subsets of the full training set, so you get multiple competencies)
    - Eisenstein 3y ago
      But you are specifically talking about one type of AI, which is a generative language model. There are tons of other AIs with different applications that do not need to be trained on the entire internet. You have computer vision which separates in object recognition, classification, OCR, etc; you have audio which has text-to-speech (and reverse), music generation, and all sorts of other things; machine translation; sentiment analysis (I won't list all the categories in hugging face but you get my point). These are not differentiated merely by 'training data' to my understanding, so that's why your comment didn't make sense to me. Calling all AI LLMs is like calling all of the internet the web. Of course if I am mistaken, corrections are welcome.
- tomrod 3y ago
  In my opinion, your response is tautological and not related to the point I made that existing AI is good enough to start building applications and functionality.
  - chewxy 3y ago
    It was good enough in 2015/2016 for me to run a startup that allowed people to program in natural language. We even had paying clients though eventually none could stomach the $2000 per month for incremental/on-line training costs. The only real difference between then and now is that OpenAI's models are significantly better than my models from 2015, and they have that because well, they can afford to pile on more data. TBH, I never even considered using a large proportion of the whole internet as a training set as even remotely possible due to the sheer mind boggling costs. Even now, to go through about 10% of The Pile would cost me way too much money.
    - renegade-otter 3y ago
      And it's not really cost effective - as well as being an epicenter of culture wars. "Your AI is woke! Your AI is fascist!". THIS part of the AI sector is just a giant pyramid scheme - impress the investors so they shovel trillions your way. That's not exactly new in Silicon Valley - keep the valuation of a hot potato going up until someone is left holding the bag. AI's most useful applications are not being a generalist.

kingraoul 3y ago

There’s another way to look at this - as the beginning of the construction of the Data Mesh: https://martinfowler.com/articles/data-mesh-principles.html https://martinfowler.com/articles/data-mesh-principles.html This can be seen as a “shift left” in data capabilities. The rise of Data Products ultimately democratizes access to data.