3 ms·
Longer term computation isn’t really the limiting factor for generative AI, it’s training data. Generative AI is like Google search before the web responded to
by Retric 2y ago
Longer term computation isn’t really the limiting factor for generative AI, it’s training data. Generative AI is like Google search before the web responded to their search engine existing. There’s a huge quantity of high quality training data which nobody had any reason to pollute ready for the scrapping.
But modern search is hampered by people responding to algorithmic indexes. Algorithms responding to metadata without directly evaluating content enabled a world of SEO and low quality websites suddenly being discoverable as long as they narrow their focus enough.
So longer term it’s going to be an arms race between the output of Generative AI and people trying to keep updating their models. In 20 years people will get much better at using these tools, but the tools themselves may be less useful. I wouldn’t be surprised if eventually someone sneaks advertising into the output of someone else’s model etc.
- Miraste 2y agoThis has already happened. Search google for a few random terms, and go through the first page of web and image results. A decent chunk will be AI-generated.
- deleted 2y ago[deleted]
- golol 2y agoI disagree. With more computation you can train a bigger model on the same size training data and it will be better. There is a lot if knowledge on the internet that GPT-4 etc. have not yet learned.
- Retric 2y agoThe issue is the training data isn’t some constant. Let’s suppose OpenAI had 10x the computing power but a vastly worse dataset, do you expect a better or worse result? The question is ambiguous without defining how much worse the dataset is.
- golol 2y agoBut why would the dataset be worse, they can just use the same one as before.
- Retric 2y agoA dataset that’s 2 years old is worse than one that’s 20 years old even if it contains the same data. Even facts age. In 2004, Pluto was still classified as a planet. Not such a big deal on its own, but stale data gets a little bit worse every day.