6 ms·

Completely wrong, the best AI will win. There is insane demand for better models.

datadrivenangel 3y ago
There is insane demand for good enough models at extremely good prices. Better beyond a certain point is unlikely to be competitive with the cheaper models.
- phillipcarter 3y ago
  Yes, this is correct. And OpenAI are currently the best at this.
oceanplexian 3y ago
Yep, quality over quantity. The difference between 99.9% accurate and 99.999% accurate can be ridiculously valuable in so many real world applications where people would apply LLMs.
gbmatt 3y ago
Only Big Tech (Microsoft,Google,Facebook) can crawl the web at scale because they own the major content companies and they severly throttle the competition's crawlers, and sometimes outright block them. I'm not saying it's impossible to get around, but it is certainly very difficult, and you could be thrown in prison for violating the CFAA.
- PaulHoule 3y ago
  I'm not sure if training on a vast amount of content is really necessary in the sense that linguistic competence and knowledge can probably be separated to some extent. That is, the "ChatGPT" paradigm leads to systems that just confabulate and "makes shit up" and making something radically more accurate means going to something retrieval-based or knowledge graph-based. In that case you might be able to get linguistic competence with a much smaller model that you end up training with a smaller, cleaner, and probably partially synthetic data set.
- Turing_Machine 3y ago
  Common Crawl claims to have 82% of the tokens used to train GPT-3, and it's available to anyone. Add all the downloadable material at archive.org and you've got a formidable corpus. https://commoncrawl.org/ https://commoncrawl.org/
wkat4242 3y ago
The improvements seem to be leveling off already. GPT-4 isn't really worth the extra price to me. It's not that much better. What I would really want though is an uncensored LLM. OpenAI is basically unusable now, most of its replies are like "I'm only a dumb AI and my lawyers don't want me to answer your question". Yes I work in cyber. But it's pretty insane now.
- bugglebeetle 3y ago
  GPT-4, correctly prompted, is head and shoulders above everything for coding. All the text generation stuff and NLP tasks, it’s a toss-up.
- jrockway 3y ago
  I haven't played with the self-hosted LLMs at all yet, but back when Stable Diffusion was brand new I had a ton of fun creating images that lawyers wouldn't want you to create. ("Abraham Lincoln and Donald Trump riding a battle elephant." It's just so much funnier with living people!) I imagine that Llama-2 and friends offer a similar experience.
- leeeeeepw 3y ago
  [dead]
- Turing_Machine 3y ago
  It's way better at writing code, or at least I've found it to be so.
PaulHoule 3y ago
Depends how you define quality. This paper reflects my own experience https://arxiv.org/abs/2305.08377 https://arxiv.org/abs/2305.08377 and shows how LLM technology has a lot more to offer than "ChatGPT". The real takeaway is that by training LLMs with real training data (even with a "less powerful" model) you can get an error rate more than 10x less than you get with the "zero shot" model of asking ChatGPT to answer a question for you the same way that Mickey Mouse asked the broom to clean up for him in Fantasia. The "few-shot" approach of supplying a few examples in the attention window was a little better but not much. The problem isn't something that will go away with a more powerful model because the problem has a lot to do with the intrinsic fuzziness of language. People who are waiting for an exponentially more expensive ChatGPT-5 to save them will be pushing a bubble around under a rug endlessly while the grinds who formulate well-defined problems and make training sets will actually cross the finish line. Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people. LLMs have some possibility of getting cheaper from a software perspective as we understand how they work and hardware can be better optimized to make the most of those transistors, but the driving force of the semiconductor revolution is spent unless people find some entirely different way to build chips. But... people really want to be like Mickey in Fantasia and hope the grinds are going to make magic for them.
- sbierwagen 3y ago
  > Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people. I am unconvinced by the idea of trying to redefine Moore's Law to be about MSRP. The NVIDIA H100 has twice the FLOPS of the A100 on a smaller die. That's Moore's Law, full stop. When NVIDIA has useful competition in the AI space, they'll be forced to cut prices, as has reliably been the case for every semiconductor vendor for the last 60 years.
  - idonotknowwhy 3y ago
    Agreed. We need Intel to get the software side of ARC together. Or we need something like the unified ram or apple silicon. Apple have accidental stumbled into being the most competitive way to run llms with their 192gb studio.
90-00-09 3y ago
It's too early to say who is winning/will win, of course. But so far the UI and its accessibility have made a huge difference in how different gen AI models are being used. For example, I struggle to see DALL-E winning over Firefly if Firefly is integrated into a very rich environment, whereas DALL-E is basically prompt UI only (while DALL-E 3 is a better model IMO).

There is insane demand for good enough models at extremely good prices. Better beyond a certain point is unlikely to be competitive with the cheaper models.

phillipcarter 3y ago
Yes, this is correct. And OpenAI are currently the best at this.

oceanplexian 3y ago

Yep, quality over quantity. The difference between 99.9% accurate and 99.999% accurate can be ridiculously valuable in so many real world applications where people would apply LLMs.

gbmatt 3y ago

Only Big Tech (Microsoft,Google,Facebook) can crawl the web at scale because they own the major content companies and they severly throttle the competition's crawlers, and sometimes outright block them. I'm not saying it's impossible to get around, but it is certainly very difficult, and you could be thrown in prison for violating the CFAA.

PaulHoule 3y ago
I'm not sure if training on a vast amount of content is really necessary in the sense that linguistic competence and knowledge can probably be separated to some extent. That is, the "ChatGPT" paradigm leads to systems that just confabulate and "makes shit up" and making something radically more accurate means going to something retrieval-based or knowledge graph-based. In that case you might be able to get linguistic competence with a much smaller model that you end up training with a smaller, cleaner, and probably partially synthetic data set.
Turing_Machine 3y ago
Common Crawl claims to have 82% of the tokens used to train GPT-3, and it's available to anyone. Add all the downloadable material at archive.org and you've got a formidable corpus. https://commoncrawl.org/ https://commoncrawl.org/

wkat4242 3y ago

The improvements seem to be leveling off already. GPT-4 isn't really worth the extra price to me. It's not that much better. What I would really want though is an uncensored LLM. OpenAI is basically unusable now, most of its replies are like "I'm only a dumb AI and my lawyers don't want me to answer your question". Yes I work in cyber. But it's pretty insane now.

bugglebeetle 3y ago
GPT-4, correctly prompted, is head and shoulders above everything for coding. All the text generation stuff and NLP tasks, it’s a toss-up.
jrockway 3y ago
I haven't played with the self-hosted LLMs at all yet, but back when Stable Diffusion was brand new I had a ton of fun creating images that lawyers wouldn't want you to create. ("Abraham Lincoln and Donald Trump riding a battle elephant." It's just so much funnier with living people!) I imagine that Llama-2 and friends offer a similar experience.
leeeeeepw 3y ago
[dead]
Turing_Machine 3y ago
It's way better at writing code, or at least I've found it to be so.

PaulHoule 3y ago

Depends how you define quality. This paper reflects my own experience https://arxiv.org/abs/2305.08377 https://arxiv.org/abs/2305.08377 and shows how LLM technology has a lot more to offer than "ChatGPT". The real takeaway is that by training LLMs with real training data (even with a "less powerful" model) you can get an error rate more than 10x less than you get with the "zero shot" model of asking ChatGPT to answer a question for you the same way that Mickey Mouse asked the broom to clean up for him in Fantasia. The "few-shot" approach of supplying a few examples in the attention window was a little better but not much. The problem isn't something that will go away with a more powerful model because the problem has a lot to do with the intrinsic fuzziness of language. People who are waiting for an exponentially more expensive ChatGPT-5 to save them will be pushing a bubble around under a rug endlessly while the grinds who formulate well-defined problems and make training sets will actually cross the finish line. Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people. LLMs have some possibility of getting cheaper from a software perspective as we understand how they work and hardware can be better optimized to make the most of those transistors, but the driving force of the semiconductor revolution is spent unless people find some entirely different way to build chips. But... people really want to be like Mickey in Fantasia and hope the grinds are going to make magic for them.

sbierwagen 3y ago
> Remember that Moore's Law is over in the sense that transistors are not getting cheaper generation after generation, that is why the NVIDIA 40xx series is such a disappointment to most people. I am unconvinced by the idea of trying to redefine Moore's Law to be about MSRP. The NVIDIA H100 has twice the FLOPS of the A100 on a smaller die. That's Moore's Law, full stop. When NVIDIA has useful competition in the AI space, they'll be forced to cut prices, as has reliably been the case for every semiconductor vendor for the last 60 years.
- idonotknowwhy 3y ago
  Agreed. We need Intel to get the software side of ARC together. Or we need something like the unified ram or apple silicon. Apple have accidental stumbled into being the most competitive way to run llms with their 192gb studio.

90-00-09 3y ago

It's too early to say who is winning/will win, of course. But so far the UI and its accessibility have made a huge difference in how different gen AI models are being used. For example, I struggle to see DALL-E winning over Firefly if Firefly is integrated into a very rich environment, whereas DALL-E is basically prompt UI only (while DALL-E 3 is a better model IMO).