3 ms·

Of course, it's nothing else. Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to fig

by Mo3 2y ago

Of course, it's nothing else. Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to figure out they could also do it with $500.

whimsicalism 2y ago
it would have been a lot cheaper for oai if they had access to llama3 in 2018
KorematsuFredt 2y ago
You have clearly not read the article. $500 is the cost of fine tuning.
- selcuka 2y ago
  Fair enough. Is it now safe to say that OpenAI could have done with a 8B model + $500 of fine tuning instead of running a (much) larger model on their GPU cluster?
  - wrycoder 2y ago
    Maybe they did
bilbo0s 2y ago
Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to figure out they could also do it with $500. People upvoting the post?? Not really sure? But PT Barnum said there's always a lot of them out there. Pretty sure they mean fine tuning though? But even that is total tripe. These guys are snake oil salesmen. (Or Sylvester McMonkey McBean is behind it.)
nomel 2y ago
It's llama 3 training cost + their cost. Meta "kindly" covered the first $700M. > We add a vision encoder to Llama3 8B
- lanceflt 2y ago
  They didn't train the vision encoder either, it's unchanged SigLIP by Google.
  - qeternity 2y ago
    “We finetuned billions of dollars of research by Google and Meta.”
deleted 2y ago
[deleted]
nickpsecurity 2y ago
While that may be true, the opposite has also happened to hundreds of companies in other areas: https://news.ycombinator.com/item?id=39136472 https://news.ycombinator.com/item?id=39136472 Many companies also optimize for tools, like Python, that have boost productivity more than price/performance ratio. OpenAI had billions of other people's money. They might just keep using tools which worked before. Lastly, there are tons of papers published on techniques that claim to reduce cost. Most of them aren't good. Their benchmarks aren't good. Even reviewing most of them is more time than a lot of AI researchers have. Those that make it to established communities usually have gotchas that come with the benefits. So, they could also simply miss a needle in a large haystack. I think you're right that they'd be using whatever really worked with no loss in model performance. It's just that they might not for a number of reasons. The rational choice is for others to keep experimenting with those things in case they get a competitive advantage.

by Mo3 2y ago

whimsicalism 2y ago

it would have been a lot cheaper for oai if they had access to llama3 in 2018

KorematsuFredt 2y ago

You have clearly not read the article. $500 is the cost of fine tuning.

selcuka 2y ago
Fair enough. Is it now safe to say that OpenAI could have done with a 8B model + $500 of fine tuning instead of running a (much) larger model on their GPU cluster?
- wrycoder 2y ago
  Maybe they did

bilbo0s 2y ago

Who could possibly believe that OpenAI and others would dump billions into development and training and aren't smart enough to figure out they could also do it with $500. People upvoting the post?? Not really sure? But PT Barnum said there's always a lot of them out there. Pretty sure they mean fine tuning though? But even that is total tripe. These guys are snake oil salesmen. (Or Sylvester McMonkey McBean is behind it.)

nomel 2y ago

It's llama 3 training cost + their cost. Meta "kindly" covered the first $700M. > We add a vision encoder to Llama3 8B

lanceflt 2y ago
They didn't train the vision encoder either, it's unchanged SigLIP by Google.
- qeternity 2y ago
  “We finetuned billions of dollars of research by Google and Meta.”

deleted 2y ago

[deleted]

nickpsecurity 2y ago

While that may be true, the opposite has also happened to hundreds of companies in other areas: https://news.ycombinator.com/item?id=39136472 https://news.ycombinator.com/item?id=39136472 Many companies also optimize for tools, like Python, that have boost productivity more than price/performance ratio. OpenAI had billions of other people's money. They might just keep using tools which worked before. Lastly, there are tons of papers published on techniques that claim to reduce cost. Most of them aren't good. Their benchmarks aren't good. Even reviewing most of them is more time than a lot of AI researchers have. Those that make it to established communities usually have gotchas that come with the benefits. So, they could also simply miss a needle in a large haystack. I think you're right that they'd be using whatever really worked with no loss in model performance. It's just that they might not for a number of reasons. The rational choice is for others to keep experimenting with those things in case they get a competitive advantage.