4 ms·

> Left unsaid in this piece is that OpenAI likely would have to increase parameters Maybe true, but he also said "We are not here to jerk ourselves off about p

by ftxbro 4y ago

> Left unsaid in this piece is that OpenAI likely would have to increase parameters Maybe true, but he also said "We are not here to jerk ourselves off about parameter count" https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward/ https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo... Also, who says that the "transformer scaling laws" are the ultimate arbiter of LLM scaling? They overturned previous scaling laws and other scaling laws might overturn them. Furthermore, it's even possible that the transformer model won't even be used in later models. I remember Ilya making the point that just because the transformer model was the first one that looks like it can scale intelligence just by lighting up billions of dollars of GPUs, it doesn't mean it's the last one. Maybe it will even be like, the vacuum tube of AI models, and other ones are being made in secret. A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer.

guyomes 4y ago
> A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer This reminds me a TV interview of the author Patrick Modiano, just after he won the literature Nobel price. The presenter asked him if the money would help. The author answered essentially that the next time he would be in front of a white page, the money surely wouldn't help. In the case of surpassing transformers, money could help to give access to more compute power. It could also help to prevent the research from being public.
- 0xfffafaCrash 4y ago
  If someone is already working on a problem full-time, money only helps to the extent that resources they can be buy with money are the limiting constraint. However, beyond deep work needed for a single individual, when you need to explore potential opportunities in a broad space of possibilities, money can hugely effect the search of that space because work needed for major breakthroughs remains parallelizable. You can delegate subtasks to people if you can afford those people. You can hire more of the few specialized people who know about a niche to work on your problem instead of other problems. You can exploit synergies from crosspollination of ideas from bringing together brilliant minds into the same conversations. The influx of money is very very likely to increase the pace of innovation in AI. The breadth of possible avenues for breakthroughs is largely yet-to-be-explored.
- ohgodplsno 4y ago
  Modiano is a rich man, born into a rich family. Wealth doesn't help in front of a white page, but it sure helps being able to stay in front of that white page instead of having to go take up a job because you're not sure what you're eating tonight. As always, wealthy people and their "money doesn't make happiness" bullshit.
  - harvey9 4y ago
    Since he already didn't need to work another job to pay the bills, the extra money from the Nobel prize does not make a difference in this case as he can already put all his time into writing.
  - sitkack 4y ago
    Money doesn't buy happiness, but it staves off abject sadness.
tootie 4y ago
I'm not an expert but isn't size the distinguishing feature of an LLM? It's the first L.
- hiddencost 4y ago
  They needed an architecture that could take advantage of the scale, first. That's what BERT did.
Dylan16807 4y ago
> They overturned previous scaling laws Can you link to a comparison or graph of obsolete and new scaling laws?
- ftxbro 4y ago
  The ones I had in mind were the newer 'Chinchilla' scaling laws (https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf) vs. the older 'Kaplan' scaling laws (https://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/2001.08361.pdf)
  - deleted 4y ago
    [deleted]
jimsimmons 4y ago
Curious if anyone can confirm $5-20M figure. Seems absurdly high but what do I know
- ftxbro 4y ago
  Here was my only source of the rumor: https://news.ycombinator.com/item?id=35565025 https://news.ycombinator.com/item?id=35565025
- Mathnerd314 4y ago
  maybe it's not per-person but the total for a small group. 7 6-figure salaries doesn't seem absurd.
  - jimsimmons 4y ago
    You mean 6 7-figure salaries
- hansvm 4y ago
  Can't confirm OpenAI's position in particular, but $500k/yr/person is table stakes for a decent engineer directly connected to the company's bottom line. Double that for an actual expert, double it again if they're consulting, and put together a team of 3-10 of them. Those numbers aren't too far off.
  - jimsimmons 4y ago
    I can see $5M per person being possible. $20M is the absurd part. 5 years with such a comp leads one to a net worth that is borderline filthy. Like elite, world renown athletes and actors level of wealth. Again, could all be true but just unexpected from my experience.
    - endorphine 4y ago
      $5-20M in total yearly salaries, not per person. I think...
    - 4y ago
imtringued 4y ago
That money won't help unless they get permission to start their own research department.

by ftxbro 4y ago

guyomes 4y ago

> A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer This reminds me a TV interview of the author Patrick Modiano, just after he won the literature Nobel price. The presenter asked him if the money would help. The author answered essentially that the next time he would be in front of a white page, the money surely wouldn't help. In the case of surpassing transformers, money could help to give access to more compute power. It could also help to prevent the research from being public.

0xfffafaCrash 4y ago
If someone is already working on a problem full-time, money only helps to the extent that resources they can be buy with money are the limiting constraint. However, beyond deep work needed for a single individual, when you need to explore potential opportunities in a broad space of possibilities, money can hugely effect the search of that space because work needed for major breakthroughs remains parallelizable. You can delegate subtasks to people if you can afford those people. You can hire more of the few specialized people who know about a niche to work on your problem instead of other problems. You can exploit synergies from crosspollination of ideas from bringing together brilliant minds into the same conversations. The influx of money is very very likely to increase the pace of innovation in AI. The breadth of possible avenues for breakthroughs is largely yet-to-be-explored.
ohgodplsno 4y ago
Modiano is a rich man, born into a rich family. Wealth doesn't help in front of a white page, but it sure helps being able to stay in front of that white page instead of having to go take up a job because you're not sure what you're eating tonight. As always, wealthy people and their "money doesn't make happiness" bullshit.
- harvey9 4y ago
  Since he already didn't need to work another job to pay the bills, the extra money from the Nobel prize does not make a difference in this case as he can already put all his time into writing.
- sitkack 4y ago
  Money doesn't buy happiness, but it staves off abject sadness.

tootie 4y ago

I'm not an expert but isn't size the distinguishing feature of an LLM? It's the first L.

hiddencost 4y ago
They needed an architecture that could take advantage of the scale, first. That's what BERT did.

Dylan16807 4y ago

> They overturned previous scaling laws Can you link to a comparison or graph of obsolete and new scaling laws?

ftxbro 4y ago
The ones I had in mind were the newer 'Chinchilla' scaling laws (https://arxiv.org/pdf/2203.15556.pdf https://arxiv.org/pdf/2203.15556.pdf) vs. the older 'Kaplan' scaling laws (https://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/2001.08361.pdf)
- deleted 4y ago
  [deleted]

jimsimmons 4y ago

Curious if anyone can confirm $5-20M figure. Seems absurdly high but what do I know

ftxbro 4y ago
Here was my only source of the rumor: https://news.ycombinator.com/item?id=35565025 https://news.ycombinator.com/item?id=35565025
Mathnerd314 4y ago
maybe it's not per-person but the total for a small group. 7 6-figure salaries doesn't seem absurd.
- jimsimmons 4y ago
  You mean 6 7-figure salaries
hansvm 4y ago
Can't confirm OpenAI's position in particular, but $500k/yr/person is table stakes for a decent engineer directly connected to the company's bottom line. Double that for an actual expert, double it again if they're consulting, and put together a team of 3-10 of them. Those numbers aren't too far off.
- jimsimmons 4y ago
  I can see $5M per person being possible. $20M is the absurd part. 5 years with such a comp leads one to a net worth that is borderline filthy. Like elite, world renown athletes and actors level of wealth. Again, could all be true but just unexpected from my experience.
  - endorphine 4y ago
    $5-20M in total yearly salaries, not per person. I think...
  - 4y ago

imtringued 4y ago

That money won't help unless they get permission to start their own research department.