2 ms·

The estimation I did 4 months ago: > there are approximately 200k common nouns in English, and then we square that, we get 40 billion combinations. At one sec

The estimation I did 4 months ago: > there are approximately 200k common nouns in English, and then we square that, we get 40 billion combinations. At one second per, that's ~1200 years, but then if we parallelize it on a supercomputer that can do 100,000 per second that would only take 3 days. Given that ChatGPT was trained on all of the Internet and every book written, I'm not sure that still seems infeasible. https://news.ycombinator.com/item?id=45455786 https://news.ycombinator.com/item?id=45455786

eli 9mo ago
How would you generate a picture of Noun + Noun in the first place in order to train the LLM with what it would look like? What's happening during that 1 estimated second?
- Terretta 9mo ago
  This is why everyone trains their LLM on another LLM. It's all about the pelicans.
- metalliqaz 9mo ago
  its pelicans all the way down
- fragmede 9mo ago
  Use any of the image generation models (eg Nanobanana, Midjourney, or ChatGPT) to generate a picture of a noun on a noun. Simonw's test is to have a Language (text) model generate a Scalar Vector Graphic, which the language model has to do by writing curves and colors, like draw a spline from point 150,100 to 200,300 of type cubic, using width 20, color orange. In that hypothetical second is freaking fascinating. It's a denoising algorithm, and then a bunch of linear algebra, and out pops a picture of a pelican on a bicycle. Stable diffusion does this quite handily. https://stablediffusionweb.com/image/6520628-pelican-bicycle-ride https://stablediffusionweb.com/image/6520628-pelican-bicycle...
AnimalMuppet 9mo ago
But you need to also include the number of prepositions. "A pelican on a bicycle" is not at all the same as "a pelican inside a bicycle". There are estimated to be 100 or so prepositions in English. That gets you to 4 trillion combinations.
- jodrellblank 9mo ago
  The prompt was "a pelican riding a bicycle"; not prepositions but every verb. Potentially every adverb+verb combination - "a pelican clumsily pushing a bicycle"

eli 9mo ago

How would you generate a picture of Noun + Noun in the first place in order to train the LLM with what it would look like? What's happening during that 1 estimated second?

Terretta 9mo ago
This is why everyone trains their LLM on another LLM. It's all about the pelicans.
metalliqaz 9mo ago
its pelicans all the way down
fragmede 9mo ago
Use any of the image generation models (eg Nanobanana, Midjourney, or ChatGPT) to generate a picture of a noun on a noun. Simonw's test is to have a Language (text) model generate a Scalar Vector Graphic, which the language model has to do by writing curves and colors, like draw a spline from point 150,100 to 200,300 of type cubic, using width 20, color orange. In that hypothetical second is freaking fascinating. It's a denoising algorithm, and then a bunch of linear algebra, and out pops a picture of a pelican on a bicycle. Stable diffusion does this quite handily. https://stablediffusionweb.com/image/6520628-pelican-bicycle-ride https://stablediffusionweb.com/image/6520628-pelican-bicycle...

AnimalMuppet 9mo ago

But you need to also include the number of prepositions. "A pelican on a bicycle" is not at all the same as "a pelican inside a bicycle". There are estimated to be 100 or so prepositions in English. That gets you to 4 trillion combinations.

jodrellblank 9mo ago
The prompt was "a pelican riding a bicycle"; not prepositions but every verb. Potentially every adverb+verb combination - "a pelican clumsily pushing a bicycle"