4 ms·

We aren't at the technological limit, unless you are talking about pre-training a state of the art model as a private citizen, in which case that limit has been

by ftxbro 3y ago

We aren't at the technological limit, unless you are talking about pre-training a state of the art model as a private citizen, in which case that limit has been reached a long time ago. We are also past the 'open research' limit where corporations are no longer publishing all of the details of their model architecture and training regimen. We might be at the social or political limit where companies aren't allowed to just release inference engines with superhuman levels of cognition for the public to use. We also might be at the limit of what normal people can superficially recognize as improvement. For example your pet parakeet can't tell the difference between GPT-2 and GPT-3 and maybe most people won't be able to superficially tell the difference between GPT-6 and GPT-7 even if the difference is huge and has quadrillion dollar effects behind the scenes.

lweitrju39p4rj 3y ago
companies are not publishing anything because they have nothing to publish. not because they are inventing anything relevant. The article is spot on. It is just a better-mechanical-turk that spew things that were already written but screwing some words on happy little accidents. The image generation are more impressive because it is harder to judge how those things fall into place. but it's mostly a perception thing versus text/code. Nobody publishes anything because the only edges now are access to better training data (which would be better monetized making a better google at this point) or tweaking the constraints (the new version of big-data-analyst pool of jobs, now titled as prompt-engineer or "scientist" where all you do is add more layers to the opponent model)
- ftxbro 3y ago
  > companies are not publishing anything because they have nothing to publish This is just very uninformed. Before GPT-4 and the 'AI race' the various LLM-adjacent research was usually done by teams with members in both academia and industry and they published all their details as a point of pride and to get prestige and high value publication venues. After GPT-4 and the 'AI race' this immediately stopped. Here is one example where OpenAI deliberately didn't publish details of GPT-4. Instead of publishing a 'research paper' they published a 'technical report' https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774 "Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar" When Google published their rival PaLM 2 results, they followed the OpenAI GPT-4 lead by publishing only a 'technical report' https://ai.google/static/documents/palm2techreport.pdf https://ai.google/static/documents/palm2techreport.pdf that looks mainly at the capabilities of the system and not at the details of 'the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.'
PaulHoule 3y ago
I agree with the OP. People who haven’t been personally involved with it tend to think science and technology progresses exponentially but if you have been personally involved you went through uncomfortable lurches divided by long periods of little or no progress. For instance supersonic transports, fast breeder reactors, space travel, neural networks and many technologies have been through waves of fast progress, even being areas of major international competition, followed by long periods of collapsed interest. If you get a PhD in the subject for a range of years you might get tenure, otherwise you wind up doing something in industry or driving a cab. (One of many realities that makes a mockery of ‘meritocracy’; you could just barely get a PhD when a field is hot and you will become a titan of the field, but Einstein, Feynman or somebody like that could work as hard and as smart as they can in theoretical physics in 2023, come up with many ideas about what the dark matter particle is, but no Nobel Prize because they can’t do an experiment to prove it.) In the late 1960s and 1970s for instance there was impressive progress on symbolic A.I., people found that logic-based methods could be used to make ‘expert systems’ covering limited domains, but those weren’t really competitive with MATLAB, programmable calculators and other tools experts could use to put their skills on wheels. ‘Business rules engines’ are still around and actually orders of magnitude better performing than expert system shells at their peak but outside of banks, nobody cares. I don’t think we’re done with LLMs yet, i think there is going to be a lot of progress on reducing their resource consumption. They will be used for many things but 20 years from now people might be looking at them the way we look at ELIZA today.

by ftxbro 3y ago

lweitrju39p4rj 3y ago

companies are not publishing anything because they have nothing to publish. not because they are inventing anything relevant. The article is spot on. It is just a better-mechanical-turk that spew things that were already written but screwing some words on happy little accidents. The image generation are more impressive because it is harder to judge how those things fall into place. but it's mostly a perception thing versus text/code. Nobody publishes anything because the only edges now are access to better training data (which would be better monetized making a better google at this point) or tweaking the constraints (the new version of big-data-analyst pool of jobs, now titled as prompt-engineer or "scientist" where all you do is add more layers to the opponent model)

ftxbro 3y ago
> companies are not publishing anything because they have nothing to publish This is just very uninformed. Before GPT-4 and the 'AI race' the various LLM-adjacent research was usually done by teams with members in both academia and industry and they published all their details as a point of pride and to get prestige and high value publication venues. After GPT-4 and the 'AI race' this immediately stopped. Here is one example where OpenAI deliberately didn't publish details of GPT-4. Instead of publishing a 'research paper' they published a 'technical report' https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774 "Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar" When Google published their rival PaLM 2 results, they followed the OpenAI GPT-4 lead by publishing only a 'technical report' https://ai.google/static/documents/palm2techreport.pdf https://ai.google/static/documents/palm2techreport.pdf that looks mainly at the capabilities of the system and not at the details of 'the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.'

PaulHoule 3y ago

I agree with the OP. People who haven’t been personally involved with it tend to think science and technology progresses exponentially but if you have been personally involved you went through uncomfortable lurches divided by long periods of little or no progress. For instance supersonic transports, fast breeder reactors, space travel, neural networks and many technologies have been through waves of fast progress, even being areas of major international competition, followed by long periods of collapsed interest. If you get a PhD in the subject for a range of years you might get tenure, otherwise you wind up doing something in industry or driving a cab. (One of many realities that makes a mockery of ‘meritocracy’; you could just barely get a PhD when a field is hot and you will become a titan of the field, but Einstein, Feynman or somebody like that could work as hard and as smart as they can in theoretical physics in 2023, come up with many ideas about what the dark matter particle is, but no Nobel Prize because they can’t do an experiment to prove it.) In the late 1960s and 1970s for instance there was impressive progress on symbolic A.I., people found that logic-based methods could be used to make ‘expert systems’ covering limited domains, but those weren’t really competitive with MATLAB, programmable calculators and other tools experts could use to put their skills on wheels. ‘Business rules engines’ are still around and actually orders of magnitude better performing than expert system shells at their peak but outside of banks, nobody cares. I don’t think we’re done with LLMs yet, i think there is going to be a lot of progress on reducing their resource consumption. They will be used for many things but 20 years from now people might be looking at them the way we look at ELIZA today.