8 ms·

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

make3 4y ago
small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money
- cypress66 4y ago
  A lot of SOTA comes from small players. It just isn't the case for LLMs.
- awestroke 4y ago
  Good point, but perhaps a leap could take small players into territories of language models that are large enough to be useful. GPT-3 crossed that threshold
- hankman86 4y ago
  Take a leaf from Seti@Home‘s book and try to come up with a distributed, volunteer-based approach to training an open source LLM. There is already an enormous amount of suitable ML hardware on end user devices.
  - Der_Einzige 4y ago
    Huggingface actually recently did this, but I think it's for inference on their giant models like BLOOM
noidiocyallowed 4y ago
Work together and fuck up companies together. That's the way to go.
- abricq 4y ago
  Or how to apply communism to software engineering. I like that. More seriously, the risk that a few companies become even more powerful thanks to their restricted access to such NN is very frightening. The worth is, without legal restrictions, there is nothing that we can do against it. And I doubt that legal restrictions come in the next months / years.
  - beepbooptheory 4y ago
    Well at that point, some people might have the crazy crazy insight that no matter how big the model is, or how many GPUs they have, it burns up all the same.
- deleted 4y ago
  [deleted]
rjtavares 4y ago
Small players should focus on applications of this tech. We now know that whatever AI Models succeed in the future, they'll be trained by a huge company and finetuned to a specific use case. Small companies should be working on use cases, and then just upgrade to the latest SOTA model.
- varispeed 4y ago
  > Small players should focus on applications of this tech. That sounds a bit condescending. We are probably at a point from which the government should intervene and help establish level playing field. Otherwise we are going to see a deeper divide between multibillion businesses conquering multiple markets and sort of neofiefdom situation. This is not good.
  - rjtavares 4y ago
    I'm not being condescending at all, we've learned that the value in AI is in the applications. If you think government should regulate the field, it should be to make AI Models a commodity, like electricity.
  - googlryas 4y ago
    Do you think you'll get a global agreement on this? Or would china just eat America's lunch then?
  - tiborsaas 4y ago
    It's not that condescending, that's todays reality. Should I feel entitled for $600k training time that may or may not work? Do you think the government is a good actor to judge if my qualifications are good enough to grant me resources worth a house? It's quite reasonable to make use of models already trained for small players.
    - mschuster91 4y ago
      > Do you think the government is a good actor to judge if my qualifications are good enough to grant me resources worth a house? Governments already routinely do that for pharmaceutical research or for nuclear (fusion) research. In fact, almost all major impact research and development was funded by the government, mostly the military. Lasers, microwaves, silicon, interconnected computers - all funded by the US tax payer, back in the golden times when you'd get laughed out of the room if you dared think about "small government". And the sums involved were ridiculously larger than the worth of a house. We're talking of billions of dollars. Nowadays, R&D funding is way WAY more complex. Some things like AI or mRNA vaccines are mostly funded by private venture capital, some are funded by large philanthropic donors (e.g. Gates Foundation), some by the inconceivably enormous university endowments, a lot by in-house researchers at large corporations, and a select few by government grants. The result of that complexity: - professors have to spend an absurd percentage of their time "chasing grants" (anecdata, up to 40% [1]) instead of actually doing research - because grants are time-restricted, it's rare to have tenure track any more - because of the time restriction and low grant amounts, it's very hard for the support staff as well. In Germany and Austria, for example, extremely low paid "chain contracts" are common - one contract after another, usually for a year, but sometimes as low as half a year. It's virtually impossible to have a social life if you have to up-root it for every contract because you have to take contracts wherever they are, and forget about starting a family because it's just so damn insecure. The only ones that can make it usually come from highly privileged environments: rich parents or, rarely, partners that can support you. Everyone in academia outside of tenured professors struggles with surviving, and the system ruthlessly grinds people to their bones. It's a disgrace. [1] https://www.johndcook.com/blog/2011/04/25/chasing-grants/ https://www.johndcook.com/blog/2011/04/25/chasing-grants/
isthisthingon99 4y ago
How long does it take to train a human? It's useless for two years then maybe it can tell you it needs to poop. The breakthrough will be developing this equivalent in an accessible manner and us taking care to train the thing for a couple of decades but then it becomes our friend.
- licebmi__at__ 4y ago
  Yes, but to be fair, the system that does the training really sucks and doesn’t scale.
  - cactusplant7374 4y ago
    Neither does OpenAI. It costs so much and still delivers so little. A human can generate breakthroughs in science and tech that can be used to reduce carbon emissions. ChatGPT can do no such thing.
    - VBprogrammer 4y ago
      What percentage of humans make meaningful contributions to advancing science or technology? The overwhelming majority of us are just worker bees servicing the needs of the human population.
      - cactusplant7374 4y ago
        I agree with you on this point. It’s also arguable that less people with a better education system could yield the same result with less environmental impact. But my point, poorly explained, is that whatever ChatGPT is, it isn’t original or creative thought as a human would do it. Chomsky’s example (which is based off Turing): Do submarines swim? Yes, they swim — if that’s what you mean by swimming.
        int_19h 4y ago
        We don't have any clear definitions for "creativity" to begin with. In practice, in these contexts, it seems to be defined as "whatever only humans can do" - that is, the goalposts are automatically moved with every AI advancement.
breck 4y ago
> Are there any possible technologal or scientific leaps on the horizon Yes. From 2017: "Prediction 4: The simplest 2D text encodings for neural networks will be TLs. High level TLs will be found to translate machine written programs into understandable trees." We have something coming out that is an OOM better than anything else out there right now.
spi 4y ago
What do you mean by "small players have no chance"? OpenAI was founded in 2015, it used to be a "small player" which just got things right and grew with it - we're not talking of Google or Facebook investing a chunk of their billions cash. In Germany, AlephAlpha has built their own supercomputer and are training similar sized models. It's expensive for sure, but well in the possibilities of startups. In France researchers trained the similarly sized BLOOM model https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom. They claim it cost between $2 and $4 millions. Sure, a single researcher can't replicate this at their university, but even though OpenAI likes to publish it this way, we're not really talking about research here. Research was inventing the transformer architecture, this is just making it bigger by (very smart) engineering choices. It's something companies should do (and are doing), not researchers.
- awestroke 4y ago
  Microsoft (using Azure DCs) built a supercomputer with 10,000 V100 GPUs exclusively for OpenAI. [0] It is estimated that it cost around $5M in compute time to train GPT-3. OpenAI has received billions in investment prior to launching GPT-3, including $1B from Microsoft in 2019. [0]: https://blogs.microsoft.com/ai/openai-azure-supercomputer/ https://blogs.microsoft.com/ai/openai-azure-supercomputer/
- nileshtrivedi 4y ago
  > we're not talking of Google or Facebook investing a chunk of their billions cash OpenAI had raised $1B from Microsoft in 2019 and used it to train a 175B param model. Now, they have raised $10B and are training GPT-4 with 1.5T params. GPUs are capital intensive and as long as there are returns to bigger models, that's exactly where things will go.
  - andy_ppp 4y ago
    Will 1.5T parameters be possible to run in the public way GPT-3 is? I can’t wait to see what happens with this much learning!
  - awestroke 4y ago
captainmuon 4y ago
I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhythm. Applied to NLP, you could heavily preprocess the input. Tokenize it, annotate parts of speech. Maybe add pronunciation, so the model doesn't have to think about weird english spelling rules, and so you can deal with audio more easily later. So I would build all these little expert-knowledge black boxes and offer them as input to my network. But there is also some inherent resource cost in large language models. If you want to store and process the knowledge of the world, it is going to be expensive no matter what. Maybe we could split the problem into two parts: Understanding language, and world knowledge (with some messy middle ground). I believe you could replace the world knowledge with a huge graph database or triple store. Not just subject-verb-object, but with attribution and certainty numbers for every fact. The idea would be to query the database at inference time. I don't know how to use this in conjunction with a transformer network like GPT-3, so you'd likely need a very different architecture. The big benefit of this would be that it is feasible to train the language part without the world knowledge part with much less resources. But you have other benefits, too. ChatGPT is trained to "win the language game". But as they say, winning the argument does not make you right. If you have a clean fact database, you can have it weigh statements from trustworthy sources higher. You then basically have a nice natural language frontend to a logical reasoning system that can respond with facts (or better: conclusions).
- visarga 4y ago
  Check out DeepMind RETRO, it's one year old already, but exactly what you say: https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens https://www.deepmind.com/publications/improving-language-mod...
- lossolo 4y ago
  It's not just about compute; if that were the case, then models like BLOOM and OPT, which also have 175 billion parameters, would have the same performance for real-world use cases as GPT-3, but they don't. Datasets are also very important.
omeysalvi 4y ago
I think AI is going to go the way of the hard sciences where the age of tinkerers making progress by leaps and bounds in their basement is over and incremental progress is going to be the domain of universities or large private companies that can afford to throw money behind it. I would love to be proven wrong and see radical shifts in how people approach these problems. Seems like the cycle started and got to this point way too soon for AI though
- swalsh 4y ago
  Tinkerers can fine tune a model though. Unfortunately most fine tuning seems to be outmatched at the next iteration of the model.
- mittermayr 4y ago
  My take on this is that (good) content is one of the bigger problems still, particularly also who exactly the original training data belongs to (or where it comes from). There's a certain risk (we'll see with Github CoPilot soon) it will slow down for a bit until the licensing issues are all sorted out. This can only be solved (for now) by bringing in public funding/data, which universities have always been a very good proxy for. Which also means it (usually) should be open access to the public, to some extent (and useful for the garage folks to catch up a bit). But, once we're past that, it'll be all about that giant body of pre-trained data, securely kept within the next Facebook or Microsoft, amounting to literal data gold (just much higher value at a lot less weight).
belter 4y ago
Model size does not necessarily correlates to quality of results. "Chinchilla (70B) Greatly Outperforms GPT-3 (175B) and Gopher (280B)" - https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...
- Der_Einzige 4y ago
  I highly doubt this in practice on a large scale. Outside of the common phenomena of "most large NNs are under trained" and "less better data is sometimes better than more worse data", there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one. I claim instead that we are still hardly scratching the surface with how we evaluate NLP systems. Also, some fields have straight up trash evaluation schemes. Summarization and ROGUE scores are totally BS and I find the claim that they even correlate with high quality summaries suspect. I say this with publications in the that subfield, so I have personal experience with just how crummy many summarizes are.
  - WithinReason 4y ago
    there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one. Overfitting?
    - Der_Einzige 4y ago
      The consensus seems to be that the majority of LMs are undertrained not overfitting though.
- espadrine 4y ago
  An interesting outcome of the nanoGPT repo is this struggle to exactly match the Chinchilla findings[0], even after discussing it with the authors. A larger discussion is that the scaling laws achieve loss-optimal compute time, but the pre-training loss only improves predictions on the corpus, which contains texts written by people that were wrong or whose prose was lacking. In a real system, what you want to optimize for is accuracy, composability, inventiveness. [0]: https://github.com/karpathy/nanoGPT/blob/master/scaling_laws.ipynb https://github.com/karpathy/nanoGPT/blob/master/scaling_laws...
davidy123 4y ago
Could this be distributed? Put all those mining GPUs to work. A lot of people like participating in public projects like this. I would!
- PartiallyTyped 4y ago
  In theory, yes. "Hogwild!" is an approach to distributed training, in essence, each worker is given a bunch of data, they compute the gradient and send that to a central authority. The authority accumulates the gradients and periodically pushes new weights. There is also Federated Learning which seemed to start taking off, but then interest rapidly declined.
- naraga 4y ago
  Exactly. This is inevitable imho. There is no way people will be ok to depend on few wall-gardened models.
- dmit 4y ago
  >> GPT-3 took 355 years to train > Could this be distributed? Put all those mining GPUs to work. Nope. It's a strictly O(n) process. If it weren't for the foresight of George Patrick Turnbull in 1668, we would not be anywhere close to these amazing results today.
  - davidy123 4y ago
    I couldn't find any references to George Patrick Turnbull. If that an ancestor of yours? If so, the comment seems rather subjective.
    - taneq 4y ago
      They're being facetious about the '355 years to train' thing. ;)
      - davidy123 4y ago
        OK haha good one then. Mine was a bit too subtle.
pprotas 4y ago
What does “355 years” mean in this context? I assume it’s not human years
- mellosouls 4y ago
  Claimed here, so this is presumably the reference (355 GPU Years): https://lambdalabs.com/blog/demystifying-gpt-3 https://lambdalabs.com/blog/demystifying-gpt-3 "We are waiting for OpenAI to reveal more details about the training infrastructure and model implementation. But to put things into perspective, GPT-3 175B model required 3.14E23 FLOPS of computing for training. Even at theoretical 28 TFLOPS for V100 and lowest 3 year reserved cloud pricing we could find, this will take 355 GPU-years and cost $4.6M for a single training run. Similarly, a single RTX 8000, assuming 15 TFLOPS, would take 665 years to run."
  - dx034 4y ago
    That's still including margins of cloud vendors. OpenAI had Microsoft providing resources which could do that at much lower cost. It still won't be cheap but you'll be way below $5m if you buy hardware yourself, given that you're able to utilize it long enough. Especially if you set it up in a region with low electricity prices, latency doesn't matter anyway.
- Manfred 4y ago
  Cumulative hours spent across training hardware.
imtringued 4y ago
As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in optimizing deep learning and that NVIDIA has made it very easy to just throw more hardware at then problem.
- visarga 4y ago
  > argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that Yes, but you don't know which 0.5% depending on the input text.
- londons_explore 4y ago
  > very little money in optimizing deep learning Oh - there are a lot of people working on optimizing AI. Amongst hobbyists, academia, and corporations alike. The thing is, if you come up with a neat optimization that saves 30% of compute for the same results, typically instead of reducing your compute budget 30%, you instead increase your model/data size 30% and get better results.
  - narrator 4y ago
    Jevon's paradox of data and AI. The more efficiently data is used, the more demand their is for data.
    - antognini 4y ago
      Any state of the art model takes about three weeks to train.
      - visarga 4y ago
        More an indication of human patience than task difficulty.
naraga 4y ago
I think something of Seti@Home kind will come.
seydor 4y ago
It should be no issue if it became massively parralelized a-la SETI. I wonder when Wikimedia or Apache foundation will jump into AI
- yreg 4y ago
  Wikimedia and other organizations that deal with moderation might want to keep this technology out of the hands of the general public for as long as possible.
xur17 4y ago
Alternatively, are there ways to train on consumer graphics cards, similar to SETI@Home or Folding@Home? I would personally be happy to donate gpu time, as I imagine many others would as well.
- mryab 4y ago
  There absolutely are! Check out hivemind (https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind), a general library for deep learning over the Internet, or Petals (https://petals.ml/ https://petals.ml/), a system that leverages Hivemind and allows you to run BLOOM-176B (or other large language models) that is distributed over many volunteer PCs. You can join it and host some layers of the model by running literally one command on a Linux machine with Docker and a recent enough GPU. Disclaimer: I work on these projects, both are based on our research over the past three years
- alfor 4y ago
  The cost of moving data from one gpu to the next will destroy performance. The system are moving in the opposite direction (look at Dojo architecture or TensTorrent) The silver lining is that the cost of training will fall substantially with those architecture that are not based in reusing gpu.
ignoramous 4y ago
Yes, see DeepMind RETRO: > In our experiments on the Pile, a standard language modeling benchmark, a 7.5 billion parameter RETRO model outperforms the 175 billion parameter Jurassic-1 on 10 out of 16 datasets and outperforms the 280B Gopher on 9 out of 16 datasets. https://www.deepmind.com/blog/improving-language-models-by-retrieving-from-trillions-of-tokens https://www.deepmind.com/blog/improving-language-models-by-r... Though, there hasn't been much follow-up research on it (or DeepMind is not publishing it). Annotated paper: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/papers/2112.04426.pdf https://github.com/labmlai/annotated_deep_learning_paper_imp...
- espadrine 4y ago
  The research is still ongoing, although perhaps lower-profile than what appears in the press. RETRO did get press, but it was not the first retrieval model, and in fact was not SOTA when it got published; FiD was, which later evolved into Atlas[0], published a few months ago. [0]: https://github.com/facebookresearch/atlas https://github.com/facebookresearch/atlas
GistNoesis 4y ago
Yes. There are plenty forward leaps, most of them are not new and are just waiting to be integrated or released : Let's pave the road for SkyNet hard lift-off : -The first obvious one is use of external knowledge store, aka instead of having to store facts in the neural weights where they struggle, just store them in a database and teach your neural network to use it. (This is also similar to something like webgpt where you allow your network to search the web). This will allow you to have a network of 1G parameters (and external indexes of a few TB) that is as performant as a network of 100G parameters, and with better scaling property too. You can probably gain at least 2 orders of magnitude there. -The second leap is better architecture of your neural networks, approximating transformer that are quadratic compute by something that is linear compute (linformer) or n log n compute (reformer) can get you an order of magnitude faster by simply reducing your iteration time. Similarly using some architectures based on sparsity can give you faster computation (although some of the gains are reduced by lesser efficiency of sparse memory access pattern). Using (analog bits) Diffusion to Generatively PreTrain sentences at a time instead of token by token. You can probably gain between 1 and 3 order of magnitude here if you write and optimize everything manually (or have your advanced network/compiler optimize your code for you) -The third leap is reduced domain : You don't have a single network that you train on everything. Training one network by domain allows you to have a smaller network that compute faster. But also it allows you to focus your training on what matters to you : for example if you want to have a mathematics network, its parameters are not influenced a lot by showing it football pictures. There is at least 2 orders of magnitude there. -The fourth one is external tool usage. It's kind of related to the first one but whereas in the first one is readily differentiable, this one necessitate some Reinforcement Learning (that's what decision transformer are used for). -Compression : compress everywhere. The bottlenecks are memory bandwidth related. Work in compressed form when relevant. One order of magnitude -Distributed training : Because the memory bandwidth of inside a GPU is in the order of TB/s where as the transfer to the GPU is in the order of 10GB/s. There is an advantage to have the parameters reside on the GPU but there is a limited quantity of memory in the GPU, so distributed training (something like petals.ml) allows you to increase your memory bandwidth by collaborating. So each actor can probably gain an order of magnitude. Provided that they can keep bad actors away. -Use free resources : The other day Steam had 10M users with GPU waiting around doing nothing, just release a dwarf fortress mod with prettier pictures and use the compute for more important tasks. -Remove any humans in the loop : it's faster to iterate when you don't have to rely any human, either for dataset construction or model building :)
QuesnayJr 4y ago
There are a couple of cases where small changes in the model make training much quicker. For example, the currently leading Go AI, KataGo, requires much less time to train than AlphaGo did.

make3 4y ago

small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money

cypress66 4y ago
A lot of SOTA comes from small players. It just isn't the case for LLMs.
awestroke 4y ago
Good point, but perhaps a leap could take small players into territories of language models that are large enough to be useful. GPT-3 crossed that threshold
hankman86 4y ago
Take a leaf from Seti@Home‘s book and try to come up with a distributed, volunteer-based approach to training an open source LLM. There is already an enormous amount of suitable ML hardware on end user devices.
- Der_Einzige 4y ago
  Huggingface actually recently did this, but I think it's for inference on their giant models like BLOOM

noidiocyallowed 4y ago

Work together and fuck up companies together. That's the way to go.

abricq 4y ago
Or how to apply communism to software engineering. I like that. More seriously, the risk that a few companies become even more powerful thanks to their restricted access to such NN is very frightening. The worth is, without legal restrictions, there is nothing that we can do against it. And I doubt that legal restrictions come in the next months / years.
- beepbooptheory 4y ago
  Well at that point, some people might have the crazy crazy insight that no matter how big the model is, or how many GPUs they have, it burns up all the same.
deleted 4y ago
[deleted]

rjtavares 4y ago

Small players should focus on applications of this tech. We now know that whatever AI Models succeed in the future, they'll be trained by a huge company and finetuned to a specific use case. Small companies should be working on use cases, and then just upgrade to the latest SOTA model.

varispeed 4y ago
> Small players should focus on applications of this tech. That sounds a bit condescending. We are probably at a point from which the government should intervene and help establish level playing field. Otherwise we are going to see a deeper divide between multibillion businesses conquering multiple markets and sort of neofiefdom situation. This is not good.
- rjtavares 4y ago
  I'm not being condescending at all, we've learned that the value in AI is in the applications. If you think government should regulate the field, it should be to make AI Models a commodity, like electricity.
- googlryas 4y ago
  Do you think you'll get a global agreement on this? Or would china just eat America's lunch then?
- tiborsaas 4y ago
  It's not that condescending, that's todays reality. Should I feel entitled for $600k training time that may or may not work? Do you think the government is a good actor to judge if my qualifications are good enough to grant me resources worth a house? It's quite reasonable to make use of models already trained for small players.
  - mschuster91 4y ago
    > Do you think the government is a good actor to judge if my qualifications are good enough to grant me resources worth a house? Governments already routinely do that for pharmaceutical research or for nuclear (fusion) research. In fact, almost all major impact research and development was funded by the government, mostly the military. Lasers, microwaves, silicon, interconnected computers - all funded by the US tax payer, back in the golden times when you'd get laughed out of the room if you dared think about "small government". And the sums involved were ridiculously larger than the worth of a house. We're talking of billions of dollars. Nowadays, R&D funding is way WAY more complex. Some things like AI or mRNA vaccines are mostly funded by private venture capital, some are funded by large philanthropic donors (e.g. Gates Foundation), some by the inconceivably enormous university endowments, a lot by in-house researchers at large corporations, and a select few by government grants. The result of that complexity: - professors have to spend an absurd percentage of their time "chasing grants" (anecdata, up to 40% [1]) instead of actually doing research - because grants are time-restricted, it's rare to have tenure track any more - because of the time restriction and low grant amounts, it's very hard for the support staff as well. In Germany and Austria, for example, extremely low paid "chain contracts" are common - one contract after another, usually for a year, but sometimes as low as half a year. It's virtually impossible to have a social life if you have to up-root it for every contract because you have to take contracts wherever they are, and forget about starting a family because it's just so damn insecure. The only ones that can make it usually come from highly privileged environments: rich parents or, rarely, partners that can support you. Everyone in academia outside of tenured professors struggles with surviving, and the system ruthlessly grinds people to their bones. It's a disgrace. [1] https://www.johndcook.com/blog/2011/04/25/chasing-grants/ https://www.johndcook.com/blog/2011/04/25/chasing-grants/

isthisthingon99 4y ago

How long does it take to train a human? It's useless for two years then maybe it can tell you it needs to poop. The breakthrough will be developing this equivalent in an accessible manner and us taking care to train the thing for a couple of decades but then it becomes our friend.

licebmi__at__ 4y ago
Yes, but to be fair, the system that does the training really sucks and doesn’t scale.
- cactusplant7374 4y ago
  Neither does OpenAI. It costs so much and still delivers so little. A human can generate breakthroughs in science and tech that can be used to reduce carbon emissions. ChatGPT can do no such thing.
  - VBprogrammer 4y ago
    What percentage of humans make meaningful contributions to advancing science or technology? The overwhelming majority of us are just worker bees servicing the needs of the human population.
    - cactusplant7374 4y ago
      I agree with you on this point. It’s also arguable that less people with a better education system could yield the same result with less environmental impact. But my point, poorly explained, is that whatever ChatGPT is, it isn’t original or creative thought as a human would do it. Chomsky’s example (which is based off Turing): Do submarines swim? Yes, they swim — if that’s what you mean by swimming.
      - int_19h 4y ago
        We don't have any clear definitions for "creativity" to begin with. In practice, in these contexts, it seems to be defined as "whatever only humans can do" - that is, the goalposts are automatically moved with every AI advancement.

breck 4y ago

> Are there any possible technologal or scientific leaps on the horizon Yes. From 2017: "Prediction 4: The simplest 2D text encodings for neural networks will be TLs. High level TLs will be found to translate machine written programs into understandable trees." We have something coming out that is an OOM better than anything else out there right now.

spi 4y ago

What do you mean by "small players have no chance"? OpenAI was founded in 2015, it used to be a "small player" which just got things right and grew with it - we're not talking of Google or Facebook investing a chunk of their billions cash. In Germany, AlephAlpha has built their own supercomputer and are training similar sized models. It's expensive for sure, but well in the possibilities of startups. In France researchers trained the similarly sized BLOOM model https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom. They claim it cost between $2 and $4 millions. Sure, a single researcher can't replicate this at their university, but even though OpenAI likes to publish it this way, we're not really talking about research here. Research was inventing the transformer architecture, this is just making it bigger by (very smart) engineering choices. It's something companies should do (and are doing), not researchers.

awestroke 4y ago
Microsoft (using Azure DCs) built a supercomputer with 10,000 V100 GPUs exclusively for OpenAI. [0] It is estimated that it cost around $5M in compute time to train GPT-3. OpenAI has received billions in investment prior to launching GPT-3, including $1B from Microsoft in 2019. [0]: https://blogs.microsoft.com/ai/openai-azure-supercomputer/ https://blogs.microsoft.com/ai/openai-azure-supercomputer/
nileshtrivedi 4y ago
> we're not talking of Google or Facebook investing a chunk of their billions cash OpenAI had raised $1B from Microsoft in 2019 and used it to train a 175B param model. Now, they have raised $10B and are training GPT-4 with 1.5T params. GPUs are capital intensive and as long as there are returns to bigger models, that's exactly where things will go.
- andy_ppp 4y ago
  Will 1.5T parameters be possible to run in the public way GPT-3 is? I can’t wait to see what happens with this much learning!
- awestroke 4y ago

captainmuon 4y ago

I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhythm. Applied to NLP, you could heavily preprocess the input. Tokenize it, annotate parts of speech. Maybe add pronunciation, so the model doesn't have to think about weird english spelling rules, and so you can deal with audio more easily later. So I would build all these little expert-knowledge black boxes and offer them as input to my network. But there is also some inherent resource cost in large language models. If you want to store and process the knowledge of the world, it is going to be expensive no matter what. Maybe we could split the problem into two parts: Understanding language, and world knowledge (with some messy middle ground). I believe you could replace the world knowledge with a huge graph database or triple store. Not just subject-verb-object, but with attribution and certainty numbers for every fact. The idea would be to query the database at inference time. I don't know how to use this in conjunction with a transformer network like GPT-3, so you'd likely need a very different architecture. The big benefit of this would be that it is feasible to train the language part without the world knowledge part with much less resources. But you have other benefits, too. ChatGPT is trained to "win the language game". But as they say, winning the argument does not make you right. If you have a clean fact database, you can have it weigh statements from trustworthy sources higher. You then basically have a nice natural language frontend to a logical reasoning system that can respond with facts (or better: conclusions).

visarga 4y ago
Check out DeepMind RETRO, it's one year old already, but exactly what you say: https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens https://www.deepmind.com/publications/improving-language-mod...
lossolo 4y ago
It's not just about compute; if that were the case, then models like BLOOM and OPT, which also have 175 billion parameters, would have the same performance for real-world use cases as GPT-3, but they don't. Datasets are also very important.

omeysalvi 4y ago

I think AI is going to go the way of the hard sciences where the age of tinkerers making progress by leaps and bounds in their basement is over and incremental progress is going to be the domain of universities or large private companies that can afford to throw money behind it. I would love to be proven wrong and see radical shifts in how people approach these problems. Seems like the cycle started and got to this point way too soon for AI though

swalsh 4y ago
Tinkerers can fine tune a model though. Unfortunately most fine tuning seems to be outmatched at the next iteration of the model.
mittermayr 4y ago
My take on this is that (good) content is one of the bigger problems still, particularly also who exactly the original training data belongs to (or where it comes from). There's a certain risk (we'll see with Github CoPilot soon) it will slow down for a bit until the licensing issues are all sorted out. This can only be solved (for now) by bringing in public funding/data, which universities have always been a very good proxy for. Which also means it (usually) should be open access to the public, to some extent (and useful for the garage folks to catch up a bit). But, once we're past that, it'll be all about that giant body of pre-trained data, securely kept within the next Facebook or Microsoft, amounting to literal data gold (just much higher value at a lot less weight).

belter 4y ago

Model size does not necessarily correlates to quality of results. "Chinchilla (70B) Greatly Outperforms GPT-3 (175B) and Gopher (280B)" - https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b-greatly-outperforms-gpt-3-175b-and-gopher-280b-408b9b4510 https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...

Der_Einzige 4y ago
I highly doubt this in practice on a large scale. Outside of the common phenomena of "most large NNs are under trained" and "less better data is sometimes better than more worse data", there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one. I claim instead that we are still hardly scratching the surface with how we evaluate NLP systems. Also, some fields have straight up trash evaluation schemes. Summarization and ROGUE scores are totally BS and I find the claim that they even correlate with high quality summaries suspect. I say this with publications in the that subfield, so I have personal experience with just how crummy many summarizes are.
- WithinReason 4y ago
  there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one. Overfitting?
  - Der_Einzige 4y ago
    The consensus seems to be that the majority of LMs are undertrained not overfitting though.
espadrine 4y ago
An interesting outcome of the nanoGPT repo is this struggle to exactly match the Chinchilla findings[0], even after discussing it with the authors. A larger discussion is that the scaling laws achieve loss-optimal compute time, but the pre-training loss only improves predictions on the corpus, which contains texts written by people that were wrong or whose prose was lacking. In a real system, what you want to optimize for is accuracy, composability, inventiveness. [0]: https://github.com/karpathy/nanoGPT/blob/master/scaling_laws.ipynb https://github.com/karpathy/nanoGPT/blob/master/scaling_laws...

davidy123 4y ago

Could this be distributed? Put all those mining GPUs to work. A lot of people like participating in public projects like this. I would!

PartiallyTyped 4y ago
In theory, yes. "Hogwild!" is an approach to distributed training, in essence, each worker is given a bunch of data, they compute the gradient and send that to a central authority. The authority accumulates the gradients and periodically pushes new weights. There is also Federated Learning which seemed to start taking off, but then interest rapidly declined.
naraga 4y ago
Exactly. This is inevitable imho. There is no way people will be ok to depend on few wall-gardened models.
dmit 4y ago
>> GPT-3 took 355 years to train > Could this be distributed? Put all those mining GPUs to work. Nope. It's a strictly O(n) process. If it weren't for the foresight of George Patrick Turnbull in 1668, we would not be anywhere close to these amazing results today.
- davidy123 4y ago
  I couldn't find any references to George Patrick Turnbull. If that an ancestor of yours? If so, the comment seems rather subjective.
  - taneq 4y ago
    They're being facetious about the '355 years to train' thing. ;)
    - davidy123 4y ago
      OK haha good one then. Mine was a bit too subtle.

pprotas 4y ago

What does “355 years” mean in this context? I assume it’s not human years

mellosouls 4y ago
Claimed here, so this is presumably the reference (355 GPU Years): https://lambdalabs.com/blog/demystifying-gpt-3 https://lambdalabs.com/blog/demystifying-gpt-3 "We are waiting for OpenAI to reveal more details about the training infrastructure and model implementation. But to put things into perspective, GPT-3 175B model required 3.14E23 FLOPS of computing for training. Even at theoretical 28 TFLOPS for V100 and lowest 3 year reserved cloud pricing we could find, this will take 355 GPU-years and cost $4.6M for a single training run. Similarly, a single RTX 8000, assuming 15 TFLOPS, would take 665 years to run."
- dx034 4y ago
  That's still including margins of cloud vendors. OpenAI had Microsoft providing resources which could do that at much lower cost. It still won't be cheap but you'll be way below $5m if you buy hardware yourself, given that you're able to utilize it long enough. Especially if you set it up in a region with low electricity prices, latency doesn't matter anyway.
Manfred 4y ago
Cumulative hours spent across training hardware.

imtringued 4y ago

As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in optimizing deep learning and that NVIDIA has made it very easy to just throw more hardware at then problem.

visarga 4y ago
> argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that Yes, but you don't know which 0.5% depending on the input text.
londons_explore 4y ago
> very little money in optimizing deep learning Oh - there are a lot of people working on optimizing AI. Amongst hobbyists, academia, and corporations alike. The thing is, if you come up with a neat optimization that saves 30% of compute for the same results, typically instead of reducing your compute budget 30%, you instead increase your model/data size 30% and get better results.
- narrator 4y ago
  Jevon's paradox of data and AI. The more efficiently data is used, the more demand their is for data.
  - antognini 4y ago
    Any state of the art model takes about three weeks to train.
    - visarga 4y ago
      More an indication of human patience than task difficulty.

naraga 4y ago

I think something of Seti@Home kind will come.

seydor 4y ago

It should be no issue if it became massively parralelized a-la SETI. I wonder when Wikimedia or Apache foundation will jump into AI

yreg 4y ago
Wikimedia and other organizations that deal with moderation might want to keep this technology out of the hands of the general public for as long as possible.

xur17 4y ago

Alternatively, are there ways to train on consumer graphics cards, similar to SETI@Home or Folding@Home? I would personally be happy to donate gpu time, as I imagine many others would as well.

mryab 4y ago
There absolutely are! Check out hivemind (https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind), a general library for deep learning over the Internet, or Petals (https://petals.ml/ https://petals.ml/), a system that leverages Hivemind and allows you to run BLOOM-176B (or other large language models) that is distributed over many volunteer PCs. You can join it and host some layers of the model by running literally one command on a Linux machine with Docker and a recent enough GPU. Disclaimer: I work on these projects, both are based on our research over the past three years
alfor 4y ago
The cost of moving data from one gpu to the next will destroy performance. The system are moving in the opposite direction (look at Dojo architecture or TensTorrent) The silver lining is that the cost of training will fall substantially with those architecture that are not based in reusing gpu.

ignoramous 4y ago

Yes, see DeepMind RETRO: > In our experiments on the Pile, a standard language modeling benchmark, a 7.5 billion parameter RETRO model outperforms the 175 billion parameter Jurassic-1 on 10 out of 16 datasets and outperforms the 280B Gopher on 9 out of 16 datasets. https://www.deepmind.com/blog/improving-language-models-by-retrieving-from-trillions-of-tokens https://www.deepmind.com/blog/improving-language-models-by-r... Though, there hasn't been much follow-up research on it (or DeepMind is not publishing it). Annotated paper: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/papers/2112.04426.pdf https://github.com/labmlai/annotated_deep_learning_paper_imp...

espadrine 4y ago
The research is still ongoing, although perhaps lower-profile than what appears in the press. RETRO did get press, but it was not the first retrieval model, and in fact was not SOTA when it got published; FiD was, which later evolved into Atlas[0], published a few months ago. [0]: https://github.com/facebookresearch/atlas https://github.com/facebookresearch/atlas

GistNoesis 4y ago

Yes. There are plenty forward leaps, most of them are not new and are just waiting to be integrated or released : Let's pave the road for SkyNet hard lift-off : -The first obvious one is use of external knowledge store, aka instead of having to store facts in the neural weights where they struggle, just store them in a database and teach your neural network to use it. (This is also similar to something like webgpt where you allow your network to search the web). This will allow you to have a network of 1G parameters (and external indexes of a few TB) that is as performant as a network of 100G parameters, and with better scaling property too. You can probably gain at least 2 orders of magnitude there. -The second leap is better architecture of your neural networks, approximating transformer that are quadratic compute by something that is linear compute (linformer) or n log n compute (reformer) can get you an order of magnitude faster by simply reducing your iteration time. Similarly using some architectures based on sparsity can give you faster computation (although some of the gains are reduced by lesser efficiency of sparse memory access pattern). Using (analog bits) Diffusion to Generatively PreTrain sentences at a time instead of token by token. You can probably gain between 1 and 3 order of magnitude here if you write and optimize everything manually (or have your advanced network/compiler optimize your code for you) -The third leap is reduced domain : You don't have a single network that you train on everything. Training one network by domain allows you to have a smaller network that compute faster. But also it allows you to focus your training on what matters to you : for example if you want to have a mathematics network, its parameters are not influenced a lot by showing it football pictures. There is at least 2 orders of magnitude there. -The fourth one is external tool usage. It's kind of related to the first one but whereas in the first one is readily differentiable, this one necessitate some Reinforcement Learning (that's what decision transformer are used for). -Compression : compress everywhere. The bottlenecks are memory bandwidth related. Work in compressed form when relevant. One order of magnitude -Distributed training : Because the memory bandwidth of inside a GPU is in the order of TB/s where as the transfer to the GPU is in the order of 10GB/s. There is an advantage to have the parameters reside on the GPU but there is a limited quantity of memory in the GPU, so distributed training (something like petals.ml) allows you to increase your memory bandwidth by collaborating. So each actor can probably gain an order of magnitude. Provided that they can keep bad actors away. -Use free resources : The other day Steam had 10M users with GPU waiting around doing nothing, just release a dwarf fortress mod with prettier pictures and use the compute for more important tasks. -Remove any humans in the loop : it's faster to iterate when you don't have to rely any human, either for dataset construction or model building :)

QuesnayJr 4y ago

There are a couple of cases where small changes in the model make training much quicker. For example, the currently leading Go AI, KataGo, requires much less time to train than AlphaGo did.