6 ms·

Nobody stopping anything ever. I predict we will have distributed training systems soon on peer to peer like network for collaboratively building free models t

Nobody stopping anything ever. I predict we will have distributed training systems soon on peer to peer like network for collaboratively building free models to use, if we can all pitch in some GPUs we can get 100s of thousand of individuals working on training giant LLM models together. It’s an unsolved problem currently but should have a solution soon. Additionally today we can build sub-$10K systems to do self hosted inference on 65B weights models. I’m seriously confused why anybody thinks anyone will even remotely care about any kind of pause is asked for. For sure everyone with computers and a reasonable level of skills are now looking at this field. The progress will be exponential and uncontrollable in my opinion. I’m fine with that and I also think there is absolutely nothing that can stop it anyway.

mlinsey 4y ago
> I’m seriously confused why anybody thinks anyone will even remotely care about any kind of pause is asked for. I can say that Google and Microsoft would definitely care, just as two specific instances of "anybody" that I have worked for and seen what their legal review process gating new product launches is like. You seem much more optimistic that the community efforts are <6 months behind those companies than I am; I would be happy to be wrong here and see an open-source self-hosted LLM with better than GPT4 performance by September.
- alchemist1e9 4y ago
  I’ve read OpenAI only used around 4000 high end GPUs ($30K each) for around 4 months to train GPT4. Yes there are some reinforcement tricks that are still unclear but input data is basically known. I think people underestimate how many entities have the ability to easily acquire that amount of resources. It’s less than $200M to purchase that amount of hardware. Militaries spend $2-4M per missile. This is nothing for a lot of organizations. We already have self hosted LLM inference that reaches gpt 3.5 turbo level. Google and Microsoft are notoriously slow moving and probably less relevant than smaller faster operations like OpenAI.
allturtles 4y ago
The fabs for gpu production are easily controllable. So are the data centers where many of them are currently concentrated.
- alchemist1e9 4y ago
  There you go. I was waiting for this comment. Yep so global totalitarianism is the clear answer. Makes sense. I’m sure nobody will resist.

mlinsey 4y ago

> I’m seriously confused why anybody thinks anyone will even remotely care about any kind of pause is asked for. I can say that Google and Microsoft would definitely care, just as two specific instances of "anybody" that I have worked for and seen what their legal review process gating new product launches is like. You seem much more optimistic that the community efforts are <6 months behind those companies than I am; I would be happy to be wrong here and see an open-source self-hosted LLM with better than GPT4 performance by September.

alchemist1e9 4y ago
I’ve read OpenAI only used around 4000 high end GPUs ($30K each) for around 4 months to train GPT4. Yes there are some reinforcement tricks that are still unclear but input data is basically known. I think people underestimate how many entities have the ability to easily acquire that amount of resources. It’s less than $200M to purchase that amount of hardware. Militaries spend $2-4M per missile. This is nothing for a lot of organizations. We already have self hosted LLM inference that reaches gpt 3.5 turbo level. Google and Microsoft are notoriously slow moving and probably less relevant than smaller faster operations like OpenAI.

allturtles 4y ago

The fabs for gpu production are easily controllable. So are the data centers where many of them are currently concentrated.

alchemist1e9 4y ago
There you go. I was waiting for this comment. Yep so global totalitarianism is the clear answer. Makes sense. I’m sure nobody will resist.