5 ms·

Now that it runs on our laptops a lot of the initial shock, mystique, and novelty has already worn off, just like with DALL-E before it. What the leaks show is

Now that it runs on our laptops a lot of the initial shock, mystique, and novelty has already worn off, just like with DALL-E before it. What the leaks show is that the community can run circles around OpenAI and the others when it comes to optimizing these things and taking them in interesting directions on a shoe string budget. For example of what one sesh of tinkering can produce: https://github.com/geohot/tinygrad/blob/master/examples/llama.py#L264-L347 https://github.com/geohot/tinygrad/blob/master/examples/llam... https://www.youtube.com/watch?v=nctqc8FBJ2U&t=4954s https://www.youtube.com/watch?v=nctqc8FBJ2U&t=4954s The cat is out of the bag, you don't need a beowulf cluster of A100's and unlimited Azure credits. You do need a budget to train that is out of reach for hobbyists but the moat is not insurmountable and I don't think VC's will be scared off anymore as a lot of startup ideas are viable and a lot of outsiders can demonstrably make a real go of it. Time for BasedAI. Game on.

can_center_divs 4y ago
Yeah, the community always seems to figure out how to do things more effective. My girlfriend asked me if I could transcribe some audio files for her with my "programming stuff". I immediately thought of Whisper from OpenAI. I first used the official CLI tool. With the largest model it took long 8 hours to transcribe a 30 min long file. I noticed it was running on the CPU - tried switching it to use the GPU instead with no luck. Running it on WSL was probably not helping. Then I found this gem: https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper A C++ Windows implementation of Whisper. I opened the program, fed it with the largest model and the file. The transcript was done in 4 minutes, instead of 8 hours... Downside? The program has a GUI, lol. Of course, I could probably get the CLI tool to run on the GPU with some tinkering and installing some Nvidia packages for Whisper to use. But frankly, I have so little experience with that kind of stuff, that installing the Windows implementation was a much easier choice.
- deleted 4y ago
  [deleted]
FooBarWidget 4y ago
ChatGPT can run on our laptops? How? The only thing I could find was that it can't because it requires 128 GB VRAM.
- recuter 4y ago
  https://github.com/ggerganov/llama.cpp/issues/39 https://github.com/ggerganov/llama.cpp/issues/39 40GB for the big llama.
  - FooBarWidget 4y ago
    But that's not ChatGPT?
    - ShamelessC 4y ago
      No one said ChatGPT. It was a vague reference to the recent LLaMA developments from the community.
    - arlcode 4y ago
      Not it isn't but what the OP is saying that with the llama models (and others) being runable on consumer hardware it's only a matter of time until we get something that's "good enough" for many use cases. Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.
    - deleted 4y ago
      [deleted]
  - karmasimida 4y ago
    Even biggest Llama is like 1/3 size of ChatGPT
    - recuter 4y ago
      Might as well compare it to AlpacaGPT, they are trained different. Carefully prompted the output shows flashes of excellence and really whips ass and it is still early days. We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.
ivalm 4y ago
People have decent oss solutions for running inference. There are currently no high quality oss code for training and the best checkpoints are still unavailable (gpt3.5+4/palm/claude)… There are only a few hundred folks who have actual experience training at this scale. llama is both not oss and not as good as da-vinci-003/etc. Prob best actual oss model is like flan-t5-xxl.
- recuter 4y ago
  So we can't lay eggs yet, at least we got one. Who knows what might hatch.
  - ivalm 4y ago
    I guess I just felt an objection about “[oss] running circles around OpenAI.” The reality is that the core technology is still very unapproachable (making these models). OSS solved like 1% of the total problem/difficulty. My biggest hope is that meta/google/anthropic actually decide to release training code + data + checkpoint. Meta is most likely to do it but still unlikely Without that it would take eutherai/carperai to really exceed expectations for OSS to be competitive.

can_center_divs 4y ago

Yeah, the community always seems to figure out how to do things more effective. My girlfriend asked me if I could transcribe some audio files for her with my "programming stuff". I immediately thought of Whisper from OpenAI. I first used the official CLI tool. With the largest model it took long 8 hours to transcribe a 30 min long file. I noticed it was running on the CPU - tried switching it to use the GPU instead with no luck. Running it on WSL was probably not helping. Then I found this gem: https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper A C++ Windows implementation of Whisper. I opened the program, fed it with the largest model and the file. The transcript was done in 4 minutes, instead of 8 hours... Downside? The program has a GUI, lol. Of course, I could probably get the CLI tool to run on the GPU with some tinkering and installing some Nvidia packages for Whisper to use. But frankly, I have so little experience with that kind of stuff, that installing the Windows implementation was a much easier choice.

deleted 4y ago
[deleted]

FooBarWidget 4y ago

ChatGPT can run on our laptops? How? The only thing I could find was that it can't because it requires 128 GB VRAM.

recuter 4y ago
https://github.com/ggerganov/llama.cpp/issues/39 https://github.com/ggerganov/llama.cpp/issues/39 40GB for the big llama.
- FooBarWidget 4y ago
  But that's not ChatGPT?
  - ShamelessC 4y ago
    No one said ChatGPT. It was a vague reference to the recent LLaMA developments from the community.
  - arlcode 4y ago
    Not it isn't but what the OP is saying that with the llama models (and others) being runable on consumer hardware it's only a matter of time until we get something that's "good enough" for many use cases. Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.
  - deleted 4y ago
    [deleted]
- karmasimida 4y ago
  Even biggest Llama is like 1/3 size of ChatGPT
  - recuter 4y ago
    Might as well compare it to AlpacaGPT, they are trained different. Carefully prompted the output shows flashes of excellence and really whips ass and it is still early days. We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.

ivalm 4y ago

People have decent oss solutions for running inference. There are currently no high quality oss code for training and the best checkpoints are still unavailable (gpt3.5+4/palm/claude)… There are only a few hundred folks who have actual experience training at this scale. llama is both not oss and not as good as da-vinci-003/etc. Prob best actual oss model is like flan-t5-xxl.

recuter 4y ago
So we can't lay eggs yet, at least we got one. Who knows what might hatch.
- ivalm 4y ago
  I guess I just felt an objection about “[oss] running circles around OpenAI.” The reality is that the core technology is still very unapproachable (making these models). OSS solved like 1% of the total problem/difficulty. My biggest hope is that meta/google/anthropic actually decide to release training code + data + checkpoint. Meta is most likely to do it but still unlikely Without that it would take eutherai/carperai to really exceed expectations for OSS to be competitive.