5 ms·

I would care more about LLaMA architecture when I can get hands on, honestly this project is more interesting and lighting fast on even a 2060 laptop https://gi

by amrb 4y ago

I would care more about LLaMA architecture when I can get hands on, honestly this project is more interesting and lighting fast on even a 2060 laptop https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM

iForgotMyPW 4y ago
[flagged]
lachlan_gray 4y ago
Has anyone tried this on an m1 machine?
- tmptmptmp1 4y ago
  Do you have access to the weights? If so you probably have better ML hardware. Wish this model was actually "open". The perf of a model of that size on the M1 will not be good. That is big enough it won't quite fit on a 3090 (24GB) without quantization.
- pumanoir 4y ago
  I think is feasible. The description even says is designed to save on vram[1]. I don't get the other comments about needing more vram than a 3090. Also, Neuralmagic may run their sparsification on ARM cpu's in the future, so keep an eye. 1. ChatRWKV v2: with "stream" and "split" strategies. 3G VRAM is enough to run RWKV 14B :)
  - IanCal 4y ago
    You have to split it up which slows it down a lot. The 14B model doesn't fit fully on a 3090, though the 7B fits easily and is very fast. Other replies either may have meant this or thought the original comment was about llama.
- fswd 4y ago
  I've ran it on a AMD 3950 which I think is half the speed of a M1, and it's plenty fast. Note I am specifically talked about RWKV
  - nl 4y ago
    It would be interesting to see a version of RWKV[1] that takes some of the improvements in LLaMA (eg the SwiGLU activation function and the Rotary Embeddings - although I think they have tried rotary embeddings in some versions of RWKV) as well as the same dataset and see how it does. The dataset is interesting. It's not dissimllar to The Pile, which RWKV is already trained on, but does seem to have quite a lot more preprocessing to increase the dataset quality. [1] https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM
pmarreck 4y ago
very interesting. Do you know of anything that would take advantage of the 128 cores of my Ryzen Threadripper even though I have a 2080 and a 3080 as well? (Or all three... lol)
- vorticalbox 4y ago
  GPU is massively faster than CPUs for this sort of work load. Even if you could use all 128 cores I would still get the 2080 would be faster
  - pmarreck 4y ago
    ah ok, that's fair.
  - zackmorris 4y ago
    Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot: https://www.pcworld.com/article/402592/nvidia-turing-gpu-geforce-rtx-2080-ti.html https://www.pcworld.com/article/402592/nvidia-turing-gpu-gef... IMHO these chips designed for specialized workloads are looking more and more ridiculous. I expect GPU manufacturers to keep dragging them out for another decade or so, as well as Apple as it explores offshoots of M1. It all makes me feel very tired (the ultimate code smell). A better design would be something like a 256+ core RISC-V with local memories in/by each core for data-locality and a content-addressable caching scheme for deduplication. Copy-on-write languages like Clojure and orchestrating processes under something like Docker would make it a breeze to program, although it would still support manually managed mutability like with Rust for innermost loops in games or whatever. It's fairly obvious how it would all work, but IP law and gatekeeping ensure that it will not happen anytime soon. Then stuff like CUDA would be just another framework run on a symmetric multiprocessor and we could get back to exploring alternatives like genetic algorithms like we did in the 1990s. Thankfully nobody cares what I think, heck even I'm sick of reading my own complaints, so it's easy enough to just unsee this.
    - dragonwriter 4y ago
la64710 4y ago
News flash : filling a google form to get access to the model based on Meta’s discretion cannot be called open source .
- MH15 4y ago
  And yet they're trying real hard to.
  - holtkam2 4y ago
    Anyone know of the best way to get approved? I’m not a academic researcher but I really want to start experimenting…
    - flangola7 4y ago
      If you can tell them which university you're with and the formal name of your research project that would be a start. They'll reach out to the admin to confirm.
UncleOxidant 4y ago
The README there doesn't seem too coherent... how does one actually get that running? The Quick Start section isn't very helpful.
dwheeler 4y ago
The nebullvm projrct and specifically chatllama, by themselves, are clearly OSS. They're licensed under Apache-2.0: https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler... However, many models including LLaMA are not OSS (they have terms such as "non-commercial use only"). And you need models. It can be confusing, it's important to try to make that clear.
nl 4y ago
> I would care more about LLaMA architecture when I can get hands on The architecture is available just fine, both in the paper[1] and in code: https://github.com/facebookresearch/llama/blob/main/llama/model.py https://github.com/facebookresearch/llama/blob/main/llama/mo... It's the model weights that you can't get as easily. [1] https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/ https://research.facebook.com/publications/llama-open-and-ef...

by amrb 4y ago

iForgotMyPW 4y ago

[flagged]

lachlan_gray 4y ago

Has anyone tried this on an m1 machine?

tmptmptmp1 4y ago
Do you have access to the weights? If so you probably have better ML hardware. Wish this model was actually "open". The perf of a model of that size on the M1 will not be good. That is big enough it won't quite fit on a 3090 (24GB) without quantization.
pumanoir 4y ago
I think is feasible. The description even says is designed to save on vram[1]. I don't get the other comments about needing more vram than a 3090. Also, Neuralmagic may run their sparsification on ARM cpu's in the future, so keep an eye. 1. ChatRWKV v2: with "stream" and "split" strategies. 3G VRAM is enough to run RWKV 14B :)
- IanCal 4y ago
  You have to split it up which slows it down a lot. The 14B model doesn't fit fully on a 3090, though the 7B fits easily and is very fast. Other replies either may have meant this or thought the original comment was about llama.
fswd 4y ago
I've ran it on a AMD 3950 which I think is half the speed of a M1, and it's plenty fast. Note I am specifically talked about RWKV
- nl 4y ago
  It would be interesting to see a version of RWKV[1] that takes some of the improvements in LLaMA (eg the SwiGLU activation function and the Rotary Embeddings - although I think they have tried rotary embeddings in some versions of RWKV) as well as the same dataset and see how it does. The dataset is interesting. It's not dissimllar to The Pile, which RWKV is already trained on, but does seem to have quite a lot more preprocessing to increase the dataset quality. [1] https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM

pmarreck 4y ago

very interesting. Do you know of anything that would take advantage of the 128 cores of my Ryzen Threadripper even though I have a 2080 and a 3080 as well? (Or all three... lol)

vorticalbox 4y ago
GPU is massively faster than CPUs for this sort of work load. Even if you could use all 128 cores I would still get the 2080 would be faster
- pmarreck 4y ago
  ah ok, that's fair.
- zackmorris 4y ago
  Keep in mind that many of the GPU cores go unused, since they are dedicated to geometry or ray tracing or whatnot: https://www.pcworld.com/article/402592/nvidia-turing-gpu-geforce-rtx-2080-ti.html https://www.pcworld.com/article/402592/nvidia-turing-gpu-gef... IMHO these chips designed for specialized workloads are looking more and more ridiculous. I expect GPU manufacturers to keep dragging them out for another decade or so, as well as Apple as it explores offshoots of M1. It all makes me feel very tired (the ultimate code smell). A better design would be something like a 256+ core RISC-V with local memories in/by each core for data-locality and a content-addressable caching scheme for deduplication. Copy-on-write languages like Clojure and orchestrating processes under something like Docker would make it a breeze to program, although it would still support manually managed mutability like with Rust for innermost loops in games or whatever. It's fairly obvious how it would all work, but IP law and gatekeeping ensure that it will not happen anytime soon. Then stuff like CUDA would be just another framework run on a symmetric multiprocessor and we could get back to exploring alternatives like genetic algorithms like we did in the 1990s. Thankfully nobody cares what I think, heck even I'm sick of reading my own complaints, so it's easy enough to just unsee this.
  - dragonwriter 4y ago

la64710 4y ago

News flash : filling a google form to get access to the model based on Meta’s discretion cannot be called open source .

MH15 4y ago
And yet they're trying real hard to.
- holtkam2 4y ago
  Anyone know of the best way to get approved? I’m not a academic researcher but I really want to start experimenting…
  - flangola7 4y ago
    If you can tell them which university you're with and the formal name of your research project that would be a start. They'll reach out to the admin to confirm.

UncleOxidant 4y ago

The README there doesn't seem too coherent... how does one actually get that running? The Quick Start section isn't very helpful.

dwheeler 4y ago

The nebullvm projrct and specifically chatllama, by themselves, are clearly OSS. They're licensed under Apache-2.0: https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler... However, many models including LLaMA are not OSS (they have terms such as "non-commercial use only"). And you need models. It can be confusing, it's important to try to make that clear.

nl 4y ago

> I would care more about LLaMA architecture when I can get hands on The architecture is available just fine, both in the paper[1] and in code: https://github.com/facebookresearch/llama/blob/main/llama/model.py https://github.com/facebookresearch/llama/blob/main/llama/mo... It's the model weights that you can't get as easily. [1] https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/ https://research.facebook.com/publications/llama-open-and-ef...