3 ms·

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llam

by gman_ 3y ago

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f315200d6ccffe92b2fd00e2575aa/src/main.rs https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...

wtarreau 3y ago
As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.
- voz_ 3y ago
  Can you chill? Stuff like this is super useful. The original c file is educational, so is this. And now by having it two ways, we have a tiny little Rosetta Stone for folks that wanna learn.
  - jheuel 3y ago
    Might also help as input to the next iteration of LLMs
- gman_ 3y ago
  Looks like you spoke too soon, I'm clocking 340+ tokens per second with my improved Rust implementation, compared to 106 with the original C. That being said, I didn't share this for any reason other than to share ideas and promote learning. Cheers
  - steeve 3y ago
    540 tok/s on the C version using -ffast-math and -Ofast https://twitter.com/karpathy/status/1683301419716313089?s=20 https://twitter.com/karpathy/status/1683301419716313089?s=20
    - gman_ 3y ago
      Wow! This is the most fun I've had programming in a while. Thanks for sharing
- elteto 3y ago
  You should list your email in your HN profile. That way the Internet could check with you to see if you approve whenever someone starts a new personal project.
l-m-z 3y ago
Another random (self) plug for a rust version, this uses the candle ML library we've been working on for the last month and can be run in the browser. https://laurentmazare.github.io/candle-llama2/index.html https://laurentmazare.github.io/candle-llama2/index.html The non-web version has full GPU support but is not at all minimalist :)
- ianpurton 3y ago
  Really nice.

by gman_ 3y ago

wtarreau 3y ago

As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.

voz_ 3y ago
Can you chill? Stuff like this is super useful. The original c file is educational, so is this. And now by having it two ways, we have a tiny little Rosetta Stone for folks that wanna learn.
- jheuel 3y ago
  Might also help as input to the next iteration of LLMs
gman_ 3y ago
Looks like you spoke too soon, I'm clocking 340+ tokens per second with my improved Rust implementation, compared to 106 with the original C. That being said, I didn't share this for any reason other than to share ideas and promote learning. Cheers
- steeve 3y ago
  540 tok/s on the C version using -ffast-math and -Ofast https://twitter.com/karpathy/status/1683301419716313089?s=20 https://twitter.com/karpathy/status/1683301419716313089?s=20
  - gman_ 3y ago
    Wow! This is the most fun I've had programming in a while. Thanks for sharing
elteto 3y ago
You should list your email in your HN profile. That way the Internet could check with you to see if you approve whenever someone starts a new personal project.

l-m-z 3y ago

Another random (self) plug for a rust version, this uses the candle ML library we've been working on for the last month and can be run in the browser. https://laurentmazare.github.io/candle-llama2/index.html https://laurentmazare.github.io/candle-llama2/index.html The non-web version has full GPU support but is not at all minimalist :)

ianpurton 3y ago
Really nice.