3 ms·

Is this similar to llama.cpp? I'm not very versed in this area.

by politelemon 3y ago

regularfry 3y ago
Inspired by, but no shared code as far as I can see on a brief scan.
- AMICABoard 3y ago
  It is not inspired by, it's friendly fork, so its more than inspiration. Code has diverged a bit, yes I do try to share. Karpathy's llama2.c upstream project has clearly stated goals of elegance and simplicity, so code that has lot of pre-processor directives like ours won't help upstream. Apart from this we are very thankful to jart & contributors from the cosmopolitan project and also the unikraft folks without which binary portability or unikernels wouldn't have been possible. TL;DR Simple elegant stuff that we added will be shared to upstream. Complex non elegant code won't be shared as it won't be accepted upstream.
  - regularfry 3y ago
    That's your relationship to upstream llama2.c, which is fairly clear. politelemon was asking about the relationship to ggerganov's llama.cpp, which seems to have inspired the upstream llama2.c.
AMICABoard 3y ago
This is a fork of https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c karpathy's llama2.c is like llama.cpp but it is written in C and the python training code is available in that same repo. llama2.c's goal is to be a elegant single file C implementation of the inference and an elegant python implementation for training. His goal is for people to understand how llama 2 and LLM's work, so he keeps it simple and sweet. As the project progresses, so will features and performance improvements added. Currently it can infer baby (small) Story models trained by Karpathy at a fast pace. It can also infer Meta LLAMA 2 7b models, but at a very slow rate such as 1 token per second. So currently this can be used for learning or as a tech preview. Our friendly fork tries to make it portable, performant and more usable (bells and whistles) over time. Since we mirror upstream closely, the inference capabilities of our fork is similar but slightly faster if compiled with acceleration. What we try to do different is that we try to make this bootable (not there yet) and portable. Right now you can get binary portablity - use the same run.com on any x86_64 machine running on any OS, it will work (possible due to cosmopolitan toolchain). The other part that works is unikernels - boot this as unikernel in VM's (possible due unikraft unikernel & toolchain). See our fork currently as a release early and release often toy tech demo. We plan to build it out into a useful product.

Is this similar to llama.cpp? I'm not very versed in this area.

regularfry 3y ago

Inspired by, but no shared code as far as I can see on a brief scan.

AMICABoard 3y ago
It is not inspired by, it's friendly fork, so its more than inspiration. Code has diverged a bit, yes I do try to share. Karpathy's llama2.c upstream project has clearly stated goals of elegance and simplicity, so code that has lot of pre-processor directives like ours won't help upstream. Apart from this we are very thankful to jart & contributors from the cosmopolitan project and also the unikraft folks without which binary portability or unikernels wouldn't have been possible. TL;DR Simple elegant stuff that we added will be shared to upstream. Complex non elegant code won't be shared as it won't be accepted upstream.
- regularfry 3y ago
  That's your relationship to upstream llama2.c, which is fairly clear. politelemon was asking about the relationship to ggerganov's llama.cpp, which seems to have inspired the upstream llama2.c.

AMICABoard 3y ago

This is a fork of https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c karpathy's llama2.c is like llama.cpp but it is written in C and the python training code is available in that same repo. llama2.c's goal is to be a elegant single file C implementation of the inference and an elegant python implementation for training. His goal is for people to understand how llama 2 and LLM's work, so he keeps it simple and sweet. As the project progresses, so will features and performance improvements added. Currently it can infer baby (small) Story models trained by Karpathy at a fast pace. It can also infer Meta LLAMA 2 7b models, but at a very slow rate such as 1 token per second. So currently this can be used for learning or as a tech preview. Our friendly fork tries to make it portable, performant and more usable (bells and whistles) over time. Since we mirror upstream closely, the inference capabilities of our fork is similar but slightly faster if compiled with acceleration. What we try to do different is that we try to make this bootable (not there yet) and portable. Right now you can get binary portablity - use the same run.com on any x86_64 machine running on any OS, it will work (possible due to cosmopolitan toolchain). The other part that works is unikernels - boot this as unikernel in VM's (possible due unikraft unikernel & toolchain). See our fork currently as a release early and release often toy tech demo. We plan to build it out into a useful product.