4 ms·
Quick steps I did to get it working 1. Download the models from here https://github.com/shawwn/llama-dl/blob/main/llama.sh https://github.com/shawwn/llama-dl/b
by renewiltord 4y ago
Quick steps I did to get it working
1. Download the models from here https://github.com/shawwn/llama-dl/blob/main/llama.sh https://github.com/shawwn/llama-dl/blob/main/llama.sh (you'll need to edit the script to make it work in bash)
2. Follow the instructions here.
3. Create an interesting prompt for the AI and put your question at the end.
It works tremendously fast on an Apple Macbook Pro with the M1 Max and 64 GiB of RAM. I only downloaded the 7B model because I wanted a quick run through. I'll try the rest later.
Here's some good fun:
Prompt: Example HN Post: Gojq: Pure Go Implementation of Jq. Example HN top comment: "gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when Im designing JSON APIs I like to put things like the "id" key first in an object layout, and when Im manipulating JSON using jq or similar I like to maintain those aesthetic choices. HN Post:"Llama.cpp - C/C++ implementation of Facebook LLama model". HN top comment:
Completion: "This is more of an example of C++s power than a breakthrough in computer science." Example HN Post: "The Moral Case for Software Patents". HN top comment: "The patent system is a vestigial technology, and the whole point of a vestigial technology is to make something go away. Its like a dinosaur that no one is using anymore. Its served its purpose." Example HN Post: "Benchmark: The Mystery of Gauss Summation". HN top comment: "But it is interesting to see how Benchmark manages to cram
main: mem per token = 14368644 bytes
main: load time = 1301.14 ms
main: sample time = 273.73 ms
main: predict time = 20970.29 ms / 82.89 ms per token
main: total time = 22939.82 ms
- crimjow 4y agoIt even maintained your spelling errors.