3 ms·
Anecdote: I can tell you that Vicuna-13B, which is a pretty decent model, runs about 4 to 5 tokens/second on an Apple M1 with 16GB. Takes about 10GB of memory
by rootusrootus 3y ago
Anecdote: I can tell you that Vicuna-13B, which is a pretty decent model, runs about 4 to 5 tokens/second on an Apple M1 with 16GB. Takes about 10GB of memory when loaded. Friend of mine with a RTX 2070 Super gets comparable results.
I'm upgrading my M1 laptop on Thursday to a newer model, M2 MAX with 96GB of memory. I'm totally going to try Falcon-40B on it, though I do not expect it to run that well. But I do expect it (the M2, I mean) will be snappier on the smaller models than my original M1 is.