3 ms·
7B & 13B can be ran on a M1 Air with 16G memory: 7B uses about 4.5G max & runs at 203.38 ms per token, 13B about 8G and does 396.58 ms per token. 30B needs ab
by boesboes 4y ago
7B & 13B can be ran on a M1 Air with 16G memory:
7B uses about 4.5G max & runs at 203.38 ms per token, 13B about 8G and does 396.58 ms per token.
30B needs about 20G and basically hangs due to swapping i guess with 16G.
- arvinsim 4y agoJust the comment that I was looking for. Guess I can run it on my M1 Pro 32GB model