3 ms·

I tried Claude. If hardware continues it's evolution of speed in the next 10 years I can have Claude but local + running constantly and yeah that would change

• Scene_Cast2 2y ago
  When ChatGPT was down a few days back, I locally booted up Codestral. It was decent and usable.
• jasonjmcghee 2y ago
  Try llama 3.3 70B. On groq or something. Runs on a 64GB macbook (4bit quantized, which seems to not impact quality much). Things have come a long way. Compare to llama 2 70b. It's wild
  • Terretta 2y ago
    Llama 3.3 70B 8-bit MLX runs on Macbook 128GB at 7+ tokens per second while running a full suite of other tools, even at the 130k tokens size, and behaves with surprising coherence. Reminded me of this time last year, first trying Mixtral 8x22 — which still offers a distinctive je ne sais quoi!
• nmfisher 2y ago
  Qwen 2.5 32B Coder is actually a viable locally-hosted alternative to Claude 3.5 Sonnet. It's not better, but if I couldn't access Claude for some reason, I would definitely use it.