3 ms·

Question: When will laptops be powerful to run a ChatGPT or GPT-3.5 like model, locally – can the M2 Max already do this? The latency of ChatGPT is a huge downs

• smoldesu 4y ago
  Probably never (doubly so if your laptop doesn't have Nvidia hardware acceleration). ChatGPT uses enormous 300+gb models that fit into VRAM partitioned across dozens of accelerator boards. Running the same program locally is not possible. There are generative models you can run locally (gpt neo 2.7b), but it's a much smaller model that's much less refined. Even still, inferencing it locally would probably be slower than ChatGPT running in the cloud. There may come a day when someone has a heavily-pruned knowledge model available for local use, but I wouldn't count on it. We chose a future where everything runs in the cloud, and our "ownership" of technology boils down to a payment voucher saying we're entitled to it. OpenAI will never let you run it locally because then what would their value be? Also, as a side note: I'm not sure these AI speculation threads are a good fit for HN. There have been so many of them in the past few months, it's been overwhelming and a little annoying. Nothing against you for posting it, but a lot of these threads have already been made in the past.