4 ms·

an interesting thread with speculation about how to eventually do this on local TPUs with llama.cpp and GGUF infrastructure: https://www.reddit.com/r/LocalLLaMA

• felarof 2y ago
  Ahh, the reddit thread is referring to edge TPU devices, will check it out. Google also has Cloud TPUs, which are their server-side accelerators, and this is what we are initially trying to build for!
• Havoc 2y ago
  That’s not happening. The coral edge tpus are ancient, slow and don’t have enough mem to be meaningful and somehow still manage to be relatively expensive even 2nd hand. They have some good uses but LLMs aint it
  • pogue 2y ago
    Are those the TPUs Google sells to consumers? I've been thinking of buying one & hooking it up to a Pi just to play around with LLMs or Stable Diffusion. But I didn't realize they were slower/worse than other options.
    • kccqzy 2y ago
      The Coral TPUs have not been updated for several years. They were last updated long before the current LLM craze. They are good for simple things like object detection in photos. They have almost nothing in common with Cloud TPUs.
      • deleted 2y ago
        [deleted]