4 ms·
What would you be able to build with this? Is it possible to inference LLMs on these devices?
by sergiomattei 3y ago
What would you be able to build with this? Is it possible to inference LLMs on these devices?
- krasin 3y agoHardly. 1) Coral Edge TPU chip has 8 MB SRAM; for comparison, Whisper Tiny (https://huggingface.co/openai/whisper-tiny https://huggingface.co/openai/whisper-tiny) is 40M parameters, and most of LLMs have billions of parameters. 2) I would be surprised if Coral Edge TPU supports all TensorFlow ops required to run Transformers. The chip was designed with convolutional networks in mind.
- refulgentis 3y agoGoogle abandonware, with the fact its Google very hidden. TensorFlow lost, or never really got started tbh, for local inference of the stuff everyones built the last few years, including LLMs. My understanding is if you wanted to build "is it a hot dog" from scratch and deploy it locally its great
- selectodude 3y agoFrigate uses Google Coral for AI-based analysis of NVR video using cheap cameras. Works really well. https://docs.frigate.video https://docs.frigate.video
- ramzyo 3y agoFrigate uses the TPU with an application processor, not this MCU + TPU version, right?
- AnarchismIsCool 3y agoYeah but their point is still relevant given the direction the thread has gone