4 ms·
There was a startup posted here which built custom hardware that let the AI respond instantly. Thousands of tokens per second.
by MagicMoonlight 5mo ago
There was a startup posted here which built custom hardware that let the AI respond instantly. Thousands of tokens per second.
- zargon 5mo agoGroq.
- beavisringdin 5mo agoNo, it was a custom ASIC chip with weights baked in for a singular model. I do envision a future where we return to cartridges. Local AI is de facto and massively optimised chips are built to be plug and play running a single SoTA model.
- SJMG 5mo agoLikely https://taalas.com https://taalas.com
- deleted 5mo ago[deleted]
- Grosvenor 5mo agocerebras They built an entire wafer ASIC. The entire thing is one huge active ASIC. it takes a lot of cool engineering and cooling to make it work, and is very cool.
- tln 5mo agoTaalas. A sibling comment of yours posted the chat demo URL - https://chatjimmy.ai/ https://chatjimmy.ai/
- 2ndorderthought 5mo agoWoah. How is this working? It's stupid fast.
- mike_hearn 5mo agoThe weights are mapped directly to transistors. It's not a generic processor, it's literally a dedicated Llama 8B chip that can't be used for anything else. When you specialize in hardware you get faster - Taalas is pushing that to the limit. They seem to be doing well. I checked recently and their API is closed to signups due to overwhelming demand.
- 2ndorderthought 5mo agoI want to buy a chip not API access!!!
- jiggawatts 5mo agoMake you think... would it be possible to make an analog AI chip? I.e.: burn the weights into resistors with a range of possible values, and do the sums through simply adding up the currents along parallel paths by simply connecting them!