3 ms·
Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovatio
by tome 3y ago
Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovation as well in a sort of hardware-software codesign. We don't see transformer models as being an impediment in terms of memory requirements or speed in the near future.
[I would be happy to say more but I just got a top-level comment flag killed, presumably because they though I was advertising, so I won't mention the company name.]
- tome 3y agoLink to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199 https://news.ycombinator.com/item?id=38739199
- andy99 3y agoAre you guys hiring? I'm actually looking for a job in this industry.
- tome 3y agoYes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)
- andy99 3y agoAwesome, thanks! Is there someone there you could connect me with for an introductory chat? I put my contact info in my profile, I completely understand if it's not possible. Thanks!
- tome 3y agoYes, sure! I've emailed a recruiter to introduce you. Thanks for reaching out and showing interest.
- deleted 3y ago[deleted]
- benchess 3y agoIf I understand correctly, Groq chips have 220MB SRAM and the next best level is DDR4? How many chips are needed to run Llama2-70B at those speeds?
- tome 3y agoCool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's currently running at 275 tokens / sec I think might be running on two GroqRacks, so 128 chips. I'm not certain of either of these figures so please don't quote me! But those are the right ball-park.
- trsohmers 3y agoThis article from less than a month ago says that it is on 576 chips https://www.nextplatform.com/2023/11/27/groq-says-it-can-deploy-1-million-ai-inference-chips-in-two-years/ https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...
- tome 3y agoThanks, looks like you're right and this demo is running on 9 GroqRacks (576 chips). I think we may also have an 8 rack version in progress. We've tried a variety of different configurations to improve performance, which is possible because of the high level of flexibility and configurability of our architecture and compiler tool chain.