9 ms·
Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of G
by tome 3y ago
Hi folks, I work for Groq. Feel free to ask me any questions.
(If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
- mechagodzilla 3y agoYou all seem like one of the only companies targeting low-latency inference rather than focusing on throughput (and thus $/inference) - what do you see as your primary market?
- tome 3y agoYes, because we're one of the only companies whose hardware can actually support low latency! Everyone else is stuck with traditional designs and they try to make up for their high latency by batching to get higher throughput. But not all applications work with high throughput/high latency ... Low latency unlocks feeding the result of one model into the input of another model. Check out this conversational AI demo on CNN. You can't do that kind of thing unless you have low latency. https://www.youtube.com/watch?v=pRUddK6sxDg&t=235s https://www.youtube.com/watch?v=pRUddK6sxDg&t=235s
- vimarsh6739 3y agoMight be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760 https://arxiv.org/abs/1704.04760). If so, does Groq actually provide hardware support for LLM inference?
- tome 3y agoJonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU. Could you clarify your question about hardware support? Currently we build out our hardware to support our cloud offering, and we sell systems to enterprise customers.
- vimarsh6739 3y agoThanks for the quick reply! About hardware support, I was wondering if the LPU has a hardware instruction to compute the attention matrix similar to the MatrixMultiply/Convolve instruction in the TPU ISA. (Maybe a hardware instruction which fuses a softmax on the matmul epilogue?)
- tome 3y agoWe don't have a hardware instruction but we do have some patented technology around using a matrix engine to efficiently calculate other linear algebra operations such as convolution.
- mirekrusin 3y agoAre you considering targeting consumer market? There are a lot of people throwing $2k-$4k into local setups and they primarily care about inference.
- tome 3y agoAt the moment we're concentrating on building out our API and serving the enterprise market.
- liberix 3y agoHow do I sign up for API access? What payment methods do you support?
- louiscklaw 3y agoprice plan ?
- zawy 3y agoWhen you start using Samsung 4 nm are you switching from SRAM to HDM? If yes, how's that going to affect all the metrics given that SRAM is so much faster? Someone said you'll eventually move to HDM because SRAM improvements is relatively stalled. I read an article that indicated your Bill Of Materials compared to NVidia's is 10x to get 1/10 the latency, and 8x BOM for throughput if Nvidia optimizes for throughput? Does this seem accurate? That CAPEX is the primary drawback? https://www.semianalysis.com/p/groq-inference-tokenomics-speed-but https://www.semianalysis.com/p/groq-inference-tokenomics-spe...
- ppsreejith 3y agoThank you for doing this AMA 1. How many GroqCards are you using to run the Demo? 2. Is there a newer version you're using which has more SRAM (since the one I see online only has 230MB)? Since this seems to be the number that will drive down your cost (to take advantage of batch processing, CMIIW!) 3. Can TTS pipelines be integrated with your stack? If so, we can truly have very low latency calls! *Assuming you're using this: https://www.bittware.com/products/groq/ https://www.bittware.com/products/groq/
- tome 3y ago1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely, see this video of a live demo on CNN! https://www.youtube.com/watch?t=235&v=pRUddK6sxDg https://www.youtube.com/watch?t=235&v=pRUddK6sxDg
- ppsreejith 3y agoThank you, that demo was insane! Follow up (noob) question: Are you using a KV cache? That would significantly increase your memory requirements. Or are you forwarding the whole prompt for each auto-regressive pass?
- tome 3y agoYou're welcome! Yes, we have KV cache. Being able to implement this efficiently in terms of hardware requirements and compute time is one of the benefits of our deterministic chip architecture (and deterministic system architecture).
- ppsreejith 3y agoThanks again! Hope I'm not overwhelming but one more question: Are you decoding with batch size = 1 or is it more?
- Oras 3y agoImpressive speed. Are there any plans to run fine-tuned models?
- tome 3y agoYes, we're working on a feature to give our partners the ability to deploy their own fine-tuned models.
- dkhudia 3y ago@tome for the deterministic system, what if the timing for one chip/part is off due to manufacturing/environmental factors (e.g., temperature) ? How does the system handle this?
- mechagodzilla 3y agoThose sorts of issues are part of timing analysis for a chip, but once a chip's clock rate is set, they don't really factor in unless there is some kind of dynamic voltage/frequency scaling scheme going on. This chip probably does not do any of that and just uses a fixed frequency, so timing is perfectly predictable.
- tome 3y agoWe know the maximum possible clock drift and so we know when we need to do a resynchronisation to keep all the chips in sync. You can read about it in section 3.3 of our recent whitepaper: https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...
- phh 3y agoYou're running fp32 models, fp16 or quantized?
- tome 3y agoFP16 for calculating all activations. Some data is stored as FP8 at rest.
- andy_xor_andrew 3y agoare your accelerator chips designed in-house? or they're some specialized silicon or FPGPU or something that you wrote very optimized code for inference? it's really amazing! the first time I tried the demo, I had to try a few prompts to believe it wasn't just an animation :)
- tome 3y agoYup, custom ASIC, designed in-house, built into a system of several racks, hundreds of chips, with fast interconnect. Really glad you enjoyed it!
- itishappy 3y agoAlright, I'll bite. Haskell seems pretty unique in the ML space! Any unique benefits to this decision, and would you recommend it for others? What areas of your project do/don't use Haskell?
- tome 3y agoHaskell is a great language for writing compilers! The end of our compilation pipeline is written in Haskell. Other stages are written in C++ (MLIR) and Python. I'd recommend anyone to look at Haskell if they have a compiler-shaped problem, for sure. We also use Haskell on our infra team. Most of our CI infra is written in Haskell and Nix. Some of the chip itself was designed in Haskell (or maybe Bluespec, a Haskell-like language for chip design, I'm not sure).
- lemie 3y agoHaskell is a great language. However, when you want to build production grade software it’s the wrong language. Specially when it comes to a complicated piece of software like the compiler for a novel chip. I can tell you for a fact it has always being the wrong choice (specially in the case of Groq).
- gotem02 3y agoWhy? Is it just because it's too hard to read and write? Or is it lacking some features?
- ianpurton 3y agoIs it possible to buy Groq chips and how much do they cost?
- ComputerGuru 3y agohttps://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109?qs=ST9lo4GX8V2eGrFMeVQmFw%3D%3D https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109...
- UncleOxidant 3y agoOnly $20,625.00!
- LoganDark 3y agoPer chip? So the current demo with 568 chips costs.... $11,715,000?!?!
- deleted 3y ago[deleted]
- michaelbuckbee 3y agoFriendly fyi - I think this might just be a web interface bug but but I submitted a prompt with the Mixtral model and got a response (great!) then switched the dropdown to Llama and submitted the same prompt and got the exact same response. It may be caching or it didn't change the model being queried or something else.
- tome 3y agoThanks, I think it's because the chat context is fed back to the model for the next generation even when you switch models. If you refresh the page that should erase the history and you should get results purely from the model you choose.
- michaelbuckbee 3y agoAppreciate the quick reply! That's interesting.
- tome 3y agoYou're welcome. Thanks for reporting. It's pretty confusing so maybe we should change it :)
- pests 3y agoI've always liked how openrouter.ai does it They allow you to configure chat participants (a model + params like context or temp) and then each AI answers each question independently in-line so you can compare and remix outputs.
- xanderatallah 3y agoopenrouter dev here - would love to get Groq access and include it!
- kkzz99 3y agoHow does the Groq PCIE Card work exactly? Does it use system ram to stream the model data to the card? How many T/s could one expect with e.g. 36000Mhz DDR4 Ram?
- tome 3y agoWe build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.
- kkzz99 3y agoHow does that work when the card only has 230MB of SRAM?
- tome 3y agoWe connect hundreds of chips across several racks with fast interconnect.
- AhtiK 3y agoHow fast is the memory bandwidth of that fast interconnect?
- tome 3y agoHave a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public. https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...
- UncleOxidant 3y agoWhen will we be able to buy Groq accelerator cards that would be affordable for hobbyists?
- tome 3y agoWe are prioritising building out whole systems at the moment I don't think we'll have a consumer level offering in the near future.
- frognumber 3y agoI will mention: A lot of innovation in this space comes bottom-up. The sooner you can get something in the hands of individuals and smaller institutions, the better your market position will be. I'm coding to NVidia right now. That builds them a moat. The instant I can get other hardware working, the less of a moat they will have. The more open it is, the more likely I am to adopt it.
- tome 3y agoDefinitely, that's why we've opened our API to everyone.
- frognumber 3y agoI don't think that quite does it. What I'd want -- if you want me to support you -- is access to the chip, libraries, and API documentation. Best-case would be something I buy for <$2k (if out-of-pocket) or under $5k (if employer). Next best case would be a cloud service with a limited free tier. It's okay if it has barely enough quota that I can develop to it, but the quota should never expire. (The mistake a lot of services make is to limit free tier to e.g. 30 day or 1 year, rather than hours/month; if I didn't get around to evaluating, switch employers, switch projects, etc. the free tier is gone). I did sign up for your API service. I won't be able to use it in prod before your (very nice) privacy guarantees are turned into lawyer-compliant regulatory language. But it's an almost ideal fit for my application.
- tudorw 3y agoAs it works at inference do you think 'Representation Engineering ' could be applied to give a sort of fine-tuning ability? https://news.ycombinator.com/item?id=39414532 https://news.ycombinator.com/item?id=39414532
- karthityrion 3y agoHi. Are these ASICs only for LLMs or could they accelerate other kinds of models(vision) as well?
- tome 3y agoIt's a general purpose compute engine for numerical computing and linear algebra, so it can accelerate any ML workloads. Previously we've accelerated models for stabilising fusion reactions and for COVID drug discovery * https://alcf.anl.gov/news/researchers-accelerate-fusion-research-argonne-s-groq-ai-platform https://alcf.anl.gov/news/researchers-accelerate-fusion-rese... * https://wow.groq.com/groq-accelerates-covid-drug-discovery-333x-versus-legacy-solutions/ https://wow.groq.com/groq-accelerates-covid-drug-discovery-3...
- karthityrion 3y agoSo, is this specific chip only for LLMs as the name LPU suggests Language Processing Unit, right?
- tome 3y agoThe chip is capable of running general numerical compute, but because we're focusing almost entirely on LLMs at the moment we've branded it the LPU.
- jart 3y agoIf I understand correctly, you're using specialized hardware to improve token generation speed, which is very latency bound on the speed of computation. However generating tokens only requires multiplying 1-dimensional matrices usually. If I enter a prompt with ~100 tokens then your service goes much slower. Probably because you have to multiply 2-dimensional matrices. What are you doing to improve the computation speed of prompt processing?
- tome 3y agoI don't think it should be quadratic in input length. Why do you think it is?
- jart 3y agoall I know is that when I run llama.cpp a lot of the matrices that get multiplied have their shapes defined by how many tokens are in my prompt. https://justine.lol/tmp/shapes.png https://justine.lol/tmp/shapes.png Notice how the B matrix is always skinny for generating tokens. But for batch processing of the initial prompt, it's fat. It's not very hard to multiply a skinny matrix but once it's fat it gets harder. Handling the initial batch processing of the prompt appears to be what your service goes slow at.
- johndough 3y agoYou can ask your website: "What is the computational complexity of self-attention with respect to input sequence length?" It'll answer something along the lines of self-attention being O(n^2) (where n is the sequence length) because you have to compute an attention matrix of size n^2. There are other attention mechanisms with better computational complexity, but they usually result in worse large language models. To answer jart: We'll have to wait until someone finds a good linear attention mechanism and then wait some more until someone trains a huge model with it (not Groq, they only do inference).
- tome 3y agoOK, thanks, that's useful to know. Personally I'm not involved directly in implementing the model, so I don't know what we do there.
- BryanLegend 3y agoHow well would your hardware work for image/video generation?
- tome 3y agoIt should work great as far as I know. We've implemented some diffusion models for image generation but we don't offer them at the moment. I'm not aware of us having implemented any video models.
- pama 3y agoFYI, I only see a repeating animation and nothing else in my iPhone on lockdown mode, with Safari or Firefox.
- karthityrion 3y agoWhat is the underlying architecture of the ASICs. Does it use systolic arrays?
- tome 3y agoYes, our matrix engine is quite similar to a systolic array. You can find more details about our architecture in our paper: https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPaper2022_ASoftwareDefinedTensorStreamingMultiprocessorForLargeScaleMachineLearning-1.pdf https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...
- amirhirsch 3y agoIt seems like you are making general purpose chips to run many models. Are we at a stage where we can consider taping out inference networks directly propagating the weights as constants in the RTL design? Are chips and models obsoleted on roughly the same timelines?
- tome 3y agoI think the models change far too quickly for that to be viable. A chip has to last several years. Currently we're seeing groundbreaking models released every few months.