tome

33 ms·

601.
by tome 3y ago
You can check out all our API features here: https://docs.api.groq.com/
602.
by tome 3y ago
Both, we sell tokens as a service and we sell enterprise systems.
603.
by tome 3y ago
We are not using 500 GPUs, we are using a large system built from many of our own custom ASICs. This allows us to do batch size 1 with no reduction in overall throughput. (We are doing pipelining though, so many users are using the same s
604.
by tome 3y ago
Definitely, that's why we've opened our API to everyone.
605.
by tome 3y ago
Have a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public. https://wow.groq.com/wp-content/u
606.
by tome 3y ago
LLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated the 100th.
607.
by tome 3y ago
We don't think so, but you be the judge! I believe we quantize both Mixtral and Llama 2 in this way.
608.
by tome 3y ago
I don't know about GPT 3.5 specifically, but on this independent benchmark (LLMPerf) Groq's time to first token is also lowest: https://github.com/ray-project/llmperf-leaderboard?tab=readm...
609.
by tome 3y ago
The chip is capable of running general numerical compute, but because we're focusing almost entirely on LLMs at the moment we've branded it the LPU.
610.
by tome 3y ago
We don't have a hardware instruction but we do have some patented technology around using a matrix engine to efficiently calculate other linear algebra operations such as convolution.
611.
by tome 3y ago
Thanks, I've reported this internally.
612.
by tome 3y ago
Yes, our matrix engine is quite similar to a systolic array. You can find more details about our architecture in our paper: https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...
613.
by tome 3y ago
Jonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU. Could you clarify your question about hardware support? Currently we
614.
by tome 3y ago
West Coast US. You would have been placed in our queuing system because with all the attention we are getting we are very busy right now!
615.
by tome 3y ago
Non-deterministic timing characteristics.
616.
by tome 3y ago
Thomas, I think for full disclosure you should also state that you left Groq to start a competitor (a competitor which doesn't have the world's lowest latency LLM engine nor a guarantee to match the cheapest per token prices, like
617.
by tome 3y ago
You can find out about the chip to chip interconnect from our paper below, section 2.3. I don't think that's custom. We achieve low latency by basically being a software-defined architecture. Our functional units operate complet
618.
by tome 3y ago
Yes, there's a huge amount of demand because Twitter discovered us yesterday. There will be a backlog, so sorry about that.
619.
by tome 3y ago
It's a general purpose compute engine for numerical computing and linear algebra, so it can accelerate any ML workloads. Previously we've accelerated models for stabilising fusion reactions and for COVID drug discovery * https:&
620.
by tome 3y ago
You're welcome. Thanks for reporting. It's pretty confusing so maybe we should change it :)
621.
by tome 3y ago
It should work great as far as I know. We've implemented some diffusion models for image generation but we don't offer them at the moment. I'm not aware of us having implemented any video models.
622.
by tome 3y ago
I don't think it should be quadratic in input length. Why do you think it is?
623.
by tome 3y ago
I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.
624.
by tome 3y ago
The speed part. We're not interested in being swallowed. The aim is to be bigger than Nvidia in three years :)
625.
by tome 3y ago
Definitely not, but even with a comparison to 500 GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :)
626.
by tome 3y ago
The weights are quantized to FP8 when they're stored in memory, but all the activations are computed at full FP16 precision.
627.
by tome 3y ago
Really glad you like it! We've been working hard on it.
628.
by tome 3y ago
By the way, we also have a new Discord server where we are hosting our developer community. If you find anything missing in our API you can ask about there: https://discord.com/invite/TQcy5EBdCP
629.
by tome 3y ago
The industry as a whole is a very long way away from that. The power requirements are too high for mobile.
630.
by tome 3y ago
> However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? For API access to our tokens as a service we guarantee to beat an

More ›