tome

36 ms·

631.
by tome 3y ago
We connect hundreds of chips across several racks with fast interconnect.
632.
by tome 3y ago
That could be possible. It's out of my area of expertise so I can't say for sure. My understanding was HBM forces on you specific access patterns and non-deterministic delays. Our compiler already deals with many other forms of
633.
by tome 3y ago
We're providing by far the lowest latency LLM engine on the planet. You can't reduce latency by scaling horizontally.
634.
by tome 3y ago
Yes, we pipeline requests so multiple users are being handled by the same hardware at one time.
635.
by tome 3y ago
We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits
636.
by tome 3y ago
Haskell is a great language for writing compilers! The end of our compilation pipeline is written in Haskell. Other stages are written in C++ (MLIR) and Python. I'd recommend anyone to look at Haskell if they have a compiler-shaped
637.
by tome 3y ago
We are prioritising building out whole systems at the moment I don't think we'll have a consumer level offering in the near future.
638.
by tome 3y ago
If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with f
639.
by tome 3y ago
You can find our API docs here, including details of our OpenAI compatibility https://docs.api.groq.com/
640.
by tome 3y ago
FP16 for calculating all activations. Some data is stored as FP8 at rest.
641.
by tome 3y ago
Yup, custom ASIC, designed in-house, built into a system of several racks, hundreds of chips, with fast interconnect. Really glad you enjoyed it!
642.
by tome 3y ago
Yeah, this is a common observation. See my comment at https://news.ycombinator.com/item?id=39431921 Maybe we should change the behavior to stop people getting confused.
643.
by tome 3y ago
Yes, because we're one of the only companies whose hardware can actually support low latency! Everyone else is stuck with traditional designs and they try to make up for their high latency by batching to get higher throughput. But no
644.
by tome 3y ago
Thanks, I think it's because the chat context is fed back to the model for the next generation even when you switch models. If you refresh the page that should erase the history and you should get results purely from the model you cho
645.
by tome 3y ago
As a fellow scientist I concur with the approach of skepticism by default. Our chat app and API are available for everyone to experiment with and compare output quality with any other provider. I hope you are enjoying your time of having an
646.
by tome 3y ago
Thanks!
647.
by tome 3y ago
We run the open source models that everyone else has access to. What we're trying to show off is our low latency and high throughput, not the model itself.
648.
by tome 3y ago
I don't think we've put a GroqChip in a mobile device yet. Interesting idea!
649.
by tome 3y ago
Nice prompting strategy :)
650.
by tome 3y ago
Yes, we're working on a feature to give our partners the ability to deploy their own fine-tuned models.
651.
by tome 3y ago
It's a system built from hundreds of GroqChips (a custom ASIC we designed). We call it the LPU (language processing unit). Unlike graphics processors, which are still best in class for training, LPUs are best in class for low latenc
652.
by tome 3y ago
Yes, agreed. We believe the benefits of reducing latency are non-linear. You can hit different phase changes as the latency reduces and new applications become viable. Roundtripping text-to-speech and speech-to-text is one example. We&#x
653.
by tome 3y ago
We know the maximum possible clock drift and so we know when we need to do a resynchronisation to keep all the chips in sync. You can read about it in section 3.3 of our recent whitepaper: https://wow.groq.com/wp-content&#x
654.
by tome 3y ago
Yeah the confusion has happened a lot to me too. All I know is that it's in the hands of our legal team.
655.
by tome 3y ago
Yeah, I don't know what the cost to us is to build out our own hardware but it's significantly less expensive than retail.
656.
by tome 3y ago
Yeah, it's nothing to do with Elon and we (Groq) had the name first. It's a natural choice of name for something in the field of AI because of the connections to the hacker ethos, but we have the trademark and Elon doesn't.
657.
by tome 3y ago
Mostly available as a service via cloud API at the moment. The systems themselves are too big for consumers but we will sell systems to corporations.
658.
by tome 3y ago
That's OK, feel free to keep asking! I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.
659.
by tome 3y ago
Some other discussions about Groq going on at * https://news.ycombinator.com/item?id=39428880 * https://news.ycombinator.com/item?id=39425949
660.
by tome 3y ago
You're welcome! Yes, we have KV cache. Being able to implement this efficiently in terms of hardware requirements and compute time is one of the benefits of our deterministic chip architecture (and deterministic system architecture).

More ›