tome

52 ms·

721.
by tome 3y ago
Our founder/CEO Jonathan Ross is a big fan of Haskell and used Haskell to design the first version of the TPU whilst he was working for Google. When he founded Groq he and the early team designed some parts of our chip using Haskell t
722.
by tome 3y ago
Link to HN discussion about the appliance: https://news.ycombinator.com/item?id=38739199
723.
by tome 3y ago
What do you mean by verification? That it computes the exact same output?
724.
by tome 3y ago
Yeah you could get something reasonable :) Reach out to our sales team: https://groq.com/contact/
725.
by tome 3y ago
It's a completely custom ASIC. Haskell was used in the hardware design, in a Bluespec-like way. Some parts of the compiler tool chain and infrastructure are also written in Haskell. We have loads of C++ and Python too, as you would
726.
by tome 3y ago
The number of input tokens is important because the bigger the context length the better. (I think our demo here is 4096 tokens of context.) But in terms of compute the important factor is how quickly you can generate the output. You wan
727.
by tome 3y ago
Yup, graphics processors are still the best for training. Groq's language processors (LPUs) are the state of the art for inference, far faster than any competitors. We have an open challenge to our competitors: can you match our infe
728.
by tome 3y ago
Thanks for sharing. It's the same silicon architecture as in that talk. We have built out different system architectures based on that silicon, and this is our fastest one so far for LLMs. Expect to see even more speed increases soo
729.
by tome 3y ago
nomel gave a good answer in a different thread > This is not about the model, it’s about the relative speed improvement from the hardware, with this model as a demo. To compare apples to apples look at the tokens per second of other syst
730.
by tome 3y ago
We're building out racks as fast as we can to keep up with customer demand :) A public demo of Mixtral is in the works, so watch this space.
731.
by tome 3y ago
Yup, deploying Mixtral is a work in progress. Watch this space!
732.
by tome 3y ago
Our hardware architecture was not designed with LLMs in mind, let alone a specific model. It's a general purpose numerical compute fabric. Our compiler allows us to quickly deploy new models of any architecture without the need that
733.
by tome 3y ago
We don't have an API in public availability yet but that's coming soon in the new year. We will be price competitive with OpenAI but much faster. Deploying Mixtral is work in progress so keep your eyes open for that too!
734.
by tome 3y ago
Yup, we will be price competitive with OpenAI, and much faster!
735.
by tome 3y ago
I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total. Graphics processors are still the best for tra
736.
by tome 3y ago
We've built and deployed racks at a number of organizations. Can you write a message to sales explaining your needs? https://groq.com/contact/ Or if you give me your contact details I can pass them on.
737.
by tome 3y ago
You can buy individual cards. For example Bittware is a reseller: https://www.bittware.com/products/groq/ But it might be best if you just contact us to explain your needs: https://groq.com/contac
738.
by tome 3y ago
It's not fixed and our chip wasn't designed with LLMs in mind. It's a general purpose, low latency, high throughput compute fabric. Our compiler toolchain is also general purpose and can compile arbitrary high performance nu
739.
by tome 3y ago
Thanks, I need to correct my earlier guess: I believe this demo is running on 9 GroqRacks (576 chips) and I think we may also have an 8 rack version in progress. I can't remember off the top of my head whether this deployment has pipe
740.
by tome 3y ago
Thanks, looks like you're right and this demo is running on 9 GroqRacks (576 chips). I think we may also have an 8 rack version in progress. We've tried a variety of different configurations to improve performance, which is poss
741.
by tome 3y ago
Cool that you know the tech specs of the GroqChip! Yes, that's right, 220 MB of SRAM per chip. I think the demo where we first broke 200 tokens / sec was running on 1 GroqRack, so 64 chips. The live public demo that's curr
742.
by tome 3y ago
Yes, sure! I've emailed a recruiter to introduce you. Thanks for reaching out and showing interest.
743.
by tome 3y ago
Yes we are! (I put links in my profile because I'm worried if link here or mention the name I'll be flag killed again.)
744.
by tome 3y ago
Yup, the company I work for has the world's fastest LLM appliance, based on a hardware design co-locating memory and compute. It requires compilation innovation as well in a sort of hardware-software codesign. We don't see trans
745.
by tome 3y ago
We use some of these techniques at Groq to make the world's fastest LLM appliance! You can see for yourself how fast it is at https://groq.com/ (click "Launch demo" in the top right). The model is Llama 2 70
746.
by tome 3y ago
Hmm, not sure what you mean about language extensions. Most language extensions just remove unnecessary restrictions in the language. What are you thinking of?
747.
by tome 3y ago
> turn on "diff3" ... there are many situations where you just don't have enough information to properly resolve a merge without it That's right. Here's an example I wrote up: https://stackoverflow.com
748.
by tome 3y ago
Correct. It's pretty sad to see such fundamental lack of understanding of economic principles.
749.
by tome 3y ago
Well, it's a good question. I'm in favour of using simple[+], abstract Haskell for practical apps, welcoming the vast majority of professional developers! [+] "simple" in that the solution to a problem is expressed in a
750.
by tome 3y ago
> it was fun watching advocates of complex, abstract Haskell argue against straightforward programming that normal developers can understand I don't think that's what they were arguing against. I think they were arguing against

More ›