3 ms·

I do like the trend. Imagine if Anthropic or someone eventually release a Claude 3.5 but at like a whopping 10x its current speed. Would be incredibly more us

by TIPSIO 2y ago

I do like the trend. Imagine if Anthropic or someone eventually release a Claude 3.5 but at like a whopping 10x its current speed. Would be incredibly more useful and game changing than a slow o1 model that may or not be x percent smarter.

sgt101 2y ago
We might see that with the inference ASICs later this year I guess?
- xendipity 2y ago
  Ooh, what are these ASICs you're talking about? My understanding was that we'll see AMD/Nvidia gpus continue to be pushed and very competitive as well as have new system architectures like cerebras or grok. I haven't heard about new compute platforms framed as ASICs.
  - Workaccount2 2y ago
    Cerebras has ridiculously large LLM ASICs that can hit crazy speeds. You can try it with llama 8B and 70B: https://inference.cerebras.ai/ https://inference.cerebras.ai/ It's pretty fast, but my understanding is that it is still too expensive even accounting for the speed-up.
    - throwup238 2y ago
      Is Cerebras an integrated circuit or more an integrated wafer? :-) And yeah their cost is ridiculous, on the order for high 6 to low 7 figures per wafer. The rack alone looks several times more expensive than the 8x NVIDIA pods [1] [1] https://web.archive.org/web/20230812020202/https://www.youtube.com/watch?v=pzyZpauU3Ig https://web.archive.org/web/20230812020202/https://www.youtu...
  - campers 2y ago
    https://www.etched.com/announcing-etched https://www.etched.com/announcing-etched I think there's another one but I can't remember the name of it. Also a bit further out is https://spectrum.ieee.org/superconducting-computer https://spectrum.ieee.org/superconducting-computer "Instead of the transistor, the basic element in superconducting logic is the Josephson-junction."
bangaladore 2y ago
Sonnet 3.5 is fast for its quality. But yeah, it's nowhere near Google's flash models. But I assume that is largely just because its a much smaller model.

by TIPSIO 2y ago

sgt101 2y ago

We might see that with the inference ASICs later this year I guess?

xendipity 2y ago
Ooh, what are these ASICs you're talking about? My understanding was that we'll see AMD/Nvidia gpus continue to be pushed and very competitive as well as have new system architectures like cerebras or grok. I haven't heard about new compute platforms framed as ASICs.
- Workaccount2 2y ago
  Cerebras has ridiculously large LLM ASICs that can hit crazy speeds. You can try it with llama 8B and 70B: https://inference.cerebras.ai/ https://inference.cerebras.ai/ It's pretty fast, but my understanding is that it is still too expensive even accounting for the speed-up.
  - throwup238 2y ago
    Is Cerebras an integrated circuit or more an integrated wafer? :-) And yeah their cost is ridiculous, on the order for high 6 to low 7 figures per wafer. The rack alone looks several times more expensive than the 8x NVIDIA pods [1] [1] https://web.archive.org/web/20230812020202/https://www.youtube.com/watch?v=pzyZpauU3Ig https://web.archive.org/web/20230812020202/https://www.youtu...
- campers 2y ago
  https://www.etched.com/announcing-etched https://www.etched.com/announcing-etched I think there's another one but I can't remember the name of it. Also a bit further out is https://spectrum.ieee.org/superconducting-computer https://spectrum.ieee.org/superconducting-computer "Instead of the transistor, the basic element in superconducting logic is the Josephson-junction."

bangaladore 2y ago

Sonnet 3.5 is fast for its quality. But yeah, it's nowhere near Google's flash models. But I assume that is largely just because its a much smaller model.