3 ms·

It is incredibly fast, on that I agree, but even simple queries I tried got very inaccurate answers. Which makes sense, it's essentially a trade off of how much

It is incredibly fast, on that I agree, but even simple queries I tried got very inaccurate answers. Which makes sense, it's essentially a trade off of how much time you give it to "think", but if it's fast to the point where it has no accuracy, I'm not sure I see the appeal.

kaashif 8mo ago
If it's incredibly fast at a 2022 state of the art level of accuracy, then surely it's only a matter of time until it's incredibly fast at a 2026 level of accuracy.
- PrimaryExplorer 8mo ago
  yeah this is mindblowing speed. imagine this with opus 4.6 or gpt 5.2. probably coming soon
  - scotty79 8mo ago
    I'd be happy if they can run GLM 5 like that. It's amazing at coding.
- Gud 8mo ago
  Why do you assume this? I can produce total jibberish even faster, doesn’t mean I produce Einstein level thought if I slow down
  - andy12_ 8mo ago
    Not what he said.
  - Closi 8mo ago
    Better models already exist, this is just proving you can dramatically increase inference speeds / reduce inference costs. It isn't about model capability - it's about inference hardware. Same smarts, faster.
scotty79 8mo ago
I think it might be pretty good for translation. Especially when fed with small chunks of the content at a time so it doesn't lose track on longer texts.
andrewdea 8mo ago
the hardwired model is Llama 3.1 8B, which is a lightweight model from two years ago. Unlike other models, it doesn't use "reasoning:" the time between question and answer is spent predicting the next tokens. It doesn't run faster because it uses less time to "think," It runs faster because its weights are hardwired into the chip rather than loaded from memory. A larger model running on a larger hardwired chip would run about as fast and get far more accurate results. That's what this proof of concept shows
- Etheryte 8mo ago
  I see, that's very cool, that's the context I was missing, thanks a lot for explaining.
  - Sabinus 8mo ago
    I don't mean to be rude, but did you read the article before commenting?
    - Etheryte 8mo ago
      I'm commenting on the link to their demo, not on the article.

kaashif 8mo ago

If it's incredibly fast at a 2022 state of the art level of accuracy, then surely it's only a matter of time until it's incredibly fast at a 2026 level of accuracy.

PrimaryExplorer 8mo ago
yeah this is mindblowing speed. imagine this with opus 4.6 or gpt 5.2. probably coming soon
- scotty79 8mo ago
  I'd be happy if they can run GLM 5 like that. It's amazing at coding.
Gud 8mo ago
Why do you assume this? I can produce total jibberish even faster, doesn’t mean I produce Einstein level thought if I slow down
- andy12_ 8mo ago
  Not what he said.
- Closi 8mo ago
  Better models already exist, this is just proving you can dramatically increase inference speeds / reduce inference costs. It isn't about model capability - it's about inference hardware. Same smarts, faster.

scotty79 8mo ago

I think it might be pretty good for translation. Especially when fed with small chunks of the content at a time so it doesn't lose track on longer texts.

andrewdea 8mo ago

the hardwired model is Llama 3.1 8B, which is a lightweight model from two years ago. Unlike other models, it doesn't use "reasoning:" the time between question and answer is spent predicting the next tokens. It doesn't run faster because it uses less time to "think," It runs faster because its weights are hardwired into the chip rather than loaded from memory. A larger model running on a larger hardwired chip would run about as fast and get far more accurate results. That's what this proof of concept shows

Etheryte 8mo ago
I see, that's very cool, that's the context I was missing, thanks a lot for explaining.
- Sabinus 8mo ago
  I don't mean to be rude, but did you read the article before commenting?
  - Etheryte 8mo ago
    I'm commenting on the link to their demo, not on the article.