4 ms·

I believe until the hardware designs catch up to be more commodized ala cryto mining evolution from GPUs to ASICS for specfic algos. Designs (like Google TPUs e

I believe until the hardware designs catch up to be more commodized ala cryto mining evolution from GPUs to ASICS for specfic algos. Designs (like Google TPUs equivalent) would also need to evolve to be more memory dense to be able to handle them. Untill then it seems will be system time shares for the larger models , probably with a bring your own model and pay as you go.

Dylan16807 4mo ago
> ala cryto mining evolution from GPUs to ASICS for specfic algos I don't see it happening. A current gen GPU with a huge and fast block of memory isn't a perfect fit for these algorithms but it's relatively close. With cryptocurrency, mass small sha256 hashing was a totally different kind of computation.
- fc417fc802 4mo ago
  > isn't a perfect fit for these algorithms but it's relatively close I don't think that's true. The best fit out of what's presently available perhaps. Inference is almost entirely memory bandwidth bound at present, to the extent that GPUs with HBM have a massive advantage over those with GDDR. TPUs appear to be a much better overall design. I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards.
  - Dylan16807 4mo ago
    > I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards. Processing scales better than DRAM does. I think an HBM-like stack where the bottom layer has the math units is probably the ultimate form of that. And it's possible that flash instead of DRAM is actually the better play, as long as you can hook up enough in parallel. RIP Optane.
- nobrains 4mo ago
  [flagged]
- wcoenen 4mo ago
  > I don't see it happening. Isn't it already happening with Cerebras? It's mentioned at the end of OpenAI's GPT 5.6 announcement: "We're also launching GPT‑5.6 Sol on Cerebras at up to 750 tokens per second in July"

Dylan16807 4mo ago

> ala cryto mining evolution from GPUs to ASICS for specfic algos I don't see it happening. A current gen GPU with a huge and fast block of memory isn't a perfect fit for these algorithms but it's relatively close. With cryptocurrency, mass small sha256 hashing was a totally different kind of computation.

fc417fc802 4mo ago
> isn't a perfect fit for these algorithms but it's relatively close I don't think that's true. The best fit out of what's presently available perhaps. Inference is almost entirely memory bandwidth bound at present, to the extent that GPUs with HBM have a massive advantage over those with GDDR. TPUs appear to be a much better overall design. I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards.
- Dylan16807 4mo ago
  > I expect that a hypothetical advance in fabrication enabling processing elements to be placed directly adjacent to dense RAM on the same silicon (not merely in the same package) would be superior in all regards. Processing scales better than DRAM does. I think an HBM-like stack where the bottom layer has the math units is probably the ultimate form of that. And it's possible that flash instead of DRAM is actually the better play, as long as you can hook up enough in parallel. RIP Optane.
nobrains 4mo ago
[flagged]
wcoenen 4mo ago
> I don't see it happening. Isn't it already happening with Cerebras? It's mentioned at the end of OpenAI's GPT 5.6 announcement: "We're also launching GPT‑5.6 Sol on Cerebras at up to 750 tokens per second in July"