3 ms·

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised

by ilaksh 1y ago

There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. But one has to imagine that seeing so many huge datacenters go up and not being able to do training runs etc. is motivating a lot of researchers to try things that are really different. At least I hope so. It seems pretty short sighted that the funding numbers for memristor startups (for example) are so low so far. Anyway, assuming that within the next several years more radically different AI hardware and AI architecture paradigms pay off in efficiency gains, the current situation will change. Fully human level AI will be commoditized, and training will be well within the reach of small companies. I think we should anticipate this given the strong level of need to increase efficiency dramatically, the number of existing research programs, the amount of investment in AI overall, and the history of computation that shows numerous dramatic paradigm shifts. So anyway "the rest of us" I think should be banding together and making much larger bets on proving and scaling radical new AI hardware paradigms.

sidewndr46 1y ago
I think a pretty good chunk of HP's history explains why memristors don't get used in a commercial capacity.
- ofrzeta 1y ago
  You remember The Machine? I had a vague memory but I had to look it up.
michelpp 1y ago
Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind of efficiency.
- lazide 1y ago
  Memristors have been tried for literally decades. If the posters other guesses pay out the same rate, this will likely play out never.
  - ilaksh 1y ago
    Other technologies tried for decades before becoming huge: Neural-network AI; Electric cars; mRNA vaccines; Solar photovoltaics; LED lighting
    - lazide 1y ago
      Ho boy, should we start listing the 10x number of things that went in the wastebasket too?
      - ToValueFunfetti 1y ago
        If I only have to try 11 things for one of them to be LED lights or electric cars, I'd better get trying. Sure, I might have to empty a wastebasket at some point, but I'll just pay someone for that.
        lazide 1y ago
        This fundamentally at odds with picking one tech and saying ‘this is the winner’ eh? Which is what the prior comment was about.
        ToValueFunfetti 1y ago
        Which prior comment? Top level offers two options plus an 'etc.', second comment adds another and says 'could be', and then first reply to you offers other tech that took decades of R&D to suggest we can't rule out memristors. I don't see what you're referring to.
        1y ago
thekoma 1y ago
Even in that scenario, what would stop the likes of OpenAI to instead throw 50M+ a day to the new way of doing things and still outcompete smaller fry?
- manquer 1y ago
  The fastest away to acquire the know-how to do for Big Co is to get the talent who have spent the years in building the new tech. Poaching, acquihirng or acquisitions and the myriad modern forms we are seeing today have been the tools and will not change. Owners and beneficiaries of the capital do not change, but that is an artifact of our economic system and is much larger a socio-economic discussion beyond the scope of innovation and research
hnuser123456 1y ago
>memory-centric compute This already exists: https://www.cerebras.ai/chip https://www.cerebras.ai/chip They claim 44 GB of SRAM at 21 PB/s.
- cma 1y ago
  They use separate memory servers, networked memory adjacent adjacent compute with small amounts of fast local memory. Waferscale severely limits bandwidth once you go beyond SRAM, because with far less chip perimeter per unit area there is less place to hook up IO.
marcosdumay 1y ago
Memristors in particular just won't happen. But memory-centric compute didn't happen because of Moore's law. (SNNs have the problem that we don't actually know how to use them.) Now that it's gone, it may have a chance, but it still takes a large amount of money thrown into the idea and the people with money are so risk-adverse that they create entire new risks for themselves. Forward neural networks were very lucky that there existed a mainstream use for the kind of hardware it needed.
plastic-enjoyer 1y ago
> There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. Because the alternatives lack the breakthroughs that give them an edge against current-state AI and don't generate the hype like transformers or diffusion models. You have stuff like neuromorphic hardware that is hardly accessible and in its infancy, e.g. SpiNNaker. You have disciplines like Computational Neuroscience that try to model the brain and come up with novel models and algorithms for learning, which, however, are computational expensive or just perform worse than conventional deep learning models and may benefit from neuromorphic hardware. But again, access is difficult to such hardware.

by ilaksh 1y ago

sidewndr46 1y ago

I think a pretty good chunk of HP's history explains why memristors don't get used in a commercial capacity.

ofrzeta 1y ago
You remember The Machine? I had a vague memory but I had to look it up.

michelpp 1y ago

Not sure why this is being downvoted, it's a thoughtful comment. I too see this crisis as an opportunity to push boundaries past current architectures. Sparse models for example show a lot of promise and more closely track real biological systems. The human brain has an estimated graph density of 0.0001 to 0.001. Advances in sparse computing libraries and new hardware architectures could be key to achieving this kind of efficiency.

lazide 1y ago
Memristors have been tried for literally decades. If the posters other guesses pay out the same rate, this will likely play out never.
- ilaksh 1y ago
  Other technologies tried for decades before becoming huge: Neural-network AI; Electric cars; mRNA vaccines; Solar photovoltaics; LED lighting
  - lazide 1y ago
    Ho boy, should we start listing the 10x number of things that went in the wastebasket too?
    - ToValueFunfetti 1y ago
      If I only have to try 11 things for one of them to be LED lights or electric cars, I'd better get trying. Sure, I might have to empty a wastebasket at some point, but I'll just pay someone for that.
      - lazide 1y ago
        This fundamentally at odds with picking one tech and saying ‘this is the winner’ eh? Which is what the prior comment was about.
        ToValueFunfetti 1y ago
        Which prior comment? Top level offers two options plus an 'etc.', second comment adds another and says 'could be', and then first reply to you offers other tech that took decades of R&D to suggest we can't rule out memristors. I don't see what you're referring to.
        1y ago

thekoma 1y ago

Even in that scenario, what would stop the likes of OpenAI to instead throw 50M+ a day to the new way of doing things and still outcompete smaller fry?

manquer 1y ago
The fastest away to acquire the know-how to do for Big Co is to get the talent who have spent the years in building the new tech. Poaching, acquihirng or acquisitions and the myriad modern forms we are seeing today have been the tools and will not change. Owners and beneficiaries of the capital do not change, but that is an artifact of our economic system and is much larger a socio-economic discussion beyond the scope of innovation and research

hnuser123456 1y ago

>memory-centric compute This already exists: https://www.cerebras.ai/chip https://www.cerebras.ai/chip They claim 44 GB of SRAM at 21 PB/s.

cma 1y ago
They use separate memory servers, networked memory adjacent adjacent compute with small amounts of fast local memory. Waferscale severely limits bandwidth once you go beyond SRAM, because with far less chip perimeter per unit area there is less place to hook up IO.

marcosdumay 1y ago

Memristors in particular just won't happen. But memory-centric compute didn't happen because of Moore's law. (SNNs have the problem that we don't actually know how to use them.) Now that it's gone, it may have a chance, but it still takes a large amount of money thrown into the idea and the people with money are so risk-adverse that they create entire new risks for themselves. Forward neural networks were very lucky that there existed a mainstream use for the kind of hardware it needed.

plastic-enjoyer 1y ago

> There is huge pressure to prove and scale radical alternative paradigms like memory-centric compute such as memristors, or SNNs, etc. That's why I am surprised we don't hear a lot about very large speculative investments in these directions to dramatically multiply AI compute efficiency. Because the alternatives lack the breakthroughs that give them an edge against current-state AI and don't generate the hype like transformers or diffusion models. You have stuff like neuromorphic hardware that is hardly accessible and in its infancy, e.g. SpiNNaker. You have disciplines like Computational Neuroscience that try to model the brain and come up with novel models and algorithms for learning, which, however, are computational expensive or just perform worse than conventional deep learning models and may benefit from neuromorphic hardware. But again, access is difficult to such hardware.