3 ms·

I think that goal is fine and good, but I would rather see huge investments toward in-memory compute like ReRAM and such. If we bridge the efficiency advancemen

I think that goal is fine and good, but I would rather see huge investments toward in-memory compute like ReRAM and such. If we bridge the efficiency advancements of TinyML with the leap of LLM abilities, perhaps we can start on the road of not being limited by the impact of training on climate.

asicsarecool 3y ago
Why the fuck was this downvoted. Very occasionally I get the feeling HN is entering the /. phase
- wmf 3y ago
  I didn't downvote it but in-memory compute is crackpot and alternative memory tech is really crackpot. It's not going to happen and it's ridiculous to propose it on the same level as GPUs with more RAM.
  - zozbot234 3y ago
    GPU's are already closer to "in-memory compute" compared to CPU's. It's just taking the existing pattern of NUMA (non-uniform memory access) to a greater extent, as a principled approach to the so-called 'Von Neumann bottleneck'.
  - imtringued 3y ago
    In-memory compute is very easy, you just don't have to fall for the pipe dream of using the same process for both the memory and the compute. All you have to do is follow a package on package strategy like we already do with smartphones. A Raspberry PI 5 gets 25GB/s memory bandwidth and it only has a single DRAM chip if I recall correctly. So if you had a DIMM with 16 of these chips, you would already be on the same bandwidth as HBM. 96 DIMMs and you get 40TB/s memory bandwidth.
    - deleted 3y ago
      [deleted]
  - datameta 3y ago
    Can you explain further why you believe it isn't worth exploring? Is it that you can't imagine how it would scale to the level of today's high performance compute hardware? Just ten years back, squeezing ML models onto microcontrollers sounded completely insane, given their tight memory and power constraints. We've seen NN compilers developed, game-changing techniques like quantization, pruning, and graph-level optimizations pruning. This allowed deployment of ML models in microcontrollers with a newly developed framework like TFLite Micro.
    - wmf 3y ago
      These technologies have already been explored and they failed every time. It's throwing good money after bad. Also, speculative basic research isn't comparable to adding more RAM to a graphics card. You can't substitute one for the other.
epistasis 3y ago
Agree with the comment, but riffing on the last few words: Climate is pretty much my #1 concern about the world, but LLM use of energy is really really far down on the list of important actions for climate. First and foremost are removing roadblocks for deploying existing technologies for clean energy, and speeding up the necessary supporting infrastructure such as transmission and market policies for choosing cheapest possible solutions (over the objections of dinosaur execs that choose last century's solutions). Then the big hard to decarbonize parts of industry like cement and steel, as well as deploying electrolyzers to get ammonia fertilizer production switched over to carbon neutral production rather than from fossil-generated hydrogen. Reducing energy consumption is important for advancing AI in general, but ultimately all its energy consumption will be from clean energy sources anyway, and the switch that needs to happen is that switch in energy sources. Reducing energy use by 2x or 10x is not good enough, we must change the sources fundamentally.

asicsarecool 3y ago

Why the fuck was this downvoted. Very occasionally I get the feeling HN is entering the /. phase

wmf 3y ago
I didn't downvote it but in-memory compute is crackpot and alternative memory tech is really crackpot. It's not going to happen and it's ridiculous to propose it on the same level as GPUs with more RAM.
- zozbot234 3y ago
  GPU's are already closer to "in-memory compute" compared to CPU's. It's just taking the existing pattern of NUMA (non-uniform memory access) to a greater extent, as a principled approach to the so-called 'Von Neumann bottleneck'.
- imtringued 3y ago
  In-memory compute is very easy, you just don't have to fall for the pipe dream of using the same process for both the memory and the compute. All you have to do is follow a package on package strategy like we already do with smartphones. A Raspberry PI 5 gets 25GB/s memory bandwidth and it only has a single DRAM chip if I recall correctly. So if you had a DIMM with 16 of these chips, you would already be on the same bandwidth as HBM. 96 DIMMs and you get 40TB/s memory bandwidth.
  - deleted 3y ago
    [deleted]
- datameta 3y ago
  Can you explain further why you believe it isn't worth exploring? Is it that you can't imagine how it would scale to the level of today's high performance compute hardware? Just ten years back, squeezing ML models onto microcontrollers sounded completely insane, given their tight memory and power constraints. We've seen NN compilers developed, game-changing techniques like quantization, pruning, and graph-level optimizations pruning. This allowed deployment of ML models in microcontrollers with a newly developed framework like TFLite Micro.
  - wmf 3y ago
    These technologies have already been explored and they failed every time. It's throwing good money after bad. Also, speculative basic research isn't comparable to adding more RAM to a graphics card. You can't substitute one for the other.

epistasis 3y ago

Agree with the comment, but riffing on the last few words: Climate is pretty much my #1 concern about the world, but LLM use of energy is really really far down on the list of important actions for climate. First and foremost are removing roadblocks for deploying existing technologies for clean energy, and speeding up the necessary supporting infrastructure such as transmission and market policies for choosing cheapest possible solutions (over the objections of dinosaur execs that choose last century's solutions). Then the big hard to decarbonize parts of industry like cement and steel, as well as deploying electrolyzers to get ammonia fertilizer production switched over to carbon neutral production rather than from fossil-generated hydrogen. Reducing energy consumption is important for advancing AI in general, but ultimately all its energy consumption will be from clean energy sources anyway, and the switch that needs to happen is that switch in energy sources. Reducing energy use by 2x or 10x is not good enough, we must change the sources fundamentally.