4 ms·

Next step - compile straight to verilog so I can buy some LLMs on aliexpress

by baq 1y ago

bigcat12345678 1y ago
https://riscv.org/blog/2021/02/hardware-description-language-chisel-diplomacy-deeper-dive/ https://riscv.org/blog/2021/02/hardware-description-language... That was one of the promising ideas before AI & GPUs come to the scene. As CPUs are stagnant, and naturally people want further optimize the middle layers software and hardware. But I suspect parallel computing in GPU style is going to dominate acclerated computing. General purpose CPUs are going to stay to become the little brain that orchestrates GPUs. Ideas of software direct to hardware transition might never be the mainstream.
- baq 1y ago
  I'm thinking more like pseudointellect over serial to attach a $3 esp32 to. Since it's basically tokens in, tokens out, let's just cut the unnecessary parts out. It's like querying the cloud models, except it's your silicon you personally soldered to the esp so nobody will break your home assistant with a system prompt update or a fine tuning run.
- mycall 1y ago
  > General purpose CPUs are going to stay to become the little brain that orchestrates GPUs Brings the deterministic compute to the indeterministic.
- jerf 1y ago
  "General purpose CPUs are going to stay to become the little brain that orchestrates GPUs." If that was going to happen, it would have happened. CPUs are genuinely good at what they do, and "what they do" is a lot of tasks that GPUs are actually terrible at. If all we had were GPUs in the world and someone invented a CPU, we'd hail them as a genius. A lot of people seem to think that GPUs are just "better", just ambiently better at everything, but that's lightyears from the truth. They are quite spectacularly terrible at a lot of very common tasks. There's many very good reasons that GPUs are still treated as accelerators for the CPUs and not vice versa.
anitil 1y ago
I mean.... LLM-in-a-box would actually be pretty neat! I'm looking at some air-gapped work coming up and having something like that would be quite handy
- fc417fc802 1y ago
  Isn't that easily accomplished by setting up a local deployment and then yanking the network cable? Anything that can quickly run a capable LLM is going to be a pretty beefy box though. More like LLM in an expensive space heater.
  - stirfish 1y ago
    I was thinking more like those Bitcoin mining usb Asics that used to be a thing, but instead of becoming ewaste, you can still use them to talk with chatgpt 2 or whatever. I'm picturing an llm appliance.
    - fc417fc802 1y ago
      There is no magic ASIC that can get around needing to do hundreds of watts worth of computations and having on the order of hundreds of gigabytes of very fast memory. Otherwise the major players would be doing that instead of (quite literally) investing in nuclear reactors to power their future data center expansions.
      - stirfish 1y ago
        That's a really good point. I wasn't thinking further than ollama on my MacBook, but I'm not deploying my laptop into production.
      - otabdeveloper4 1y ago
        24 gigabytes is more than enough to run a local LLM for a small household or business. This is "gaming PC" territory, not "space heater". I mean people already have PS5's and whatnot in their homes. The hundreds of gigabytes thing exists because the big cloud LLM providers went down the increasing parameter count path. That way is a dead end and we've reached negative returns already. Prompt engineering + finetunes is the future, but you need developer brains for that, not TFLOPs.
fc417fc802 1y ago
Because training costs weren't high enough already so lets add mask costs on top. More seriously, isn't that pretty much what all those AI hardware startups have already been doing for a while now?
- adgjlsfhk1 1y ago
  most of them are much more general purpose. they might be specializing somewhat on the architecture, but not on the weights
  - fc417fc802 1y ago
    Realistically specializing on the data flow is all you can do. Assuming a modern CPU contains on the order of 10 billion transistors that only amounts to 1.2 GiB storage before you account for any actual logic (ie 1 bit per transistor). DRAM hardware is quite different from that of processing elements and it takes quite a lot of DRAM chips to hold the weights of a single model.
nialse 1y ago
In five to ten years, when LLMs have stabilized, mapping them straight onto hardware will probably make sense. With today’s processes a hundred billion parameters might fit onto a single silicon wafer using ~1.5 bit precision implemented directly in logic gates. Using higher precision raises the gate count exponentially, so it makes more sense to keep the weights in memory and reuse shared compute blocks for the math for now. We need to get the ultra low precision LLMs working for the future though.

Next step - compile straight to verilog so I can buy some LLMs on aliexpress

by baq 1y ago

Next step - compile straight to verilog so I can buy some LLMs on aliexpress

https://riscv.org/blog/2021/02/hardware-description-language-chisel-diplomacy-deeper-dive/ https://riscv.org/blog/2021/02/hardware-description-language... That was one of the promising ideas before AI & GPUs come to the scene. As CPUs are stagnant, and naturally people want further optimize the middle layers software and hardware. But I suspect parallel computing in GPU style is going to dominate acclerated computing. General purpose CPUs are going to stay to become the little brain that orchestrates GPUs. Ideas of software direct to hardware transition might never be the mainstream.

baq 1y ago
I'm thinking more like pseudointellect over serial to attach a $3 esp32 to. Since it's basically tokens in, tokens out, let's just cut the unnecessary parts out. It's like querying the cloud models, except it's your silicon you personally soldered to the esp so nobody will break your home assistant with a system prompt update or a fine tuning run.
mycall 1y ago
> General purpose CPUs are going to stay to become the little brain that orchestrates GPUs Brings the deterministic compute to the indeterministic.
jerf 1y ago
"General purpose CPUs are going to stay to become the little brain that orchestrates GPUs." If that was going to happen, it would have happened. CPUs are genuinely good at what they do, and "what they do" is a lot of tasks that GPUs are actually terrible at. If all we had were GPUs in the world and someone invented a CPU, we'd hail them as a genius. A lot of people seem to think that GPUs are just "better", just ambiently better at everything, but that's lightyears from the truth. They are quite spectacularly terrible at a lot of very common tasks. There's many very good reasons that GPUs are still treated as accelerators for the CPUs and not vice versa.

anitil 1y ago

I mean.... LLM-in-a-box would actually be pretty neat! I'm looking at some air-gapped work coming up and having something like that would be quite handy

fc417fc802 1y ago
Isn't that easily accomplished by setting up a local deployment and then yanking the network cable? Anything that can quickly run a capable LLM is going to be a pretty beefy box though. More like LLM in an expensive space heater.
- stirfish 1y ago
  I was thinking more like those Bitcoin mining usb Asics that used to be a thing, but instead of becoming ewaste, you can still use them to talk with chatgpt 2 or whatever. I'm picturing an llm appliance.
  - fc417fc802 1y ago
    There is no magic ASIC that can get around needing to do hundreds of watts worth of computations and having on the order of hundreds of gigabytes of very fast memory. Otherwise the major players would be doing that instead of (quite literally) investing in nuclear reactors to power their future data center expansions.
    - stirfish 1y ago
      That's a really good point. I wasn't thinking further than ollama on my MacBook, but I'm not deploying my laptop into production.
    - otabdeveloper4 1y ago
      24 gigabytes is more than enough to run a local LLM for a small household or business. This is "gaming PC" territory, not "space heater". I mean people already have PS5's and whatnot in their homes. The hundreds of gigabytes thing exists because the big cloud LLM providers went down the increasing parameter count path. That way is a dead end and we've reached negative returns already. Prompt engineering + finetunes is the future, but you need developer brains for that, not TFLOPs.

fc417fc802 1y ago

Because training costs weren't high enough already so lets add mask costs on top. More seriously, isn't that pretty much what all those AI hardware startups have already been doing for a while now?

adgjlsfhk1 1y ago
most of them are much more general purpose. they might be specializing somewhat on the architecture, but not on the weights
- fc417fc802 1y ago
  Realistically specializing on the data flow is all you can do. Assuming a modern CPU contains on the order of 10 billion transistors that only amounts to 1.2 GiB storage before you account for any actual logic (ie 1 bit per transistor). DRAM hardware is quite different from that of processing elements and it takes quite a lot of DRAM chips to hold the weights of a single model.

nialse 1y ago

In five to ten years, when LLMs have stabilized, mapping them straight onto hardware will probably make sense. With today’s processes a hundred billion parameters might fit onto a single silicon wafer using ~1.5 bit precision implemented directly in logic gates. Using higher precision raises the gate count exponentially, so it makes more sense to keep the weights in memory and reuse shared compute blocks for the math for now. We need to get the ultra low precision LLMs working for the future though.