3 ms·

I think that endgame for generative language models is models embedded directly into chips. Computers that run english language instead of machine code and for

I think that endgame for generative language models is models embedded directly into chips. Computers that run english language instead of machine code and for which CPU, GPU and what is currently known as PC is more like a peripheral IO device.

dragonwriter 3y ago
> I think that endgame for generative language models is models embedded directly into chips. Computers that run english language instead of machine code and for which CPU, GPU and what is currently known as PC is more like a peripheral IO device. That may be the endgame, but I think if it is there is a long time before attempts to jump to it aren't going to fail like every high-level-system-in-hardware for other than very niche applications, because general purpose (comparatively, even if specialized for running AI models) hardware will be good enough that the value of being able to upgrade the models it is running will outweigh any marginal temporary edge that current-models-in-hardware have.
intrasight 3y ago
Such a model could not be updated nor learn. Even very simple biological organisms can learn. The end-game is growing brains ;) (only partially kidding)
- dragonwriter 3y ago
  > Such a model could not be updated nor learn. It might have some limited ability for updates if the hardware had the model code but the weights were in memory that was updatable. It might do in-context learning even without that.
  - intrasight 3y ago
    > weights in memory That's basically how our neurons work. New neuron growth and connection isn't much of a factor in learning. Rather it's the synaptic restructuring (equivalent to AI model weights) that change relatively quickly. So we need to figure how how to "grow" mechanical brains. I envision this being done with a new generation of FPGAs tailored to this task.
- layer8 3y ago
  That might not be a serious issue if we keep updating our personal hardware at the present frequency. So you’ll be able to decide whether to update to the new iPhone with English 4.7, or rather stay with English 3.5 for another year.

dragonwriter 3y ago

> I think that endgame for generative language models is models embedded directly into chips. Computers that run english language instead of machine code and for which CPU, GPU and what is currently known as PC is more like a peripheral IO device. That may be the endgame, but I think if it is there is a long time before attempts to jump to it aren't going to fail like every high-level-system-in-hardware for other than very niche applications, because general purpose (comparatively, even if specialized for running AI models) hardware will be good enough that the value of being able to upgrade the models it is running will outweigh any marginal temporary edge that current-models-in-hardware have.

intrasight 3y ago

Such a model could not be updated nor learn. Even very simple biological organisms can learn. The end-game is growing brains ;) (only partially kidding)

dragonwriter 3y ago
> Such a model could not be updated nor learn. It might have some limited ability for updates if the hardware had the model code but the weights were in memory that was updatable. It might do in-context learning even without that.
- intrasight 3y ago
  > weights in memory That's basically how our neurons work. New neuron growth and connection isn't much of a factor in learning. Rather it's the synaptic restructuring (equivalent to AI model weights) that change relatively quickly. So we need to figure how how to "grow" mechanical brains. I envision this being done with a new generation of FPGAs tailored to this task.
layer8 3y ago
That might not be a serious issue if we keep updating our personal hardware at the present frequency. So you’ll be able to decide whether to update to the new iPhone with English 4.7, or rather stay with English 3.5 for another year.