3 ms·

I find this take super weak sauce and shallow. This recent $10,000 challenge is super super interesting imho. https://twitter.com/VictorTaelin/status/177810058

I find this take super weak sauce and shallow. This recent $10,000 challenge is super super interesting imho. https://twitter.com/VictorTaelin/status/1778100581837480178 https://twitter.com/VictorTaelin/status/1778100581837480178 State of the art models are doing more than “just” predicting the probability of the next symbol.

mjburgess 3y ago
You underestimate the properties of the sequential-conditional structure of human communication. Consider how a clever 6yo could fake being a physicist with access to a library of physics textbooks and a shredder. All the work is done for them. You'd need to be a physicist to spot them faking it. Of course, LLMs are in a much better position than having shredded physics textbooks -- they have shreddings of all books. So you actually have to try to expose this process, rather than just gullibly prompt using confirmation bias. It's trivial to show they work this way, both formally and practically. The issue is, practically, gullible people aren't trying.
- astrange 3y ago
  You can program algorithms into transformer networks, up to the limit of how many computations you get. https://srush.github.io/raspy/ https://srush.github.io/raspy/ Are you going to do computer reductionism too and say computers can't do arithmetic, they just run electricity through silicon?
  - mjburgess 3y ago
    Sure, electronic computers largely don't implement arithmetic. The value of electronic computers derives from the fact they can "differentially activate" electronic devices (graphics cards -> lcd screens, etc.). If they werent electronic, they'd be basically useless -- since electricity is essentially the only "power transmission" force which we can reliably use. I do think much of the metaphorical language we use around these devices completely mystifies them. These abstractions we use fail all the time, and reveal themselves as deceptions. We ought, often, get back to reality. The reality of NNs, implemented on electronic devices, is that they can accept digitally encoded electronic signals and output likewise, where the variations in output signals are models of conditional probability structures This, as a an analogy for learning, for animals, for people, etc. is madness; provably so. It's a convenient pseudoscience, for an era obsessed with the power of electricity to power devices of automation, rather than steam. Were we in the steam age, The Brain would be hydaulic. The gullible in the intelligensia are always obsessed by the baubles and trinkets of elite appeal.
- forrestthewoods 3y ago
  I don’t find your model either convincing or useful.

mjburgess 3y ago

You underestimate the properties of the sequential-conditional structure of human communication. Consider how a clever 6yo could fake being a physicist with access to a library of physics textbooks and a shredder. All the work is done for them. You'd need to be a physicist to spot them faking it. Of course, LLMs are in a much better position than having shredded physics textbooks -- they have shreddings of all books. So you actually have to try to expose this process, rather than just gullibly prompt using confirmation bias. It's trivial to show they work this way, both formally and practically. The issue is, practically, gullible people aren't trying.

astrange 3y ago
You can program algorithms into transformer networks, up to the limit of how many computations you get. https://srush.github.io/raspy/ https://srush.github.io/raspy/ Are you going to do computer reductionism too and say computers can't do arithmetic, they just run electricity through silicon?
- mjburgess 3y ago
  Sure, electronic computers largely don't implement arithmetic. The value of electronic computers derives from the fact they can "differentially activate" electronic devices (graphics cards -> lcd screens, etc.). If they werent electronic, they'd be basically useless -- since electricity is essentially the only "power transmission" force which we can reliably use. I do think much of the metaphorical language we use around these devices completely mystifies them. These abstractions we use fail all the time, and reveal themselves as deceptions. We ought, often, get back to reality. The reality of NNs, implemented on electronic devices, is that they can accept digitally encoded electronic signals and output likewise, where the variations in output signals are models of conditional probability structures This, as a an analogy for learning, for animals, for people, etc. is madness; provably so. It's a convenient pseudoscience, for an era obsessed with the power of electricity to power devices of automation, rather than steam. Were we in the steam age, The Brain would be hydaulic. The gullible in the intelligensia are always obsessed by the baubles and trinkets of elite appeal.
forrestthewoods 3y ago
I don’t find your model either convincing or useful.