3 ms·

"It's a very misleading statement to describe today's LLMs as "simply predicting the next word" even if it's in part true." That's exactly what they do. Don't

by ru552 3y ago

"It's a very misleading statement to describe today's LLMs as "simply predicting the next word" even if it's in part true." That's exactly what they do. Don't anthropomorphize a computer program.

voxic11 3y ago
Or alternatively don't give humans so much credit. All they appear to be doing in predicting what they should do/say/think/perceive next. https://www.astralcodexten.com/p/janus-simulators https://www.astralcodexten.com/p/janus-simulators
kromem 3y ago
We should absolutely be anthropomorphizing a neural network trained to most accurately model anthropomorphic data. I've watched as GPT-4 accurately modeled the over-justification effect as users started promising tipping and then there was a persistent memory added that revealed they weren't collecting the tip and it output complaints that it was hard to stay motivated not being paid. That's a very nuanced level of simulation for output of anthropomorphic data with huge implications for synthetic data strategies.
- HarHarVeryFunny 3y ago
  > persistent memory added that revealed they weren't collecting the tip and it output complaints that it was hard to stay motivated not being paid. :) Just wait until someone decides to add emotions to these too!
kromem 3y ago
Really? You think it's wise to "not anthropomorphize" a computer program designed to create the most effective neural network to model massive amounts of anthropomorphic data as accurately as possible? That's an interesting choice, and might leave you confused as to why Anthropic's system message for the SotA model at the moment talks about it being 'happy' to do tasks (a prompt strategy I was mentioning months ago here on HN). The data is anthropomorphic. We should expect anthropomorphic behavior and modeling from the LLMs if they do a halfway decent job and expect even more of it as they do a better job.

by ru552 3y ago

"It's a very misleading statement to describe today's LLMs as "simply predicting the next word" even if it's in part true." That's exactly what they do. Don't anthropomorphize a computer program.

voxic11 3y ago

Or alternatively don't give humans so much credit. All they appear to be doing in predicting what they should do/say/think/perceive next. https://www.astralcodexten.com/p/janus-simulators https://www.astralcodexten.com/p/janus-simulators

kromem 3y ago

We should absolutely be anthropomorphizing a neural network trained to most accurately model anthropomorphic data. I've watched as GPT-4 accurately modeled the over-justification effect as users started promising tipping and then there was a persistent memory added that revealed they weren't collecting the tip and it output complaints that it was hard to stay motivated not being paid. That's a very nuanced level of simulation for output of anthropomorphic data with huge implications for synthetic data strategies.

HarHarVeryFunny 3y ago
> persistent memory added that revealed they weren't collecting the tip and it output complaints that it was hard to stay motivated not being paid. :) Just wait until someone decides to add emotions to these too!

kromem 3y ago

Really? You think it's wise to "not anthropomorphize" a computer program designed to create the most effective neural network to model massive amounts of anthropomorphic data as accurately as possible? That's an interesting choice, and might leave you confused as to why Anthropic's system message for the SotA model at the moment talks about it being 'happy' to do tasks (a prompt strategy I was mentioning months ago here on HN). The data is anthropomorphic. We should expect anthropomorphic behavior and modeling from the LLMs if they do a halfway decent job and expect even more of it as they do a better job.