3 ms·
The best was the Ted Chiang article making numerous category errors and forest/trees mistakes in arguing that LLMs just store lossy copies of their training dat
by brookst 4y ago
The best was the Ted Chiang article making numerous category errors and forest/trees mistakes in arguing that LLMs just store lossy copies of their training data. It was well-written, plausible, and so very incorrect.
- dr_dshiv 4y agoI felt the same way. But I’d love to read a specific critique. Have you seen one?
- elefanten 4y agoHere’s one from a researcher (which also links to another), though I’m not qualified to assess it’s content in depth. https://twitter.com/raphaelmilliere/status/1624073150475431940 https://twitter.com/raphaelmilliere/status/16240731504754319...
- supriyo-biswas 4y agoNeural network based compression algorithms[1] are a thing, so I believe Ted Chiang's assessment is right. Memorization (albeit lossy) is also how the human brain works and develops reasoning[2]. [1] https://bellard.org/nncp/ https://bellard.org/nncp/ [2] https://www.pearlleff.com/in-praise-of-memorization https://www.pearlleff.com/in-praise-of-memorization
- brookst 4y agoThe fact that some neural network architectures can compress data does not mean that data compression is the only thing any neural network can do. It’s like saying that GPUs can render games, so GPT is a game because it uses GPU.