4 ms·

Maybe. But I have yet to see (for example) an LLM write a novel. Or build out a web framework. GitHub Copilot + Grammarly are the only two useful apps it has

Maybe. But I have yet to see (for example) an LLM write a novel. Or build out a web framework. GitHub Copilot + Grammarly are the only two useful apps it has, at least so far. Feel free to convince me otherwise!

FeepingCreature 2y ago
It seems like you're judging LLMs by human standards. If it could do those things, it would almost definitionally be AGI already. LLMs are "just" pattern-based token prediction engines with large scale. The interesting thing is that it looks like we should be able to build a general intelligence from that - like we're getting at least some facets of intelligence from that alone, that we previously had no idea how to produce - not that we're already there! Treat it as a text-based addon to your brain, IMO. A human brain has many components necessary for general intelligence. Some of those components can now be augmented by a giant neural net; others still benefit from human execution. All the LLM programming work I do is based on a cooperative dialog, where I fill in the gaps where the LLM doesn't have an appropriate pattern. Which are many. (But getting less.) At any rate, the idea that LLMs regurgitate their training data is completely unrelated to this and afaict one of those "true but trivial, or important but false" ones. (True in the sense that humans also represent a reflection of their inputs; false in the sense that what's going on is definitely more than a collage of samples.)
throwthrowuknow 2y ago
I would bet you X donuts that if you give me a novel and I’m allowed access to a corpus of all novels and other writing that running a similarity search on the embedded content of each would turn up a shocking number of matches. I’d bet dollars instead of donuts on the equivalent experiment with a web framework.

4 ms·

Maybe. But I have yet to see (for example) an LLM write a novel. Or build out a web framework. GitHub Copilot + Grammarly are the only two useful apps it has

by djaouen 2y ago

FeepingCreature 2y ago
It seems like you're judging LLMs by human standards. If it could do those things, it would almost definitionally be AGI already. LLMs are "just" pattern-based token prediction engines with large scale. The interesting thing is that it looks like we should be able to build a general intelligence from that - like we're getting at least some facets of intelligence from that alone, that we previously had no idea how to produce - not that we're already there! Treat it as a text-based addon to your brain, IMO. A human brain has many components necessary for general intelligence. Some of those components can now be augmented by a giant neural net; others still benefit from human execution. All the LLM programming work I do is based on a cooperative dialog, where I fill in the gaps where the LLM doesn't have an appropriate pattern. Which are many. (But getting less.) At any rate, the idea that LLMs regurgitate their training data is completely unrelated to this and afaict one of those "true but trivial, or important but false" ones. (True in the sense that humans also represent a reflection of their inputs; false in the sense that what's going on is definitely more than a collage of samples.)
throwthrowuknow 2y ago
I would bet you X donuts that if you give me a novel and I’m allowed access to a corpus of all novels and other writing that running a similarity search on the embedded content of each would turn up a shocking number of matches. I’d bet dollars instead of donuts on the equivalent experiment with a web framework.