4 ms·
The other thing that bugged me is the built in assumption that today's model have learned everything there is to learn from the Internet corpus. This is quite e
by XenophileJKO 2y ago
The other thing that bugged me is the built in assumption that today's model have learned everything there is to learn from the Internet corpus. This is quite easy to disprove. Both in factual retention, but also meta cognition on the context of the content.
- jebarker 2y agoYeah, exactly. A human can learning vastly more about, say, math from a much smaller quantity of text. I doubt we're anywhere close to exhausting the knowledge extraction potential from web data.
- bbor 2y agoWhich is exactly the point he's making, I believe; that simply collecting more data isn't the next step. That we've reached a local plateau in scaling ability based on corpus size. Which was assumed by pretty much everyone outside the DL elite the whole time, AFAIU
- esperent 2y agoRight, but there's nothing new in that statement. I've been hearing that we're running out of data for training AIs for two years at least.