3 ms·
I think these sort of efforts are mostly self-soothing at this point. It is almost certainly the case that the labs are at a minimum running inference over the
by supern0va 5mo ago
I think these sort of efforts are mostly self-soothing at this point. It is almost certainly the case that the labs are at a minimum running inference over the information they're pulling and ensuring that it's useful/suitable for pre-training. The models are at least good enough to know whether they're looking at utter nonsense.
- bauldursdev 5mo agoYa I feel like these AI companies have the ability to be somewhat selective about their training sets. They don't have to add everything. I guess the idea is the filters wouldn't catch it, but if the junk is indistinguishable from the real stuff, then won't the platforms just be ruined by a bunch of junk?
- hansmayer 5mo agoActually it was shown a couple of times already, some of it also by Anthropic's own research, that the LLMs are extremely easy to poison with small datasets.
- supern0va 5mo agoThat's correct, and their recent work on natural language autoencoders has given extremely compelling evidence of that...which is why their data collection practices for pre-training have almost certainly evolved, particularly since they've already scraped most of the internet.