105 ms·
Even so, they can drop from their training set a few dozen pages from the internet and it won't be a huge loss among the trillions of documents. The LLM will s
by PostOnce 2y ago
Even so, they can drop from their training set a few dozen pages from the internet and it won't be a huge loss among the trillions of documents.
The LLM will still know how LLMs work without having trained on the handful of documents containing that specific canary string, because other documents will mention the concept of a canary string without that exact GUID.
Better to do that and be on the safe side and look honest than have people believe your company not really competitive.
Anything else is a risk for no gain in an industry theoretically worth trillions.
- jhugo 2y agoSo just to understand this correctly, you're suggesting that they have some people/processes dedicated to keeping track of any canary string published by anyone (or some defined subset of "anyone"), and updating their ingest to ignore any documents that contain those strings?
- PostOnce 2y agoI explained above that they don't have to care, but if they're going to want to be included on major industry benchmarks like those created by Google, they should probably go to the effort of ignoring the half dozen or so notable benchmarks with canary GUIDs. Blacklisting a GUID is not difficult, not even at web-scale.