4 ms·
Claude 3.5 Sonnet Reproduces BIG-Bench Canary String
- Havoc 2y agoDoesn't feel conclusive. A markdown file in a somewhat popular repo is likely elsewhere on the internet too
- gpm 2y agoIt seems to me that the problem isn't that an LLM memorized the benchmark, but that the benchmark is relying on a piece of published information to remain secret instead of simply randomly generating the secret data as we do for every other important secret.
- PollardsRho 2y agoThe point isn't that it's a secret, it's that it's a string that shouldn't appear in any context besides the benchmark. It's an easy way to identify contamination in datasets. People often don't explicitly repeat them because that leads to false positives.
- gpm 2y agoAh, you're right, I should have clicked through a few more links. I was assuming it was a needle that they were asking if the model could recall after hiding it in a bunch of text, not a canary for the test data being included in the training set at all. Still, I think it's less than clear that the problem here isn't just the idea that you can publish benchmark data on the open web at all without making the benchmark outdated. Expecting secret (to the model) benchmark data to be reliably detected and filtered from training data seems... unlikely. Especially in this day and age where lots of people dislike AI enough that they would be interested in actively sabotaging efforts...
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- tiltowait 2y agoHow does a canary string provide any certainty one way or the other? It's easy to consider a false positive scenario: a blog post talking about canary strings, what they do, and then saying "here's the BIG-Bench canary string". A false-negative is similarly easy to consider: an unscrupulous employee could scrub the string itself from the training set but keep the rest of the benchmark data. I expect people more clever than me have spent more than the three minutes I did thinking on it, so I'm genuinely curious as to how the two scenarios above are protected against. From my limited understanding, though, it feels like there's just an honor system involved.
- a_bonobo 2y agoWas it really trained on BIG-Bench, or was it trained on an article or forum post about BIG-Bench that mentioned or reproduced the canary string?
- PostOnce 2y agoEither way, shouldn't a company the size of Anthropic be scrubbing benchmarks from their training set regardless of how the benchmark data got into the training set?
- jnhl 2y agoUntraining models is an open research question and not par for the course like you're implying
- PostOnce 2y agoNo, not "untraining" after the fact, but scrubbing the benchmark data from the dataset prior to training. That's absolutely trivial. Take the set of benchmarks you care about, and then as you build a training dataset (by scraping, or whatever), you scrub each new item for benchmark questions (or just discard that entire item/webpage/whatever for being contaminated). Otherwise you're negligently and willingly inflating your benchmark performance and defrauding new investors / users / customers, I think.
- dialup_sounds 2y agoThe string itself is a fact, not a benchmark.
- PostOnce 2y agoThe string is to indicate that it was found alongside the benchmark data, that's the whole point. If that string shows up, then it's extremely likely the test / benchmark data was there too, indicating it's contaminated. In other words, if that string is present, the benchmark results for that model are a lie.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- cpldcpu 2y agoSearching for the string on google yields hundreds of hits. Likely that it appears in recent webscrape-data.
- andrewflnr 2y agoThe model knows the original context too, though. So it doesn't make a lot of practical difference if it found it through a secondary source.
- falcor84 2y agoBut the discussion has mostly centered around whether Anthropic have properly scrubbed benchmark data out of their training corpus before training. So it seems to me that even if they did it properly (maybe especially so), the presence of the canary string in Claude's output is likely if it saw it in a secondary source (e.g. a blog post), which should then be fair game, no?
- tracerbulletx 2y agoIt makes literally all the difference that matters?