3 ms·
I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)
by thoughtpeddler 2mo ago
I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)
- walrus01 2mo agoThere isn't SchlongBench(TM) yet, it's a specific question I've been asking of differently sized models as a randomly chosen gauge of how much less commonly used knowledge is perma-baked into it. In this case a question about a specific yiddish origin slang term. Small/bad models don't know it's from middle high german or Yiddish and get its origin and meaning totally wrong (or it runs into model censorship related to slang related to the male anatomy). It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.
- thoughtpeddler 2mo agoOh ya, this is like the approach from the Incompressible Knowledge Probes [0] paper - smart! [0] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity [https://arxiv.org/abs/2604.24827 https://arxiv.org/abs/2604.24827]