I think it's interesting that forcing models out of memorization don't always show a steep drop in ability. I've definitely had instances where 4 memorized a c

I think it's interesting that forcing models out of memorization don't always show a steep drop in ability. I've definitely had instances where 4 memorized a common puzzle and failed a subtly altered variant but then got the variant after changing variable names or otherwise making it look different from what it would have memorized.