One example is the three killers problem There are three killers in a room, someone enters the room and kills one of them, how many killers are now in the ro

One example is the three killers problem There are three killers in a room, someone enters the room and kills one of them, how many killers are now in the room? Apart from the basic reasoning errors of small models (where they come up with all sorts of numbers), larger models that fail, do so because they fail to take into account that killing makes a killer. Some models succeed at this but it is unclear if this is because they are familiar with this specific problem. The model has to make the distinction between that and There are three fish in a room, someone enters the room and kills one of them, how many fish are now in the room? or There are three fish in a room, someone enters the room and kills one of them, how many killers are now in the room? In these examples the word kills is pushed in one direction or another, but when it serves double duty in the original problem it can notice one premise and miss the other. It's interesting task in frustration to try and get a model that gets it wrong to COT out of that particular error.