As long as the system has a probability to output any arbitrary series of tokens, there will be contexts where an otherwise improbably sequence of tokens is out

As long as the system has a probability to output any arbitrary series of tokens, there will be contexts where an otherwise improbably sequence of tokens is output. Training can push around the weights for undesirable outputs, but it can't push those weights to zero.