I bet this can be resolved in the future. The problem is that some things appear very, very often in similar forms in the training data, this leads to the model

I bet this can be resolved in the future. The problem is that some things appear very, very often in similar forms in the training data, this leads to the model memorising the answer instead to generalising. This phenomenon can be seen for other tasks in other models, e.g. vision.