This is essentially the reason that differentiable methods alone don't work that well for NLP. Language is a discrete arbitrary symbol system, so small changes

This is essentially the reason that differentiable methods alone don't work that well for NLP. Language is a discrete arbitrary symbol system, so small changes may have large effects at many levels. ("Dog bites man" is not like "Man bites dog", discrete is not like discreet), etc.