You are misunderstanding me, I am not primarily talking about outliers. I am saying if the deltas are big in magnitude but concentrated on only a few different

You are misunderstanding me, I am not primarily talking about outliers. I am saying if the deltas are big in magnitude but concentrated on only a few different values, then encoding the deltas directly is wasteful. Whether you then use entropy coding, which also deals with outliers for free, or whether you use something more sophisticated than deltas, for example double deltas or subtracting the minimum delta, was not really my point. Maybe it was a bit confusing that I first brought this up in the context of outliers.