Now that I notice that behavior it is really weird. Anyway, if that is a problem/annoyance for you, then you could try switching to insert-mode by using append

Now that I notice that behavior it is really weird. Anyway, if that is a problem/annoyance for you, then you could try switching to insert-mode by using append ('a') instead, which will bump the cursor up a character and counter ESC bumping you down one.