"You left me waiting for days." What have I just said? If the context is that it's been a handful of minutes, we don't say my usage is wrong; we definitely do

"You left me waiting for days." What have I just said? If the context is that it's been a handful of minutes, we don't say my usage is wrong; we definitely don't say that "sometimes days means minutes" and fret about how anyone will communicated time. We say that sometimes people exaggerate. You can still object, if you wish, on stylistic grounds. You can object that you'd prefer we keep "literally" apart from some standard uses of words lest we allow inappropriate ambiguity. But none of that means anyone is using "literally" to mean "figuratively".