The problem, as I understand it, is in order for a UI to feel responsive you often have to start reacting to the mousedown/mousemove before the mouseup occurs.

The problem, as I understand it, is in order for a UI to feel responsive you often have to start reacting to the mousedown/mousemove before the mouseup occurs. So you need to do some cleanup when that mouseup event happens. Only you never got a mouseup event because it happened over some other random element.