I don't really get this viewpoint, you can load JQuery once then it's cached. Is a barely noticeable load time difference really worth the extra power and time

I don't really get this viewpoint, you can load JQuery once then it's cached. Is a barely noticeable load time difference really worth the extra power and time saving you get from JQuery?