You're 100% right, and that bad feeling you have is a sign that you understand where both sides are coming from. This is why so many people say "not your keys,

You're 100% right, and that bad feeling you have is a sign that you understand where both sides are coming from. This is why so many people say "not your keys, not your coin". Giving up control of your keys is never the best option, but if that's what you choose to do, audits might help you make the least bad choice out of those available.