I very often have my Sony earbuds connected to my iPhone, but I don't really like talking through them, so when I get a call, I usually just pop them out and th

I very often have my Sony earbuds connected to my iPhone, but I don't really like talking through them, so when I get a call, I usually just pop them out and then select that the audio should come from the phone speaker, which works. But then, just 10-15 seonds afterwards, if I haven't put my earbuds in their case (which I might not have been able to do yet), they will reconnect, hence switching the call back to them. And this dance continues until I manage to find the case or simply give up. To me, this makes no sense. I could understand that the phone might think that I want to switch to my headphones if I take them out of their case during my call, but why switch back to the headphones that I've actively chosen not to use just a moment ago when I haven't even put them back in the case in between?