Another comment told you how to change this behavior, but I heard someone rationalize the default behavior as taking out one earbud can signal you're trying to

Another comment told you how to change this behavior, but I heard someone rationalize the default behavior as taking out one earbud can signal you're trying to talk to someone so it disables noise cancelation so you can hear them with both ears.