I would go the exact opposite route for this exact reason: something like 24Kbit ADPCM or 16Kbit G728 can provide an absolutely decent quality with only a 5 ms

I would go the exact opposite route for this exact reason: something like 24Kbit ADPCM or 16Kbit G728 can provide an absolutely decent quality with only a 5 ms delay. This kind of bandwidth is now available in the vast majority of VoIP scenarios, an 100ms 3Kbps codec is only relevant for the extreme niches.