That's effectively what eSpeak-ng is doing. For the CMU flite voices they represent the data as LPC (linear predictive coding) data with residual remainder (re

That's effectively what eSpeak-ng is doing. For the CMU flite voices they represent the data as LPC (linear predictive coding) data with residual remainder (residual excited LPC). The HTS models use simple neural networks to predict the waveforms -- IIRC, these are similar to RNNs. The MBROLA models use OLA (overlapped add) to overlap small waveform samples. They also use diphone samples taken from midpoint to midpoint in order to create better phoneme transitions.