3 ms·

I don't think orthogonality is important here. Taylor series basis functions are not orthogonal, and nor are ReLUs.