We've considered video rendering using the H264 encoders on the GPU, another approach might be to create a format that has all info needed for interactions and

We've considered video rendering using the H264 encoders on the GPU, another approach might be to create a format that has all info needed for interactions and that could be deciphered on the frontend (i.e in WebGL). So it's something we're definitely looking at. We already do provide some support for hover interactions in our charting library on top of Vega, but it would be nicer to do this in Vega. Right now customers still get a lot of value from server-side rendering simply bc we can render visualizations of billions of records interactively and in real time, which is difficult or impossible to achieve in other platforms without some sort of pre-computation.