3 ms·
can this achieve real-time performance or how far are we from a real-time model?
by billconan 2y ago
can this achieve real-time performance or how far are we from a real-time model?
- bpanahij 2y agoCheckout Tavus.io for realtime. They have a great API for realtime conversational replicas. You can configure the CVI to do just about anything you want to do with a realtime streaming replica.
- andrew-w 2y agoThe model configuration that is publicly available is about 5x slower than real-time (~6fps). At lower resolution and with a less conservative number of diffusion steps, we are able to generate the video at 20-23 fps, which is just about real-time. Here is an example: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/fast_example.mp4 https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/fast... We use rectified flow for denoising, which is a (relatively) recent advancement in diffusion models that allow them to run a lot faster. We also use a 3D VAE that compresses the video along both spatial and temporal dimensions. Temporal compression also improves speed.