andrew-w

17 ms·

31.
by andrew-w 2y ago
I agree, I think power users are happy to go to specific platforms, but APIs open up more use cases that can reach a broader audience. What kind of application would you use it for? We don't have specific plans at the moment, but are g
32.
by andrew-w 2y ago
No plans at the moment, but there seems to be a decent amount of interest here. Our main focus has been making the model as good as it can be, since there are still many failure modes. What kind of application would you use it for?
33.
by andrew-w 2y ago
To be honest, one of our main goals as a startup is to move quickly, and using readily accessible cloud providers for training makes that much more easy.
34.
by andrew-w 2y ago
The web app does allow you to upload any audio, but in order to use your voice, you would need to either record a sample for each video or clone your voice with a 3rd party TTS provider. We would like to make it easier to do all that within
35.
by andrew-w 2y ago
We use Modal for cloud compute and autoscaling. The model is our own.
36.
by andrew-w 2y ago
It's something we're thinking about. Our main focus right now is to make the model as good as it can be. There are still many edge cases and failure modes.
37.
by andrew-w 2y ago
I see a lot of potential in animating memes and making them more fun to share with friends. Hopefully, we can do better on orcs soon!
38.
by andrew-w 2y ago
I know what will be in my nightmares tonight...
39.
by andrew-w 2y ago
No, it's just a hallucination of the model. The audio in your clip is synthetic and doesn't reflect any video in the real world. Hopefully we can animate your bear cartoon one day!
40.
by andrew-w 2y ago
Cartoons are definitely a limitation of the current model.
41.
by andrew-w 2y ago
This is why we do what we do lol
42.
by andrew-w 2y ago
Totally agree. We tweaked some settings after other commenters asked about speed, and got it up to 23fps generation (at the cost of lower resolution). Here is the example: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com&#
43.
by andrew-w 2y ago
All for suggestions! We've tried white noise as well, but it only works on plain talking samples (not music, for example). My guess is that the most robust solution will come from updating how it's trained.
44.
by andrew-w 2y ago
I don't think we've seen laughing quite that expressive before. Good find!
45.
by andrew-w 2y ago
Thank you! It has learned a surprising amount of world knowledge.
46.
by andrew-w 2y ago
Very creative use cases! We are trying to better understand the model behavior at the very end of the video. We currently extend the audio a bit to mitigate other end-of-video artifacts ( https://news.ycombinator.com/item?id=
47.
by andrew-w 2y ago
This is exactly what we do today! It seems to work better the more you extend it, but extending it too much introduces other side effects (e.g. the avatar will start to open its mouth, as if it were preparing to talk).
48.
by andrew-w 2y ago
This is a good idea. We have discussed incorporating an additional "identity" signal to the conditioning, but simply enforcing consistency with the original character as a post-processing step would be a lot easier to try. Are the
49.
by andrew-w 2y ago
Haha, wouldn't we all love that? In the long run, we will definitely need to move beyond talking heads, and have tools that can generate full actors that are just as expressive. We are optimistic that the approach used in our V2 model
50.
by andrew-w 2y ago
The model configuration that is publicly available is about 5x slower than real-time (~6fps). At lower resolution and with a less conservative number of diffusion steps, we are able to generate the video at 20-23 fps, which is just about re
51.
by andrew-w 2y ago
Thanks for the feedback! We used to have more voices, but didn't love the experience, since users had no way of knowing what each voice sounded like without creating a clip themselves. Probably having pre-generated samples for each one
52.
by andrew-w 2y ago
Thanks for the feedback. The current model was trained at ~320x320 resolution. We believe going higher will result in better videos with finer detail, which we plan to do soon.
53.
by andrew-w 2y ago
Yeah, it's about 5x slower than realtime with the current configuration. The good news is that diffusion models and transformers are constantly benefitting from new acceleration techniques. This was a big reason we wanted to take a bet
54.
by andrew-w 2y ago
It is possible to fine-tune the model with videos of a specific actor, but not images. You need videos to train the model. We actually did this in early overfitting experiments (to confirm our code worked!), and it worked surprisingly well.
55.
by andrew-w 2y ago
This can happen with non-humanoid images. The model doesn't know how to animate them.