jpcl

7 ms·

1.
by jpcl 7mo ago
What this means is that it does not support things like acting instructions or creating a voice from a text description. If you prompt it with a matching text+voice sample it will be able to generate more speech based on more text, just lik
2.
by jpcl 3y ago
Hi, I used the [WhisperSpeech]( https://github.com/collabora/WhisperSpeech ) model for the TTS part after I did some serious torch.compile optimizations to bring the latency down. The Whisper speech recognition and the L
3.
by jpcl 3y ago
Not really, the Polish effort is run by a non-profit and hired professional voice actors.
4.
by jpcl 3y ago
Yeah, thanks. I'd love to try to clarify this (I'll put this into our documentation as well ASAP) for anyone that may be reading this in the future: Our model is not a derivative of Whisper but we do use the transcripts (and encod
5.
by jpcl 3y ago
Did exactly that, thanks for spotting that. :) https://github.com/collabora/WhisperSpeech/commit/398b889060...
6.
by jpcl 3y ago
Good point, thanks. And I was thinking it will show that the model can really synthesize very varied samples... ;)
7.
by jpcl 3y ago
That's an interesting thought. The semantic tokens we get from Whisper serve a similar purpose – you can convert existing speech to different voices, I did not try with accents yet. There is still a lot to explore in this space – we ce
8.
by jpcl 3y ago
That's true but you make it sound like it's totally obvious where the line of fair use should be drawn for AI training. Until courts or lawmakers make it clearer I personally believe non-generative models (Whisper, ResNet, DINOv2)
9.
by jpcl 3y ago
We looked at this at one point and it seems whisper.cpp/llama.cpp have all the bits needed to make it work. I'd love to help if someone wanted to give it a shot.
10.
by jpcl 3y ago
https://wolnelektury.pl/katalog/audiobooki/ is the Polish audiobook collection. The English audiobooks are public domain recordings from LibriVox (via the LibriLight dataset).
11.
by jpcl 3y ago
Thanks, I'll check it out. I don't know any Chinese so I'll probably reach out to you for some help :)
12.
by jpcl 3y ago
Yeah, we'd love to help you when you decide to give it a try so feel free to reach out. We also have quite a few people working on VR at Collabora.
13.
by jpcl 3y ago
We are working hard to uphold all the licensing rules but nobody can absolve you from all legal risks. There may be a court ruling/new law that any training needs a special permission from the original author and then even a CC-BY lice
14.
by jpcl 3y ago
For Polish I have around 700hr. I suspect that we will need less hours if we add more languages since they do overlap to some extent. Fixed transcripts would be nice although we need to align them with the audio really precisely (we cut the
15.
by jpcl 3y ago
Yes, you download the weights once from Huggingface and you can do whatever you want with it. :) We have no cloud APIs or usage tracking of any kind.
16.
by jpcl 3y ago
Both models are using around 3GB right now (converted into FP16 for speed). But I checked that the (slower) FP32 version uses 2.3GB so we are probably doing something suboptimal here. We support CUDA right now although it should not be too
17.
by jpcl 3y ago
There is a plot of language performance on their repo: https://github.com/openai/whisper I am not aware of a multi-lingual leaderboard for speech recognition models.
18.
by jpcl 3y ago
Training the T2S model from scratch takes around 8h on 96 A100 GPUs. Training the `tiny` S2A model is around 3x faster (training HQ `small` variant is comparable to T2S). I think you would get good results with fine-tuning but unfortunately
19.
by jpcl 3y ago
Yeah, Whisper is not clear-cut but since it is not a generative model I think their data usage is a lot more likely to be considered fair-use. And the part of that which we use for WhisperSpeech is just the phonetic representation so our mo
20.
by jpcl 3y ago
Hi, WhisperSpeech dev here. Thanks for all the nice comments, I was working really hard on this model for quite a few months now but there are still a lot of ways we can make it better. Thanks to generosity of Collabora this is a real open-
21.
by jpcl 3y ago
Hi, thanks a lot for the tip, I'll update the README samples ASAP. :) I was busy working on inference performance in the last few weeks and totally did not expect to land on Hackernews today. Only noticed it an hour ago because my GitH
22.
by jpcl 3y ago
Both Polish and English samples are actually synthesized with a voice trained on the WolneLektury audiobooks. They are the highest quality open source (CC BY-SA) audiobooks I could find. By using the Whisper-derived phonetic representation
23.
by jpcl 3y ago
Yeah, the Mimic is a lot less resource intensive. We are working to improve WhisperSpeech in this regard but it's probably always going to require more compute (but in return you'll get higher quality). That said if you have a mod
24.
by jpcl 3y ago
Yup, we are using Whisper to transcribe automatically so we can train the model on just speech recordings, without human transcripts. This works for any language that is well supported by the OpenAI Whisper model.
25.
by jpcl 3y ago
Thanks a lot. :) We are constantly working on these models and we push new versions every two months or so. It should get even better soon. :)
26.
by jpcl 3y ago
It should run with PyTorch but the performance might not be great. As a long-time Mac user myself I would love if someone would send a PR to port it to MLX :)
27.
by jpcl 3y ago
Yes, on a consumer 4090 card it's 12x faster than real-time. We'll benchmark some older cards as well for comparison. I think it should work pretty good with the Apple's MLX framework as well if anyone would be willing to con
28.
by jpcl 3y ago
We support voice cloning so you can mimic the sound of any real voice (or try to create random ones). The prosody/emotions are more difficult to control right now but we are looking into this. To check how this works in practice you ca
29.
by jpcl 3y ago
Hi, WhisperSpeech dev here, we only support Polish and English at the moment but we just finished doing some inference optimizations and are looking to add more languages. What we seem to need is high-quality speech recordings in any langua
30.
by jpcl 5y ago
It does look much nicer than `git add -p` and manually editing patches in an editor which is what I am using right now. The question is whether my muscle memory will agree if I switch ;)

More ›