11 ms·
Whisper.api: Open-source, self-hosted speech-to-text with fast transcription
- tnhoang088 3y agoThank you. I love it
- innovatorved 3y agoWhisper API - Speech to Text Transcription This open source project provides a self-hostable API for speech to text transcription using a finetuned Whisper ASR model. The API allows you to easily convert audio files to text through HTTP requests. Ideal for adding speech recognition capabilities to your applications. Key features: - Uses a finetuned Whisper model for accurate speech recognition - Simple HTTP API for audio file transcription - User level access with API keys for managing usage - Self-hostable code for your own speech transcription service - Quantized model optimization for fast and efficient inference - Open source implementation for customization and transparency
- atajwala 3y agoAny plans to add phrase timestamps, channel separation and other equivalent ASR features to make this API more approachable?
- innovatorved 3y agoI am working on the timestamp feature. You will be able to see the option for timestamp soon.
- atajwala 3y agoAppreciate it! All the best.
- rrsp 3y agoAre you able to provide more information on the fine tuning? Any improvement in WER and what language it was fine tuned in and the size of the dataset used?
- brianjking 3y agoWhat was the fine tune? How does this compare to what is possible using https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper for example? Thanks!
- stavros 3y agoThis looks great, does recognition use the GPU? What's the speed you get on it?
- 1024core 3y agoDoes Android OS come with ASR?
- pizzafeelsright 3y agoThis is not fully self-hosted so much as middle-ware, no?
- innovatorved 3y agoIt is completely self-hosted, but it currently supports only the tiny and base models. You can soon expect support for large models. For any requests, you can create an issue.
- deleted 3y ago[deleted]
- distantsounds 3y agohow is this open source, or self-hosted, when it requires an API key and a login from a third party?
- innovatorved 3y agoNo, it is not a third-party. It is a just PostgreSQL database for logging everything. You can simply visit the /docs endpoint. It is just for authentication so that you can work with different users. One Again its completely self hosted
- freedomben 3y agoI'm guessing the installation instructions just need a bit of love. It could seem confusing to see a token request to https://innovatorved-whisper-api.hf.space/api/v1/users/get_token https://innovatorved-whisper-api.hf.space/api/v1/users/get_t...
- innovatorved 3y agohttps://innovatorved-whisper-api.hf.space/docs https://innovatorved-whisper-api.hf.space/docs Just visit the swagger create account and then gettoken to grab a token
- tommiegannert 3y agoI agree with freedomben that the reason people are confused is probably because your README says > curl -X 'POST' 'https://innovatorved-whisper-api.hf.space/api/v1/users/get_token https://innovatorved-whisper-api.hf.space/api/v1/users/get_t...' while later saying > curl -X 'POST' 'http://localhost:8000/api/v1/transcribe/?model=tiny.en.q5 http://localhost:8000/api/v1/transcribe/?model=tiny.en.q5' If you just change the first "curl" to also be localhost:8000, this will be cleared up.
- d-lisp 3y agoWhat's the point of logging everything ? I don't understand, isn't it possible to just deal with authentification locally ?
- innovatorved 3y agoMany of you are asking if the project is completely self-hosted and does not rely on any third-party services. Yes, it is completely self-hosted and does not rely on any third-party services. The user is for authentication, so no one can use the service without authentication.
- awwaiid 3y agoMaybe the auth token example is meant to also hit localhost?
- mkl 3y agoGetting an authentication token does rely on a third-party service, if the README instructions are correct. It requires sending an email address to that third party.
- Animats 3y agoHuh? "This project provides an API with user level access support to transcribe speech to text using a finetuned and processed Whisper ASR model." Why is this a service at all? Why not just a library? Or a subprocess?
- devmor 3y agoFrom what I can see, it runs in a docker container and uses an HTTP server to handle interaction.
- Dig1t 3y ago>Get Your token If it's completely self-hosted why do I need to get a token? Where does the actual model run?
- innovatorved 3y agogetToken is just an authentication layer for authenticating your request. If you want to self-host it, just clone the repo and please check the .env.example file.
- reilly3000 3y agoI think people are confused by the README showing localhost in some places and your instance url in others, causing them to assume you were centrally issuing tokens for using this project.
- edgarvaldes 3y agoRelated to whisper: whisperX is a god send. I can finally watch old or uncommon tv series with subtitles.
- jcims 3y agoOh dang, diarization? How well does it work?
- Libbum 3y agoIt is surprisingly good! Not clear if you can go without a manual step where you need to change the likes of SPEAKER_00 to Bob and SPEAKER_01 to Sarah, but I've not had it mess up on me at all transcribing 2 hour long conversations between 6 people.
- LeoPanthera 3y agoSo is "real time" translation a thing yet? I've long wanted to be able watch non-english television and have the audio translated into English subtitles. It's doable for pre-recorded things, but not for live. An iPhone app that could do this from the microphone would also be amazing. Google Translate and it's various competitors from Microsoft/Apple are nearly there, but they all stop listening inbetween sentences. Something that just listened constantly, printing translated text onto the screen, would be amazing.
- innovatorved 3y agoJust wait for a couple of weeks. I am working on speech-to-speech translation. Instead of subtitles, you can listen to it directly. I am also working on subtitles.
- LeoPanthera 3y agoBut I don't want that. I just want a live stream of translated text.
- xigency 3y agoYou can do this with PowerPoint actually. I bumped something once and Japanese subtitles popped up following what I was saying in my confusion.
- videogreg93 3y agoI've been using the Microsoft Speech api for an app and so far it's been surprisingly very good for realtime speech to text.
- deleted 3y ago[deleted]
- DarthNebo 3y agoFor long running stuff https://developer.apple.com/tutorials/app-dev-training/transcribing-speech-to-text https://developer.apple.com/tutorials/app-dev-training/trans... should be straightforward to translate as well using ported on-device BERT models
- ChrisArchitect 3y agoNot to be confused with Whisper – open source speech recognition by OpenAI https://news.ycombinator.com/item?id=34985848 https://news.ycombinator.com/item?id=34985848
- innovatorved 3y agohttps://openai.com/research/whisper https://openai.com/research/whisper
- 3abiton 3y agoI thought that was the same. I still don't see the difference.
- innovatorved 3y ago[flagged]
- devmor 3y agoIt is the same, this is a self-hosted solution.
- v7n 3y agoMany live streamers, and platforms, would love to have custom real-time transcription elements. I actually looked into this exact project of yours when I thought about creating such a thing. Even if it meant delaying the broadcast for a second while transcribing the accessibility value could be immense.
- nchudleigh 3y agoThis is awesome. For anyone confused about the project, it is using whisper.cpp, a C-based runner and translation of the open whisper model from OpenAI. It is built by the team behind GGML and llama.cpp. https://github.com/ggerganov https://github.com/ggerganov You can fork this code, run it on your own server, and hit the API. The server itself will use FFmpeg to convert the audio file into the required format and run the C translation of the whisper model against the file. By doing this you can separate yourself from the requirement of paying the fee that OpenAI charges for their Whisper service and fully own your translations. The models that the author has supplied here are rather small but should run decent on a CPU. If you want to go to larger model sizes you would likely need to change the compilation options and use a server with a GPU. Similar to this project, my product https://superwhisper.com https://superwhisper.com is using these whisper.cpp models to provide really good Dictation on macOS. Its runs really fast on the M series chips. Most of this message was dictated using superwhisper. Congrats to the author of this project. Seems like a useful implementation of the whisper.cpp project. I wonder if they would accept it upstream in the examples.
- mikeravkine 3y agoOne caveat here is that whisper.cpp does not offer any CUDA support at all, acceleration is only available for Apple Silicon. If you have Nvidia hardware the ctranslate2 based faster-whisper is very very fast: https://github.com/guillaumekln/faster-whisper https://github.com/guillaumekln/faster-whisper
- kkielhofner 3y agoctranslate2 is amazing, I don’t know why it doesn’t get more attention. We use it for our Willow Inference Server which has an API that can be used directly like OP project and supports all Whisper models, TTS, etc: https://github.com/toverainc/willow-inference-server https://github.com/toverainc/willow-inference-server The benchmarks are pretty incredible (largely thanks to ctranslate2).
- stavros 3y ago
- geekodour 3y agoNice! This will be very useful for me. Think I can run this locally can spin a basic telegram bot around it for personal use. One issue I faced with all the whisper based transcript generators is that there seems to be no good way to make editing/correcting the generated text with word level timestamp. I created a small web based tool[0] for that. By any chance if anyone is looking to edit transcripts generated using whisper, you'd probably find it useful. [0] https://github.com/geekodour/wscribe-editor https://github.com/geekodour/wscribe-editor
- pdntspa 3y agoSo Whisper is all the rage with speech-to-text, but what about text-to-speech?
- joy_void_joy 3y ago[dead]
- grzes 3y agoi dont understand the excitement here. it's just a HTTP wrapper for CLI command. you can build it easily on your own with any decent RAD framework