12 ms·
FOSDEM 2023 Transcribed by Whisper
- tuukkah 4y agoWow! Is there an easy way to combine these subtitles with the video in the browser?
- usr1106 4y agoNot a format I would recognize, but I am not an expert in the area. Two lines of awk would convert it to srt I guess.
- tuukkah 4y agoIt's even closer to WebVTT, just had to add the line "WEBVTT" to the top and replace all square brackets with line breaks: curl https://jonatron.github.io/fosdem2023whisper/files/celebrating_25_years_of_open_source.webm.txt | ( echo "WEBVTT"; sed -e 's/^\[\|\] \?/\n/g' ) >celebrating_25_years_of_open_source.webm.vtt Plays in the browser with HTML like this: <video controls src="https://video.fosdem.org/2023/Janson/celebrating_25_years_of_open_source.webm" > <track default srclang="en" src="celebrating_25_years_of_open_source.webm.vtt" > </video>
- karencarits 4y agoA sidenote, but adding subtitles to pure audio in HTML is surprisingly difficult!
- tuukkah 4y agoDoes it not work to put a track element inside an audio element?
- karencarits 4y agoYou can try to fake it as a video, but in my experience, it is difficult to properly format the element so that the subtitles or captions actually shows https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio https://developer.mozilla.org/en-US/docs/Web/HTML/Element/au... > The <audio> element doesn't directly support WebVTT. You will have to find a library or framework that provides the capability for you, or write the code to display captions yourself. One option is to play your audio using a <video> element, which does support WebVTT.
- cratermoon 4y agoIs my version of sed dumb, or is 's/^\[\|\] \?/\n/g' effectively a no-op? Or did HN garble something?
- tuukkah 4y agoI used GNU sed on Ubuntu. The only difference I notice is that you have one space where I had two, but that difference shouldn't matter actually. Perhaps your sed has a different opinion on which of the backslashes should be there. You can access the resulting WebVTT files here: https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt/files https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt...
- cratermoon 4y agoOnce again macOS non-standard utilities bite me. -E Interpret regular expressions as extended (modern) regular expressions rather than basic regular expressions (BRE's) -r Same as -E for compatibility with GNU sed. Thanks for providing the complete set of vtt files. I found some interesting transcription errors in the talk on Elliptic curves in FOSS. For example "jealous two curves" instead of "genus two curves" and "ethyl curbs" instead of "elliptical curves"
- jonatron 4y agoIt looks pretty easy, but I'm not going to do it, if someone else wants to.
- AstixAndBelix 4y agoDoesn't Mozilla have their own speech recognition library? I think it would have been more interesting to see the results, since we're talking about FOSDEM. Also, since OpenAI was already used, it could have been cool to provide a summary of the transcript!
- adgjlsfhk1 4y agowhisper is mit licensed.
- homarp 4y agoand run "offline" and locally, so the content transcribed is not sent anywhere. There is also a CPU only version discussed a few times: https://news.ycombinator.com/item?id=33877893 https://news.ycombinator.com/item?id=33877893
- PhilippGille 4y ago> Also, since OpenAI was already used, it could have been cool to provide a summary of the transcript! I liked this idea, so I opened a PR for exactly that: https://github.com/jonatron/fosdem2023whisper/pull/2 https://github.com/jonatron/fosdem2023whisper/pull/2
- nindalf 4y agoThis is pretty good! I spot checked one video (https://jonatron.github.io/fosdem2023whisper/files/rust_coreutils.webm.txt https://jonatron.github.io/fosdem2023whisper/files/rust_core...) and it's done a decent job. It mostly makes mistakes around a few proper nouns. - coreutils => corridors/coroutils/cori-teals/curricules/corretails - Rust => rest/Resi - glibc => GLC - Gentoo => Gen2 - Fuchsia => Fushia - SELinux => AC Linux Other proper nouns it transcribed well - LLVM, Mozilla, Debian, Ubuntu, clang. These are much better known I suppose. It finds the speaker's French accent a bit difficult, because I see a few mistakes like "find => fine", "imposter => imposterous", "command => comment". Overall pretty good. The only thing I'd like to improve readability is a pass with an LLM to break it into paragraphs but that would be way more compute and no longer local (like Whisper).
- margorczynski 4y agoMaybe fine tuning on "tech lingo" would improve that. Give that to your average person to transcribe and the results most probably would be worse than that.
- bonzini 4y agoI checked my own talk and it's not perfect but it's more than understandable, despite my having an accent, speaking fast and not having rehearsed very much. Together with the slides it would be clear what was going on.
- simonw 4y agoI wonder how hard it would be to automatically add slides by scanning the talk video for points at which the on-screen slides change and mixing a frame from the video into the transcript at the relevant point.
- angrais 4y ago1. Automatically detect when (timestamp) change in slide, e.g., large % of colour change in the image of the video (I assume video is slide-deck only else we can crop to the area that is); 2. Use the timestamps provided by whisper to group and align with (1).
- CraftingLinks 4y agoMissed opportunity to feed the transcripts to LLM and prompt it to clean it up, (remove stop words, stuff like that)? -> instant blog post to go with the talk.
- delusional 4y agoI tried to get chat.openai.com to generate something interesting for the transcript linked to in another comment and it failed rather spectacularly. The transcript contains error for Rust (which becomes rest) and coreutils (which becomes corridors) making the output of the LLM useless. The LLM also seems to completely misinterpret the author when he speaks about working for Mozilla, omitting the part where he explicitly mentions that he's only saying this so that you wont associate this work with Mozilla. It also repeatedly gets confused about what the author is talking about, likely due to a small context size. It would be a very low quality blog-spam level blog post.
- arlort 4y agoI'm curious if you could use the transcript + something like the title/subtitle of the talk with chatgpt to fix the spelling mistakes
- delusional 4y agoInterestingly, If i give it the filename title "Rust Coreutils" it spots "the corridor -> coreutils", but if i provide the proper title of the talk "Reimplementing the Coreutils in a modern language (Rust) Doing old things with modern tools" it doesn't find any mistakes. Once again lending creadence to the observation that it doesn't actually "understand" since the proper title is quite clearly not markedly different (in relation to the transcription mistakes) from the filename title.
- CraftingLinks 4y agoI had something more basic in mind: Just clean up syntax without much, if any, semantic re-interpretations.
- carbocation 4y agoCan you describe your setup / how you piped the audio from YouTube to Whisper in order to generate these transcripts?
- jonatron 4y agoUsed a small python script to download the webm files from https://fosdem.org/2023/schedule/events/ https://fosdem.org/2023/schedule/events/ Used a small bash script to run `whisper $f --output_format json --language en | tee $f.txt` over every file. Ran a small python script to make the HTML. All totally trivial, it just required some GPU-hours.
- carbocation 4y agoThanks, that's very helpful. (I hadn't realized that their native utility used ffmpeg and could decode basically any audio because of that -- very cool!)
- zX41ZdbW 4y agoIt did not get confused by my heavy accent, and the result is surprisingly good: https://jonatron.github.io/fosdem2023whisper/files/fast_data_analytical_apps_with_clickhouse.webm.txt https://jonatron.github.io/fosdem2023whisper/files/fast_data... Original: https://www.youtube.com/watch?v=JlcI2Vfz_uk https://www.youtube.com/watch?v=JlcI2Vfz_uk But some mistakes are hilarious! > 500 megabytes per second, Britain. Britain. Guess what it actually was :)
- jonatron 4y agoTo be fair, you did say Britain, when I think you mean B Written.
- wslh 4y agoI don't know how companies like Grammarly will survive this AI summer.
- jonatron 4y agoSubtitles, LLM, DeepSpeech comparisons are good ideas, so say if you're going to do them. I'm not going to do much more.
- tuukkah 4y agoYou can git pull from here for subtitles in the WebVTT format: https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt (I'm not able to make a pull request because GitHub is giving me the pink unicorn error page when I try to fork your repo.)
- tuukkah 4y agoGot a reply from GitHub that they fixed the problem, so here's the pull request: https://github.com/jonatron/fosdem2023whisper/pull/1 https://github.com/jonatron/fosdem2023whisper/pull/1
- deleted 4y ago[deleted]
- ldubost 4y agoVery Nice. I've use the "all transcript" to make a Word Cloud: https://framapiaf.org/@ldubost/109851711189159771 https://framapiaf.org/@ldubost/109851711189159771
- fareesh 4y agoThe accuracy of whisper seems to be the best available for mere mortals, but the resource requirements are fairly high. It doesn't seem like it is possible to use it for real-time transcription. What solutions are Twitter spaces and the like using for real-time live captions?
- adgjlsfhk1 4y agothere's a c++ fork of whisper that can run the smaller models on a laptop CPU in real time
- fareesh 4y agoLooks like it has improved a lot since I last checked it out. Do you happen to know what kind of EC2 instance (or equivalent config) would be required to run it?
- funstuff007 4y agoThe New Yorker had a recent piece on Whisper as well. https://www.newyorker.com/tech/annals-of-technology/whispers-of-ais-modular-future https://www.newyorker.com/tech/annals-of-technology/whispers... For them, the key difference (other than it's great performance) was that it was a standalone executable that did not need to access remote resources or "share" sensitive information with such remote resources.
- college_physics 4y agoWow! In terms of resource usage, for this and many other "emerging" opportunities, is there some thinking about pooling idle CPU's/GPUs for creating open and widely available models and/or datasets based on models? SETI type crowdsourcing of compute was always a bit niche but maybe now we have some use cases of very broad interest
- grammers 4y agoUh, nice, thanks for sharing!
- sluongng 4y agoThis is wonderful. I was able to search for my favorite topics by cloning the repo and grep through all the TXT files. Amazing!
- ronef 4y agoThis is great! Does it include all talks? I just noticed that it might be missing https://fosdem.org/2023/schedule/event/nix_and_nixos_runix/ https://fosdem.org/2023/schedule/event/nix_and_nixos_runix/.
- nbenitezl 4y agoThis is gold for us not native english speakers.
- specproc 4y agoIt's gold for non-native speakers of any supported language. I'm trying to learn some Spanish, and a fun and easy way to study is to watch films in Spanish with Spanish-language subs. Finding Jurassic Park on piratebay was easy, finding the right subs was impossible. Enter Whisper. My favourite little hack this year.
- trynewideas 4y agoEvery use of "fuck": > So, you can see my Firefox is in Italian, but you can see that it automatically detected that the page is in French and it is suggesting me to translate it to Italian. I will change it to English. Oh, fuck. > The reason for that is because there is so many ways to extend or to define an architecture, and there is some really fucked up things that can be done in so many architectures > We don't measure RSS because it measures anything useful we measure it because it's really fucking easy to measure. > There were a few customers that didn't respond, but at some point you just have to decide to. Don't give a fuck. > Computational storage, what the fuck's that? > like on MacOs you can have a recovery key, on Windows you can have a recovery key, and this is what we should do here as well, right, like use TPMs absolutely, and then hide entropy recovery keys that things are fucked up, things are fucked up > we get money, we invest in open source, and then we have the problem that what we do in open source could be reused by non-European companies. And thinking, oh, my God. Oh, that's so fucking wrong. Why do you do that? > So, you know, if Monty is not screaming at me saying you are fucking moron, Peter, that is not how it is, then probably I am not doing my job properly. > I'm going to wail and gnash my teeth and hope it doesn't happen in practice because at the moment, you know, something like EDK2, to be clear, by the way, I don't give a fuck about non-free firmwares > Is there anybody here who doesn't know what a kubelet is? Okay. The kubelet, oh, you fuck, anyway. > So for now we're targeting your own CA so you can just say fuck it, I will enroll my own keys to the firmware. > So one of the idea is to start to come to a more predictable fashion, which is one major API break and API break every year around December, January, so we're in February, and we fuck this year. > It's really not sure whether we are going to be able to do that because you're transpiling assembly. Like, what the fuck are you talking about? > your problem was where you called, in this case, pop on an empty stack, and not the completely pointless unwrap in there, because you want to know where you're fucked up and not here, obviously. > So if it is, then say, say like, this is fucking wrong, Peter, you know, so I can fix my slides when I talk next time, I have the wrong stuff, right?
- PeterZaitsev 4y agoOOps. Busted :) At least one of those is mine
- remram 4y agoI was wondering how that compared to Vosk, a lightweight pip-installable speech-to-text library. So I ran Vosk on some of the videos to compare. The model I used is 40MB and transcribed the 32min keynote in 2m12s on 1 CPU core, using 400MB of memory. The result is not that far. There are some errors of course, notably on acronyms and technical terms ("IBM", "GNU", "GitHub"), but a lot of the mismatches are in areas where both models got things wrong. Neither of the models got "FOSDEM" right, though Whisper is consistent at using the word "fosstem", whatever that is?
- brokenmachine 4y agoHow heavy on resources is Whisper? How long does an average GPU need to transcribe, say 2 hours of video?