One other less well-defined project I'll mention: Using whisper on aircraft radio traffic. I think of ATC radio as a completely unindexed, unsearchable, "dark w

One other less well-defined project I'll mention: Using whisper on aircraft radio traffic. I think of ATC radio as a completely unindexed, unsearchable, "dark web" of information, and Whisper can open it up and make it searchable. Whisper is the first speech recognition system I've seen that can handle not just the typically low quality audio, but also can take into account contextual information. E.g. some of the most useful information in a transmission on an ATC radio frequency is the call sign of the aircraft. But it's very hard for most speech recognizers to accurately transcribe: "7XY" is essentially just as likely as "1AC". Short, basically random utterances are hell on speech recognizers. But Whisper's killer feature IMO (but weirdly one that people rarely seem to use or even know about) is the powerful language model and its ability to be prompted. Level 1 prompt engineering for Whisper is simply using a prompt like "Let's pretend we're air traffic controllers" or something to prime it to expect the specialized ATC lingo vs. any other thing people might be talking about. This prompt is specific to ATC, but is otherwise very general. Level 2 becomes specific to the frequency you're transcribing: "Cessna, El Monte Tower, cleared for the option runway 01." Now Whisper knows that it's ATC, and that the name of the tower (which it will hear a lot) is El Monte, and that there's a runway numbered 01. Level 3 is where you add additional time- and situation-dependent prompting to increase accuracy. If you look at ADS-B data, you can figure out which aircraft are/were in the area when the audio was recorded, that might be communicating on the radio. You can create prompts using those call signs, greatly increasing accuracy of transcription. (Some researchers have done work along these lines, pre-Whisper.) An example of what Whisper makes possible: Here's a "supercut" of all the times either a pilot or ATC mentioned "laser", across multiple frequencies, across multiple days: https://twitter.com/lemonodor/status/1578516727549153280 https://twitter.com/lemonodor/status/1578516727549153280 Here's an example of what I'd like to be able to do (I created it manually, but I don't think it's too far out of reach), a video showing the aircraft map synchronized with ATC audio across frequencies, from a few days ago when a Cessna busted the presidential TFR near Philadelphia: https://twitter.com/lemonodor/status/1605293275333607424 https://twitter.com/lemonodor/status/1605293275333607424 Not Whisper-related, but just a fun proof-of-concept of a browser extension that lets you click on aircraft on the map and listen to them on the radio: https://twitter.com/lemonodor/status/1521551159206416384 https://twitter.com/lemonodor/status/1521551159206416384