4 ms·

I'm planning a beat-aware YouTube player. Unfortunately, it is not possible to access the audio stream data across the YouTube embed. (For beat-detection analys

I'm planning a beat-aware YouTube player. Unfortunately, it is not possible to access the audio stream data across the YouTube embed. (For beat-detection analysis.) I considered using the microphone like this. It's nice to see it works. Although there seems to be a time limit to how long the microphone can record? My plan was to download video (youtube-dlp), then make a CLI tool that analyzes and uploads the beat-detection data. An advantage of the CLI tool is it can complete the analysis faster than playing the video at 1X.

kavinaidoo 1y ago
Yeah, I also wanted to "pipe" the audio directly to the Web Speech API but had to resort to using the mic. Another feature I wanted to add was to have a keyboard shortcut insert a screenshot of the video into the notes. Handy for diagrams etc but I hit some roadblocks there too and the workarounds were getting crazy. Regarding the time-limit, I'm entirely accessing the audio stream using the Web Speech API and it "decides" (usually when there's a sufficient pause) when to finalize a recognition result. It was also firing a "recognition.onend" event after a certain amount of time (some old details here: https://stackoverflow.com/questions/38213580/chromes-webkitspeechrecognition-ends-randomly https://stackoverflow.com/questions/38213580/chromes-webkits...) so I have a workaround where if the user did not turn off speech-to-text, it is immediately restarted. You'll see console warnings "userStopped == false - trying to restart" when this happens. I assume you'll have no such issues because you'll be handling the "raw" audio data and not working through this API. Also, I've noticed that when using the Web Speech API on Safari, it does not "hear" what the tab is playing. It only hears external audio. I'm not sure whether this will be an issue for you but for me it means my app is Chrome only (suboptimal). Forgive me for my naïveté in this arena, It's the first time I'm using any browser audio API. Would your tool run on a server and work with a frontend for YT link ingestion or will you just use it yourself from the cli?
- Leftium 1y ago
  Ideally, my tool would run on the server. Especially if I wanted to monetize the service. (However I think there may be legal issues that are larger than the technical ones...) However it's just a small hobby project. So realistically I think this is how it will work: - If user tries to load a video without beat data yet, instructions are shown for how to add the beat data. The instructions will be running CLI command like `npx upload-beat-data [YOUTUBE-URL]` - The CLI tool will download the audio, then upload the beat data to my site. - The site will also log which videos got a lot of requests, but are missing beat data. So I can manually add the beat data myself.

kavinaidoo 1y ago

Yeah, I also wanted to "pipe" the audio directly to the Web Speech API but had to resort to using the mic. Another feature I wanted to add was to have a keyboard shortcut insert a screenshot of the video into the notes. Handy for diagrams etc but I hit some roadblocks there too and the workarounds were getting crazy. Regarding the time-limit, I'm entirely accessing the audio stream using the Web Speech API and it "decides" (usually when there's a sufficient pause) when to finalize a recognition result. It was also firing a "recognition.onend" event after a certain amount of time (some old details here: https://stackoverflow.com/questions/38213580/chromes-webkitspeechrecognition-ends-randomly https://stackoverflow.com/questions/38213580/chromes-webkits...) so I have a workaround where if the user did not turn off speech-to-text, it is immediately restarted. You'll see console warnings "userStopped == false - trying to restart" when this happens. I assume you'll have no such issues because you'll be handling the "raw" audio data and not working through this API. Also, I've noticed that when using the Web Speech API on Safari, it does not "hear" what the tab is playing. It only hears external audio. I'm not sure whether this will be an issue for you but for me it means my app is Chrome only (suboptimal). Forgive me for my naïveté in this arena, It's the first time I'm using any browser audio API. Would your tool run on a server and work with a frontend for YT link ingestion or will you just use it yourself from the cli?

Leftium 1y ago
Ideally, my tool would run on the server. Especially if I wanted to monetize the service. (However I think there may be legal issues that are larger than the technical ones...) However it's just a small hobby project. So realistically I think this is how it will work: - If user tries to load a video without beat data yet, instructions are shown for how to add the beat data. The instructions will be running CLI command like `npx upload-beat-data [YOUTUBE-URL]` - The CLI tool will download the audio, then upload the beat data to my site. - The site will also log which videos got a lot of requests, but are missing beat data. So I can manually add the beat data myself.