4 ms·
Ask HN: ML models on live video stream for production
I was trying to run a deep learning model on a live video stream in the server (and then send that video back to client). I tried using aiortc for this but doesn't seem scalable. Does anyone know other ways?
- tlack 6y agoYou could try running the model in the browser with Tensorflow.js. I recently tried this for something like that and found the overall workflow much easier. TFJS tooling is pretty good. Before I had tried using custom setups with Janus in the loop and it was slow going. AioRTC flat out never worked for me.
- yshanks 6y agoI am trying to run a segmentation model and tfjs is too slow for that. How do people do it on the server side? Something like zoom does with virtual background.
- tlack 6y agoI heard a rumor way back that they transported individual frames outside of WebRTC; if so, that saves them from having to tie in with that stack. I really think it should be possible or even easy to write a WebRTC videoframes-only client, I just haven't had any luck. In fact, I've seen a few people saying they simply scripted Chrome with a <video> tag instead! That's madness. I'm also looking for a fast depth or semantic segmentation model for use in my robotic arm end of arm. Let me know what you wind up using.