3 ms·

Video Rendering with Node.js and FFmpeg

46 points by fagnerbrack 4y ago

wartron 4y ago
Im gonna need to dive further into this, I do js canvas and webgl rendering to videos with the chrome/puppeteer via 'puppeteer-screen-recorder'. But they do at times cause some issues. Latest 3d renders have been causing memory issues that I think would be solved with bigger boxes but havent need to investigate for a little while. Have thought about it just outputting frames and then later having ffmpeg sticth them into a video, but havent gotten around to really testing it. Im guessing this limited to 2d canvas, but excited to check it out. Thanks!
- brianshaler 4y ago
  > Im guessing this limited to 2d canvas WebGL/3D works fine, just with some additional dependencies (e.g. mesa drivers) and a little more setup in Nodejs to create the context and copy the framebuffer to node-canvas to do the image encoding. Here's a little 3D animation I've rendered using a similar technique (plus WebGL) in a docker container: https://www.carvana.com/shareyourcarvana/MjM2MTc2OTpBbGV4YW5kZXI?utm_source=TwitterSYC https://www.carvana.com/shareyourcarvana/MjM2MTc2OTpBbGV4YW5... The main thing to watch out for is whether you need specific WebGL extensions that might not be supported. Array instancing is the main one I use, which is supported.
msla 4y ago
"Thinking quickly, I made a megaphone by taking a squirrel and tying it to a megaphone I had." Seriously: This is an interesting article because it's about more than just rendering video, it's about creating a new video procedurally and then rendering it with that pre-existing rendering program. It would be nice if the title reflected that.
sbarre 4y ago
Worth also looking at Remotion[0] if you want to do motion graphics via Javascript 0: https://www.remotion.dev/ https://www.remotion.dev/
- undershirt 4y ago
  Also Motion Canvas: https://motioncanvas.io https://motioncanvas.io
notpushkin 4y ago
Oddly enough, I've done something pretty similar, but using Python + PIL instead: https://github.com/notpushkin/glitzz https://github.com/notpushkin/glitzz Two improvements you can borrow: 1. FFmpeg can read frames from HTTP URL, which you can (ab)use to generate frames on the fly without hitting the disk 2. If you do that, you can also switch from PNG to BMP, since CPU is now your bottleneck
- brianshaler 4y ago
  Another way to do it on the fly is to feed image data into stdin of ffmpeg. I haven't tried with BMP with that method, so the intermediary encoding might not be spared.
sandreas 4y ago
Would this also be possible with the vlc WASM project? https://code.videolan.org/jbk/vlc.js https://code.videolan.org/jbk/vlc.js
midasuni 4y ago
> There is no better way to convey your message than by using video If you start your article with something so blatantly untrue why should I believe the rest?

46 points by fagnerbrack 4y ago

Im gonna need to dive further into this, I do js canvas and webgl rendering to videos with the chrome/puppeteer via 'puppeteer-screen-recorder'. But they do at times cause some issues. Latest 3d renders have been causing memory issues that I think would be solved with bigger boxes but havent need to investigate for a little while. Have thought about it just outputting frames and then later having ffmpeg sticth them into a video, but havent gotten around to really testing it. Im guessing this limited to 2d canvas, but excited to check it out. Thanks!

brianshaler 4y ago
> Im guessing this limited to 2d canvas WebGL/3D works fine, just with some additional dependencies (e.g. mesa drivers) and a little more setup in Nodejs to create the context and copy the framebuffer to node-canvas to do the image encoding. Here's a little 3D animation I've rendered using a similar technique (plus WebGL) in a docker container: https://www.carvana.com/shareyourcarvana/MjM2MTc2OTpBbGV4YW5kZXI?utm_source=TwitterSYC https://www.carvana.com/shareyourcarvana/MjM2MTc2OTpBbGV4YW5... The main thing to watch out for is whether you need specific WebGL extensions that might not be supported. Array instancing is the main one I use, which is supported.

msla 4y ago

"Thinking quickly, I made a megaphone by taking a squirrel and tying it to a megaphone I had." Seriously: This is an interesting article because it's about more than just rendering video, it's about creating a new video procedurally and then rendering it with that pre-existing rendering program. It would be nice if the title reflected that.

sbarre 4y ago

Worth also looking at Remotion[0] if you want to do motion graphics via Javascript 0: https://www.remotion.dev/ https://www.remotion.dev/

undershirt 4y ago
Also Motion Canvas: https://motioncanvas.io https://motioncanvas.io

notpushkin 4y ago

Oddly enough, I've done something pretty similar, but using Python + PIL instead: https://github.com/notpushkin/glitzz https://github.com/notpushkin/glitzz Two improvements you can borrow: 1. FFmpeg can read frames from HTTP URL, which you can (ab)use to generate frames on the fly without hitting the disk 2. If you do that, you can also switch from PNG to BMP, since CPU is now your bottleneck

brianshaler 4y ago
Another way to do it on the fly is to feed image data into stdin of ffmpeg. I haven't tried with BMP with that method, so the intermediary encoding might not be spared.

sandreas 4y ago

Would this also be possible with the vlc WASM project? https://code.videolan.org/jbk/vlc.js https://code.videolan.org/jbk/vlc.js

midasuni 4y ago

> There is no better way to convey your message than by using video If you start your article with something so blatantly untrue why should I believe the rest?