3 ms·

This is a bit off topic, but I'm constantly looking at ways to efficiently stream 4K cameras live to local displays as well as remote displays at the highest fr

This is a bit off topic, but I'm constantly looking at ways to efficiently stream 4K cameras live to local displays as well as remote displays at the highest framerate and resolution possible. How feasible would it be on the xavier to stream 2 4k cameras and display them on at least 2 4k screens? Extra points if you could do that and simultaneously upload to a streaming service, such as twitch.

m463 6y ago
I've wondered about this too. I think the magic camera interconnect is CSI/CSI2 and it's not really flexible enough. You either have really short copper interconnects, or unavailable fiber interconnects. What would be cool is if csi to ethernet were a thing. either low latency put-it-on-the-wire or compressed. I don't know, maybe it is. But make it a standard like rca jacks.
- achuwilson 6y ago
  It is possible to extend CSI through good HDMI cables https://www.robotshop.com/en/arducam-csi-hdmi-cable-extension-module-2pk.html https://www.robotshop.com/en/arducam-csi-hdmi-cable-extensio...
- manofmanysmiles 6y ago
  You can buy kits that send video over coax including power for somewhat reasonable prices: https://leopardimaging.com/product/nvidia-jetson-cameras/nvidia_agx_xavier_fpdlinkiii_camera_kits/li-ar0231-ti953-xavier/ https://leopardimaging.com/product/nvidia-jetson-cameras/nvi... I haven't tried them, but am considered them for project.
TaylorAlexander 6y ago
You can certainly do this using machine vision cameras. Either USB3, Gig Ethernet, or CSI interface (16cm max run length I believe). I forget how best to attach two displays to the Xavier but that’s seems doable. I got my cameras from e-consystems and they’ve got some USB3 cameras that could do it. At least I’m pretty sure. My USB3 cameras just showed up and I haven’t tried them yet.
- epmaybe 6y ago
  I actually have two econ systems cameras that I've been testing on my desktop/laptop, and it does all right but even then struggles at 4k resolutions.
  - TaylorAlexander 6y ago
    Interesting. I know e-consystems provided their own kernel modules for the Jetson which help with some of the camera streaming, and they provide a special binary of I think guvcview to view live camera output. That is for Xavier though so I don't know if they have something for just a regular computer. The other thing to look at is using hardware transcoding in an NVIDIA GPU, a raw 4k feed from a 4k camera is huge, but transcoded to h.264 or better h.265 the footage is much more playable from disk in my experience. It may help with live footage. Here are some notes I made when setting up GPU transcoding. Using xstack filter: https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos%20using%20xstack https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%... Using h265 (required to support the extra resolution of 4x4k) http://ntown.at/knowledgebase/cuda-gpu-accelerated-h264-h265-hevc-video-encoding-with-ffmpeg/ http://ntown.at/knowledgebase/cuda-gpu-accelerated-h264-h265... https://devblogs.nvidia.com/nvidia-ffmpeg-transcoding-guide/ https://devblogs.nvidia.com/nvidia-ffmpeg-transcoding-guide/ What is your monitor resolution? If you can't display the full resolution, I've found that using the nvidia encoder hardware to resize each 4k stream to 720p makes transcoding much faster. I've added my video conversion scripts to github so you can see how I've done that. https://github.com/tlalexander/rover_video_scripts https://github.com/tlalexander/rover_video_scripts You can also contact e-consystems, as they seem eager to provide application support. Finally feel free to email me to the email in my profile, or better yet create an account and ask the question on my website http://reboot.love http://reboot.love so other people can see our conversation and benefit from what we learn. EDIT: I JUST saw that you meant one local display and one remote display. Sorry busy day. In that case the other poster mentioning gstreamer is spot on, and I believe econsystems has some gstreamer plug ins at least for some cameras, or maybe nvidia does...
01100011 6y ago
Best bet is to take the raw video and run it through gstreamer. You should be able to setup a pipeline which displays the raw video locally while sending a compressed stream to the network. I'd bet that Nvidia has gstreamer modules which make use of their compression HW, so it might be possible. To be honest though, that's a lot of data, so I don't know how well dual 4k would work. You can always scale it down in gstreamer before you send it to the compression module. You'll probably want to use the CSI-2 interfaces to connect the cameras, but that depends. CSI-2 was developed for cell phones and is hard to run over long distances. It's optimized for low-power and designed for very short interconnects. We had a ton of problems using it at the last company I worked for. I really wish there was a competing standard for embedded cameras.
deeth_starr_v 6y ago
Not on a Xavier but on a TX2 I was able to get a single reasonable 4K stream with NVENC and gstreamer. Quality wasn't super awesome since it's realtime encoding. We ended up using two TX2s (one for each stream). I'd probably go with two Xaviers for better quality or step down and do 2 higher quality 1080p streams.

m463 6y ago

I've wondered about this too. I think the magic camera interconnect is CSI/CSI2 and it's not really flexible enough. You either have really short copper interconnects, or unavailable fiber interconnects. What would be cool is if csi to ethernet were a thing. either low latency put-it-on-the-wire or compressed. I don't know, maybe it is. But make it a standard like rca jacks.

achuwilson 6y ago
It is possible to extend CSI through good HDMI cables https://www.robotshop.com/en/arducam-csi-hdmi-cable-extension-module-2pk.html https://www.robotshop.com/en/arducam-csi-hdmi-cable-extensio...
manofmanysmiles 6y ago
You can buy kits that send video over coax including power for somewhat reasonable prices: https://leopardimaging.com/product/nvidia-jetson-cameras/nvidia_agx_xavier_fpdlinkiii_camera_kits/li-ar0231-ti953-xavier/ https://leopardimaging.com/product/nvidia-jetson-cameras/nvi... I haven't tried them, but am considered them for project.

TaylorAlexander 6y ago

You can certainly do this using machine vision cameras. Either USB3, Gig Ethernet, or CSI interface (16cm max run length I believe). I forget how best to attach two displays to the Xavier but that’s seems doable. I got my cameras from e-consystems and they’ve got some USB3 cameras that could do it. At least I’m pretty sure. My USB3 cameras just showed up and I haven’t tried them yet.

epmaybe 6y ago
I actually have two econ systems cameras that I've been testing on my desktop/laptop, and it does all right but even then struggles at 4k resolutions.
- TaylorAlexander 6y ago
  Interesting. I know e-consystems provided their own kernel modules for the Jetson which help with some of the camera streaming, and they provide a special binary of I think guvcview to view live camera output. That is for Xavier though so I don't know if they have something for just a regular computer. The other thing to look at is using hardware transcoding in an NVIDIA GPU, a raw 4k feed from a 4k camera is huge, but transcoded to h.264 or better h.265 the footage is much more playable from disk in my experience. It may help with live footage. Here are some notes I made when setting up GPU transcoding. Using xstack filter: https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos%20using%20xstack https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%... Using h265 (required to support the extra resolution of 4x4k) http://ntown.at/knowledgebase/cuda-gpu-accelerated-h264-h265-hevc-video-encoding-with-ffmpeg/ http://ntown.at/knowledgebase/cuda-gpu-accelerated-h264-h265... https://devblogs.nvidia.com/nvidia-ffmpeg-transcoding-guide/ https://devblogs.nvidia.com/nvidia-ffmpeg-transcoding-guide/ What is your monitor resolution? If you can't display the full resolution, I've found that using the nvidia encoder hardware to resize each 4k stream to 720p makes transcoding much faster. I've added my video conversion scripts to github so you can see how I've done that. https://github.com/tlalexander/rover_video_scripts https://github.com/tlalexander/rover_video_scripts You can also contact e-consystems, as they seem eager to provide application support. Finally feel free to email me to the email in my profile, or better yet create an account and ask the question on my website http://reboot.love http://reboot.love so other people can see our conversation and benefit from what we learn. EDIT: I JUST saw that you meant one local display and one remote display. Sorry busy day. In that case the other poster mentioning gstreamer is spot on, and I believe econsystems has some gstreamer plug ins at least for some cameras, or maybe nvidia does...

01100011 6y ago

Best bet is to take the raw video and run it through gstreamer. You should be able to setup a pipeline which displays the raw video locally while sending a compressed stream to the network. I'd bet that Nvidia has gstreamer modules which make use of their compression HW, so it might be possible. To be honest though, that's a lot of data, so I don't know how well dual 4k would work. You can always scale it down in gstreamer before you send it to the compression module. You'll probably want to use the CSI-2 interfaces to connect the cameras, but that depends. CSI-2 was developed for cell phones and is hard to run over long distances. It's optimized for low-power and designed for very short interconnects. We had a ton of problems using it at the last company I worked for. I really wish there was a competing standard for embedded cameras.

deeth_starr_v 6y ago

Not on a Xavier but on a TX2 I was able to get a single reasonable 4K stream with NVENC and gstreamer. Quality wasn't super awesome since it's realtime encoding. We ended up using two TX2s (one for each stream). I'd probably go with two Xaviers for better quality or step down and do 2 higher quality 1080p streams.