4 ms·

Fun fact: USB 2.0 webcams have been existing for over 10 years. USB 2.0 is 60 MB/s. A pixel of an image is 3 Bytes. A 1920x1080 FullHD image is 6.2 MB. At 30 f

Fun fact: USB 2.0 webcams have been existing for over 10 years. USB 2.0 is 60 MB/s. A pixel of an image is 3 Bytes. A 1920x1080 FullHD image is 6.2 MB. At 30 frames per second, second of a FullHD video is 186 MB. How did they do that? Answer: frames are transferred as JPEG files. Even a cheap $15 webcam is a tiny computer (with a CPU, RAM, etc), which runs a JPEG encoder program.

kevin_thibedeau 4y ago
The cheap ones are using hardware JPEG encoders. The associated micro isn't powerful enough to do it in firmware alone.
- masklinn 4y ago
  Surprised they don't use a hardware video encoder, is it because the well and efficiently supported formats are all MPEG, and thus have fairly high licensing cost on top of the hardware? Or because even efficient HVEs use more resources than webcams can afford? Or because inter-frame coding requires more storage, which (again) means higher costs, which (again) eats into the margin, which cheap webcam manufacturers consider not worth the investment?
  - martinmunk 4y ago
    My older Logitech C920 has an on-board H.264 encoder. Newer revisions of the same model does not. I haven't figured out why they chose to remove it, but your point about licensing cost combined with them not advertising it much as a feature, and most of their competitors not including "proper" video encoding might explain it. Edit: Found an official explanation here: https://www.logitech.com/en-us/video-collaboration/resources/think-tank/articles/article-logitech-and-h264-encoding.html https://www.logitech.com/en-us/video-collaboration/resources... TLDR, they figure most computers at that point had HW encoders.
    - lxgr 4y ago
      Unfortunately, this makes it much harder to use these as webcams on a Raspberry Pi (which even has H.264 hardware acceleration – the bottleneck is decoding the MJPEG stream from the camera, for which ffmpeg does not have hardware acceleration on the RPi).
      - broomhall 4y ago
        As an alternative to ffmpeg, GStreamer provides hardware accelerated MJPEG decoding on the Pi. I think there are bugs, though, which makes it unsuitable for some use cases. Here's an example pipeline - https://forums.raspberrypi.com/viewtopic.php?p=1989575#p1989575 https://forums.raspberrypi.com/viewtopic.php?p=1989575#p1989...
  - izacus 4y ago
    MJPEG is just a very simple "video" format that needs very simple and cheap electronics to work. Video encoding blocks are mostly part of bigger SoCs and comes with licensing costs. Same goes on the other hand for the receiving end - decoding a stream of JPEGs is just much simpler in both CPU use and code complexity than dealing with something like H.264.
grishka 4y ago
Hm. But then wouldn't it make more sense to just stream the raw sensor data, which is 1 byte per pixel (or up to 12 bits if you want to get fancy), and then demosaic it on the host? Full HD at 30 fps would be 59.33 MB/s, barely but still fitting into that limit. But then also I think some webcams use H264? I remember reading that somewhere.
- masklinn 4y ago
  > Full HD at 30 fps would be 59.33 MB/s, barely but still fitting into that limit. It's not fitting into anything I fear, best case scenario the effective bulk transfer rate of USB2 is 53MB/s. 60 is the signaling rate, but that doesn't account for the framing or the packet overhead.
- BayAreaEscapee 4y ago
  I don't know where you get "1 byte per pixel" from. At minimum, raw 4:2:0 video would be two bytes per pixel, and RGB would be three bytes per pixel with 8-bit color depth.
  - Dylan16807 4y ago
    When talking about digital cameras, each "pixel" is a single color sensor. Blame marketing. Also 4:2:0 is 6 values per 4 pixels. 1.5 bytes per pixel at 8-bit.
  - nybble41 4y ago
    You're talking about processed color frames. The GP was suggesting that the camera stream the raw sensor data, which doesn't have individual color channels, just a monochrome grid with 10 or 12 bits of usable data per pixel. A bayer filter[0] is placed in front of the sensor so that a given color of light falls on each cell. The USB host would be responsible for applying a demosaicing[0] algorithm to create the color channels from the raw sensor data. If we take the AR0330 sensor used in the USB Camera C1[2] as an example, it has a native resolution of 2304H x 1296V and outputs 10 bits per native pixel after internal A-Law compression[3] for a total raw frame size of 3.56 MiB, assuming optimal packing. The corresponding image, demosaiced and downscaled to Full HD (1920x1080), in RGB with eight bits per channel would be 5.93 MiB. [0] https://en.wikipedia.org/wiki/Bayer_filter https://en.wikipedia.org/wiki/Bayer_filter [1] https://en.wikipedia.org/wiki/Demosaicing https://en.wikipedia.org/wiki/Demosaicing [2] https://www.kurokesu.com/shop/cameras/CAMUSB1 https://www.kurokesu.com/shop/cameras/CAMUSB1 [3] https://www.onsemi.com/products/sensors/image-sensors/ar0330 https://www.onsemi.com/products/sensors/image-sensors/ar0330
verall 4y ago
It needs a uC with some special hardware anyways to do demosaic or else it would require special drivers that would peg some people's crappy laptop CPUs. Also the raw YUV 4:2:0 is 1.5 bytes per pixel so that's doing half of the "compression" work for you.
pseudosavant 4y ago
Most webcams, especially 10 years ago are not 1080p, or even 60fps. Many aren't even 720p. 1280 x 720 x 3 bytes x 30 fps = ~80MB/second. 480p @ 30 fps = 26MB. That is how many webcams can get by without hardware JPEG/H264 encoding. 4K @ 60fps = 1.4GB/sec. USB 3, even with 2 lanes, will have trouble with that.

kevin_thibedeau 4y ago

The cheap ones are using hardware JPEG encoders. The associated micro isn't powerful enough to do it in firmware alone.

masklinn 4y ago
Surprised they don't use a hardware video encoder, is it because the well and efficiently supported formats are all MPEG, and thus have fairly high licensing cost on top of the hardware? Or because even efficient HVEs use more resources than webcams can afford? Or because inter-frame coding requires more storage, which (again) means higher costs, which (again) eats into the margin, which cheap webcam manufacturers consider not worth the investment?
- martinmunk 4y ago
  My older Logitech C920 has an on-board H.264 encoder. Newer revisions of the same model does not. I haven't figured out why they chose to remove it, but your point about licensing cost combined with them not advertising it much as a feature, and most of their competitors not including "proper" video encoding might explain it. Edit: Found an official explanation here: https://www.logitech.com/en-us/video-collaboration/resources/think-tank/articles/article-logitech-and-h264-encoding.html https://www.logitech.com/en-us/video-collaboration/resources... TLDR, they figure most computers at that point had HW encoders.
  - lxgr 4y ago
    Unfortunately, this makes it much harder to use these as webcams on a Raspberry Pi (which even has H.264 hardware acceleration – the bottleneck is decoding the MJPEG stream from the camera, for which ffmpeg does not have hardware acceleration on the RPi).
    - broomhall 4y ago
      As an alternative to ffmpeg, GStreamer provides hardware accelerated MJPEG decoding on the Pi. I think there are bugs, though, which makes it unsuitable for some use cases. Here's an example pipeline - https://forums.raspberrypi.com/viewtopic.php?p=1989575#p1989575 https://forums.raspberrypi.com/viewtopic.php?p=1989575#p1989...
- izacus 4y ago
  MJPEG is just a very simple "video" format that needs very simple and cheap electronics to work. Video encoding blocks are mostly part of bigger SoCs and comes with licensing costs. Same goes on the other hand for the receiving end - decoding a stream of JPEGs is just much simpler in both CPU use and code complexity than dealing with something like H.264.

grishka 4y ago

Hm. But then wouldn't it make more sense to just stream the raw sensor data, which is 1 byte per pixel (or up to 12 bits if you want to get fancy), and then demosaic it on the host? Full HD at 30 fps would be 59.33 MB/s, barely but still fitting into that limit. But then also I think some webcams use H264? I remember reading that somewhere.

masklinn 4y ago
> Full HD at 30 fps would be 59.33 MB/s, barely but still fitting into that limit. It's not fitting into anything I fear, best case scenario the effective bulk transfer rate of USB2 is 53MB/s. 60 is the signaling rate, but that doesn't account for the framing or the packet overhead.
BayAreaEscapee 4y ago
I don't know where you get "1 byte per pixel" from. At minimum, raw 4:2:0 video would be two bytes per pixel, and RGB would be three bytes per pixel with 8-bit color depth.
- Dylan16807 4y ago
  When talking about digital cameras, each "pixel" is a single color sensor. Blame marketing. Also 4:2:0 is 6 values per 4 pixels. 1.5 bytes per pixel at 8-bit.
- nybble41 4y ago
  You're talking about processed color frames. The GP was suggesting that the camera stream the raw sensor data, which doesn't have individual color channels, just a monochrome grid with 10 or 12 bits of usable data per pixel. A bayer filter[0] is placed in front of the sensor so that a given color of light falls on each cell. The USB host would be responsible for applying a demosaicing[0] algorithm to create the color channels from the raw sensor data. If we take the AR0330 sensor used in the USB Camera C1[2] as an example, it has a native resolution of 2304H x 1296V and outputs 10 bits per native pixel after internal A-Law compression[3] for a total raw frame size of 3.56 MiB, assuming optimal packing. The corresponding image, demosaiced and downscaled to Full HD (1920x1080), in RGB with eight bits per channel would be 5.93 MiB. [0] https://en.wikipedia.org/wiki/Bayer_filter https://en.wikipedia.org/wiki/Bayer_filter [1] https://en.wikipedia.org/wiki/Demosaicing https://en.wikipedia.org/wiki/Demosaicing [2] https://www.kurokesu.com/shop/cameras/CAMUSB1 https://www.kurokesu.com/shop/cameras/CAMUSB1 [3] https://www.onsemi.com/products/sensors/image-sensors/ar0330 https://www.onsemi.com/products/sensors/image-sensors/ar0330

verall 4y ago

It needs a uC with some special hardware anyways to do demosaic or else it would require special drivers that would peg some people's crappy laptop CPUs. Also the raw YUV 4:2:0 is 1.5 bytes per pixel so that's doing half of the "compression" work for you.

pseudosavant 4y ago

Most webcams, especially 10 years ago are not 1080p, or even 60fps. Many aren't even 720p. 1280 x 720 x 3 bytes x 30 fps = ~80MB/second. 480p @ 30 fps = 26MB. That is how many webcams can get by without hardware JPEG/H264 encoding. 4K @ 60fps = 1.4GB/sec. USB 3, even with 2 lanes, will have trouble with that.