4 ms·

Everything is doable if you have detailed description of the sensor. Do you know any sensor to get a better resolution with comparable frame rate and detailed s

Everything is doable if you have detailed description of the sensor. Do you know any sensor to get a better resolution with comparable frame rate and detailed specs? Can RAM limits be improved with a simple compression?

penagwin 7y ago
He did his testing on a raspberry pi 3 which means only 1GB of LPDDR2. I'd love to see how the RAM limits improve with 4GB LPDDR4, especially with usb 3.0 and real ethernet....
- novaRom 7y ago
  Camera interface is not via USB on Raspberry Pi. There's a lot of redundancy in high frame rate, so the best way to improve memory footprint is compression.
  - penagwin 7y ago
    My comment on the USB part is more asking the increased bandwidth make it possible to store it on a disk/network.
    - londons_explore 7y ago
      Most new devices have realised that storing RAW image data in realtime is impractical - it can be gigabytes per second! Instead, one needs to use a GPU, DSP, or other dedicated hardware to process/compress the data into a common video or image format.
      - 420codebro 7y ago
        I hope I don't sound like a retard asking this: where does one read more about detailed [hardware] image processing?
      - codetrotter 7y ago
        I don’t know enough about video compression algorithms to do what I would like to do, so I started reading a bit more about it, starting with an overview of lossless compression techniques from 2012 [1] and looking at the data sheet [2] for the image sensor linked to from [3]. The first thing of note to me in [1] is the following: > Digital video cameras perform a process called demosaicking before compression, which normally is lossy. While capturing images, digital video camera, uses only one of the three primary colors (red, green, blue) at each pixel and the remaining colors are interpolated through color demosaicking to reconstruct the full color image. A demosaicking algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). The most commonly used CFA is Bayer Filter, where the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. The process of demosaicking, followed by lossy compression is irreversible and the video cannot be improved later on. Further, this process increases complexity, reduces compression ratio and burdens the camera I/O bandwidth. > [...] if a lossless compression is performed first in the camera itself before demosaicking, then a sophisticated codec can be designed which is needed in applications where high visual quality has paramount importance. This motivated the use of lossless-compression first followed by demosaicking in medical videos where even the slightest degradation result is catastrophic situations. The algorithm uses a hybrid scheme of inter- and intraframe lossless compression to increase the compression and quality of the medical video data. The working of the encoding algorithm is briefed below and is given pictorially in Figure 1. Skimming through the data sheet, I see a few things: - Among the features on page 1 it mentions a 10-bit ADC, and “R, G, B primary color pigment mosaic filters on chip”. - On page 10, there is a block diagram, figure 1. - Starting at page 52 and throughout various pages of the remainder of the data sheet, there are figures that indicate to me that the output of the IMX 219 consists of mosaic pixel data. So if I am understanding correctly, demosaicing happens outside of the IMX 219. So I am wondering, with the v2 camera module as a whole, is demosaicing performed on the module itself or in software on the Raspberry Pi? And if it happens on the module itself, can it optionally be disabled so that you can get the mosaic pixel array data? Since the ADC is 10-bit, each pixel in the pixel array is presumably represented by 10 bits. The data sheet might say, I only skimmed through it on the initial read. If so, then 640x64 pixels * 10 bits/pixel * 660 FPS ~= 270 Mbps ~= 34 MBps. A benchmark from 2018 [4] puts the Raspberry Pi model 3B+ at being capable of writing to the SD card at about 22 MBps. So if we get the mosaic pixel array data, the most simple and naive thing we could do would be to “crop” the image to say 360 pixels wide in memory by skipping past the 140 first pixels of each line and copying the 360 next pixels of each of the 64 lines for a few frames, writing batches of 360x64 pixels of cropped frame data to a single file on the SD card. If the mosaic pixel array is what we are given by default, or we can get the camera module to send it anyway, then with discarding data we have a starting point. At this point we are able to record for as long as the capacity of our SD card will allow us. 128 GB SD cards are not terribly expensive so it should be possible to record data for like a couple of minutes. Then we can later post process the data. And if that is possible, then one can start to get really serious and try to device an efficient lossless compression algorithm specifically for our 640x64 pixels of video frames, and perhaps even optimized for different use-cases like horizontal motion only, vertical motion only, and motion in horizontal and vertical directions at the same time. [1]: https://pdfs.semanticscholar.org/57ee/90a3e59003ec83c2f52aba1f86b1b05c1231.pdf https://pdfs.semanticscholar.org/57ee/90a3e59003ec83c2f52aba... [2]: https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Raspberry-Pi-V2-CMOS/master/RASPBERRY%20PI%20CAMERA%20V2%20DATASHEET%20IMX219PQH5_7.0.0_Datasheet_XXX.PDF https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Rasp... [3]: https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camera-Module https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camer... [4]: https://www.jeffgeerling.com/blog/2018/raspberry-pi-microsd-card-performance-comparison-2018 https://www.jeffgeerling.com/blog/2018/raspberry-pi-microsd-...
      - HermannSW 7y ago
        There is a drawback with Raspbian, while Raspbian itself is open source, the GPU code is closed source and you cannot debug it. What you can do is to reverse engineer the I2C traffic from Raspberry GPU to camera (that is one-way): https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=109137&p=1237033#p1237647 https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=1091... And you can hack the camera by injecting I2C commands to achieve effects impossible with the normal Raspbian camera software, like taking odd/even frames with different shutter speed: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212694&p=1313054#p1313054 https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=2126... For reverse engineering camera I2C traffic, the v1 camera ov5647 datasheet is useful to have: https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_full.pdf https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_f...
  - zaarn 7y ago
    At high framerate, there isn't much time to do any type of fancy compression. 660fps means you have 1.5 milliseconds to process the image. The bandwidth will roughly max out between 50 Megabytes/s or 432Mbit/s. LPDDR2 (on the RPi3) can do 800 Transfers are second, at a maxmium transfer size of 4kb, maxing out almost all of your memory bandwidth already. So you literally do not have the memory bandwidth to do compression while also capturing a good 640x64x32/660 Video. If you have an RPi4, you get LPDDR4, the memory bandwidth now allows for compression, IF the CPU can handle that throughput (which it likely can't since the RPi4 chokes on such large bandwidths easily).
    - rasz 7y ago
      Ram speed is not the issue, camera interface is limited to 120MB/s, ram bandwidth is >2GB/s on pee3, 4GB/s on pee4.
      - zaarn 7y ago
        If you want to compress the frames, RAM bandwidth will be an issue.
        rasz 7y ago
        Compressing 640x64@660 takes exactly the same amount of work as compressing 1280x720@30. You could even use hardware h.264 encoder as long as you can debayer raw 10bit in software fast enough (or convince Broadcom employee 6by9 to make internal Videocore firmware debayering resolution agnostic, its already fast enough keeping up with twice the amount of data when using camera in 1920x1080@25 mode). But you dont even need fancy compression, at those framerates things change really slowly so simple delta encoding on raw data will do wonders.
        zaarn 7y ago
        Compressing 660fps in real time is a much different task than 30 frames per second, notably because your latency budget is a lot lower. At 30fps you have 33ms to load the frame, compress it and send it off to the destination, at 660fps you have 1.5ms for the same task, which does not necessarily get exponentially faster with lowered resolution.

penagwin 7y ago

He did his testing on a raspberry pi 3 which means only 1GB of LPDDR2. I'd love to see how the RAM limits improve with 4GB LPDDR4, especially with usb 3.0 and real ethernet....

novaRom 7y ago
Camera interface is not via USB on Raspberry Pi. There's a lot of redundancy in high frame rate, so the best way to improve memory footprint is compression.
- penagwin 7y ago
  My comment on the USB part is more asking the increased bandwidth make it possible to store it on a disk/network.
  - londons_explore 7y ago
    Most new devices have realised that storing RAW image data in realtime is impractical - it can be gigabytes per second! Instead, one needs to use a GPU, DSP, or other dedicated hardware to process/compress the data into a common video or image format.
    - 420codebro 7y ago
      I hope I don't sound like a retard asking this: where does one read more about detailed [hardware] image processing?
    - codetrotter 7y ago
      I don’t know enough about video compression algorithms to do what I would like to do, so I started reading a bit more about it, starting with an overview of lossless compression techniques from 2012 [1] and looking at the data sheet [2] for the image sensor linked to from [3]. The first thing of note to me in [1] is the following: > Digital video cameras perform a process called demosaicking before compression, which normally is lossy. While capturing images, digital video camera, uses only one of the three primary colors (red, green, blue) at each pixel and the remaining colors are interpolated through color demosaicking to reconstruct the full color image. A demosaicking algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). The most commonly used CFA is Bayer Filter, where the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. The process of demosaicking, followed by lossy compression is irreversible and the video cannot be improved later on. Further, this process increases complexity, reduces compression ratio and burdens the camera I/O bandwidth. > [...] if a lossless compression is performed first in the camera itself before demosaicking, then a sophisticated codec can be designed which is needed in applications where high visual quality has paramount importance. This motivated the use of lossless-compression first followed by demosaicking in medical videos where even the slightest degradation result is catastrophic situations. The algorithm uses a hybrid scheme of inter- and intraframe lossless compression to increase the compression and quality of the medical video data. The working of the encoding algorithm is briefed below and is given pictorially in Figure 1. Skimming through the data sheet, I see a few things: - Among the features on page 1 it mentions a 10-bit ADC, and “R, G, B primary color pigment mosaic filters on chip”. - On page 10, there is a block diagram, figure 1. - Starting at page 52 and throughout various pages of the remainder of the data sheet, there are figures that indicate to me that the output of the IMX 219 consists of mosaic pixel data. So if I am understanding correctly, demosaicing happens outside of the IMX 219. So I am wondering, with the v2 camera module as a whole, is demosaicing performed on the module itself or in software on the Raspberry Pi? And if it happens on the module itself, can it optionally be disabled so that you can get the mosaic pixel array data? Since the ADC is 10-bit, each pixel in the pixel array is presumably represented by 10 bits. The data sheet might say, I only skimmed through it on the initial read. If so, then 640x64 pixels * 10 bits/pixel * 660 FPS ~= 270 Mbps ~= 34 MBps. A benchmark from 2018 [4] puts the Raspberry Pi model 3B+ at being capable of writing to the SD card at about 22 MBps. So if we get the mosaic pixel array data, the most simple and naive thing we could do would be to “crop” the image to say 360 pixels wide in memory by skipping past the 140 first pixels of each line and copying the 360 next pixels of each of the 64 lines for a few frames, writing batches of 360x64 pixels of cropped frame data to a single file on the SD card. If the mosaic pixel array is what we are given by default, or we can get the camera module to send it anyway, then with discarding data we have a starting point. At this point we are able to record for as long as the capacity of our SD card will allow us. 128 GB SD cards are not terribly expensive so it should be possible to record data for like a couple of minutes. Then we can later post process the data. And if that is possible, then one can start to get really serious and try to device an efficient lossless compression algorithm specifically for our 640x64 pixels of video frames, and perhaps even optimized for different use-cases like horizontal motion only, vertical motion only, and motion in horizontal and vertical directions at the same time. [1]: https://pdfs.semanticscholar.org/57ee/90a3e59003ec83c2f52aba1f86b1b05c1231.pdf https://pdfs.semanticscholar.org/57ee/90a3e59003ec83c2f52aba... [2]: https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Raspberry-Pi-V2-CMOS/master/RASPBERRY%20PI%20CAMERA%20V2%20DATASHEET%20IMX219PQH5_7.0.0_Datasheet_XXX.PDF https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Rasp... [3]: https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camera-Module https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camer... [4]: https://www.jeffgeerling.com/blog/2018/raspberry-pi-microsd-card-performance-comparison-2018 https://www.jeffgeerling.com/blog/2018/raspberry-pi-microsd-...
    - HermannSW 7y ago
      There is a drawback with Raspbian, while Raspbian itself is open source, the GPU code is closed source and you cannot debug it. What you can do is to reverse engineer the I2C traffic from Raspberry GPU to camera (that is one-way): https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=109137&p=1237033#p1237647 https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=1091... And you can hack the camera by injecting I2C commands to achieve effects impossible with the normal Raspbian camera software, like taking odd/even frames with different shutter speed: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212694&p=1313054#p1313054 https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=2126... For reverse engineering camera I2C traffic, the v1 camera ov5647 datasheet is useful to have: https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_full.pdf https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_f...
- zaarn 7y ago
  At high framerate, there isn't much time to do any type of fancy compression. 660fps means you have 1.5 milliseconds to process the image. The bandwidth will roughly max out between 50 Megabytes/s or 432Mbit/s. LPDDR2 (on the RPi3) can do 800 Transfers are second, at a maxmium transfer size of 4kb, maxing out almost all of your memory bandwidth already. So you literally do not have the memory bandwidth to do compression while also capturing a good 640x64x32/660 Video. If you have an RPi4, you get LPDDR4, the memory bandwidth now allows for compression, IF the CPU can handle that throughput (which it likely can't since the RPi4 chokes on such large bandwidths easily).
  - rasz 7y ago
    Ram speed is not the issue, camera interface is limited to 120MB/s, ram bandwidth is >2GB/s on pee3, 4GB/s on pee4.
    - zaarn 7y ago
      If you want to compress the frames, RAM bandwidth will be an issue.
      - rasz 7y ago
        Compressing 640x64@660 takes exactly the same amount of work as compressing 1280x720@30. You could even use hardware h.264 encoder as long as you can debayer raw 10bit in software fast enough (or convince Broadcom employee 6by9 to make internal Videocore firmware debayering resolution agnostic, its already fast enough keeping up with twice the amount of data when using camera in 1920x1080@25 mode). But you dont even need fancy compression, at those framerates things change really slowly so simple delta encoding on raw data will do wonders.
        zaarn 7y ago
        Compressing 660fps in real time is a much different task than 30 frames per second, notably because your latency budget is a lot lower. At 30fps you have 33ms to load the frame, compress it and send it off to the destination, at 660fps you have 1.5ms for the same task, which does not necessarily get exponentially faster with lowered resolution.