11 ms·

A Guide to Recording 660FPS Video on a $6 Raspberry Pi Camera

445 points by robertelder 7y ago

jschwartzi 7y ago
This is amazing. These framerates are 1/10th of what a cursory google search says I can get out of a top of the line high-speed camera, but for probably 1/100th the price. You could make so many awesome instructional videos with a tool like this.
- ortusdux 7y ago
  The chronos 1.4 I have will do 640x96 @ 21,649fps for $3k. So that's 20 times faster for 50 times the price; pretty close to your estimate. The chronos does have a few other amenities that make it worth the price.
  - ekr 7y ago
    That's 500 times the price (unless you're taking into account the raspberry pi itself).
    - hoseja 7y ago
      ... which you should
  - kragen 7y ago
    Is it made in the US? Is it export-controlled? Here in Argentina getting export-controlled gear is a pain.
    - ortusdux 7y ago
      They are made in and shipped from Canada. I doubt they are export-controlled, but I would email them. They were quite responsive to most of my inquires. Just don't ask when they might implement features. I'm still waiting on some stuff that are listed on the 2 year old spec sheet as "*features (that) are fully supported in the camera's hardware, but are not yet supported in software. They will be added in a free software update after the camera's initial release."
- bufferoverflow 7y ago
  You can shoot 960fps at 720p/1080p with some smartphones these days. If you decide to go that route, make sure to get one that doesn't interpolate the frames, but does true 960fps.
  - CobrastanJorji 7y ago
    Could you suggest such a phone because that sounds wicked awesome to experiment with.
    - CarVac 7y ago
      The Galaxy S10 series (including the S10e).
    - deleted 7y ago
      [deleted]
- ktpsns 7y ago
  Which brings me to this Gedankenexperiment: If you buy/build 10 of the raspberry pi based high speed cameras and run them in parallel and could make sure each camera sees the same (we don't consider details here for the time being), is it possible to merge the videos in order to reach the frame rate of a "top of the line high-speed camera"? I could imagine two scenarios: 1. Start recording with time delay. Probably requires excellent clock synchronisation, a real time kernel, reliable/comparable hardware (gatter timings differ at small timescales). If this works, this would provide a high quality equidistand framerate. or 2. Start recording randomly, synchronize by time or with an video event, and merge the frames. This will result in a stochastic distribution of frame rates. The usefulness of such a movie depends on what it is supposed to be used for (thinking of scientific usage). Will probably not give nice slow-mo movies. But here comes the catch: With interpolation, the shortest reachable frame rate dictates the quality of this "superposition camera". How many cheap cameras do we need to archieve, on average, the top line high speed camera? I leave the questions open for friends of Gedankenexperiments ;-)
  - rasz 7y ago
    pee cameras are clocked externally(EMI made them include 25MHz clock gen on camera module itself), so in theory you could connect them all to a single clock distribution source with appropriately shifted phases
    - sansnomme 7y ago
      How do you disable onboard clock or is it automatic?
      - rasz 7y ago
        its already external, on the back of each camera
        sansnomme 7y ago
        But it's only selectively enabled no?
- HermannSW 7y ago
  You can get Raspberry v1 camera clone now for less than 4$(!) with free shipping on aliexpress: https://www.aliexpress.com/wholesale?SearchText=raspberry+5MP&SortType=price_asc&g=y&isFreeShip=y&switch_new_app=y https://www.aliexpress.com/wholesale?SearchText=raspberry+5M... And unlike v2 camera, the cheap v1 camera allows (besides high framerate capturing) for global external shutter capturing as well(!). I did capture an in flight 109m/s airgun pellet 10 times in a single multiple exposure frame! https://stamm-wilbrandt.de/en/Raspberry_camera.html#global_external_shutter https://stamm-wilbrandt.de/en/Raspberry_camera.html#global_e...
yeahdef 7y ago
very impressive.
bavell 7y ago
Resolution: 640x64
- penagwin 7y ago
  I thought that was pretty low - but it certainly enough for some impressive shots! [0] Video from article: https://www.youtube.com/watch?v=-gMy8k4nHtw https://www.youtube.com/watch?v=-gMy8k4nHtw
- msie 7y ago
  Then get 10 Pi's to get 640x640?
  - azhenley 7y ago
    6400x640?
    - jefft255 7y ago
      Nope, that’s a 100 pies...
    - bduerst 7y ago
      You're stacking pis and sensors side-by-side, not multiplying them.
alvern 7y ago
1107 fps on the V2 IMX219 camera. I wonder how fast or how large of a frame it could do on a Pi4 or odroid (article mentions the 640x64 frame is limited by memory bandwidth).
- rasz 7y ago
  I think Author meant mipi CSI-2 2 lane bandwidth limit = 1.1 Gbps, ~somewhere around 120MB/s
- HermannSW 7y ago
  You can do 640x64@665fps with v1 camera, and 640x128@667fps with v2 camera: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212518&p=1310445#p1320034 https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=2125... In tools directory you can use *_s tools to get double FOV vertically while keeping the framerate. So you can get 640x128_s@665fps/640x256_s@667fps for v1/v2 camera. There real limit for v2 camera is 1007fps, with can be achieved with 640x75 or 640x150_s tools. You can reduce vertical resolution below 75, but you will not get more than 1007fps (if you find out how to raise that limit, please shout! Deal for raspiraw high framerate capturing basically is "halve vertical resolution and get double framerate").
canada_dry 7y ago
> processor registers on the image sensor are set so that raw data is sent continuously (to be processed later) Very slick. Make me wonder what crazy high-speed imaging hacks might be possible with a top-of-the-line sensor used in iPhone/Pixel phones.
- Gcfcow 7y ago
  The Samsung Galaxy S9 & S10 can record short bursts of 960fps video at 720p out of the box. I'd imagine a hack could extract similar numbers from the pixel and iPhone.
novaRom 7y ago
Everything is doable if you have detailed description of the sensor. Do you know any sensor to get a better resolution with comparable frame rate and detailed specs? Can RAM limits be improved with a simple compression?
- penagwin 7y ago
  He did his testing on a raspberry pi 3 which means only 1GB of LPDDR2. I'd love to see how the RAM limits improve with 4GB LPDDR4, especially with usb 3.0 and real ethernet....
  - novaRom 7y ago
    Camera interface is not via USB on Raspberry Pi. There's a lot of redundancy in high frame rate, so the best way to improve memory footprint is compression.
    - penagwin 7y ago
      My comment on the USB part is more asking the increased bandwidth make it possible to store it on a disk/network.
      - londons_explore 7y ago
        Most new devices have realised that storing RAW image data in realtime is impractical - it can be gigabytes per second! Instead, one needs to use a GPU, DSP, or other dedicated hardware to process/compress the data into a common video or image format.
        420codebro 7y ago
        I hope I don't sound like a retard asking this: where does one read more about detailed [hardware] image processing?
        codetrotter 7y ago
jermaustin1 7y ago
I read through, and I didn't find a reason that the resolution is 640x64@660 other than that is what was input into the raspiraw program. Could this record at a higher resolution?
- sp332 7y ago
  It's probably bottlenecked by how fast the sensor data can be read.
  - djsumdog 7y ago
    Yea they mention in the video, but don't mention what happens with a higher aspect ratio. I suspect it will cause artifacts, like part of one frame appearing on the previous or tearing or other distortions. Remember these sensors read a single row at a time. There's a Smarter Every Day video where they show how cellphones get those weird effects when recording plane propellers due to the way they scan images.
    - HermannSW 7y ago
      Yes, for objects moving with high speed vertically, rolling shutter capturing produces weird frames. But v1 camera can do "global external shutter" as well. See examples on "extreme rolling/global external" shutter for propellers here: https://github.com/Hermann-SW/Raspberry_camera_gallery/blob/master/propeller/README.md https://github.com/Hermann-SW/Raspberry_camera_gallery/blob/...
- bufferoverflow 7y ago
  Limited by the chip that reads the data off the sensor - ASIC of FPGA. Or maybe by the sensor design itself.
- pfranz 7y ago
  I haven't read through everything yet, but in the examples video[1] it's stated that they can only read a portion of the sensor to keep a high frame rate and that the camera returns one row at a time. A square image requires reading more rows and takes more time. [1] https://youtu.be/-gMy8k4nHtw?t=154 https://youtu.be/-gMy8k4nHtw?t=154
an4rchy 7y ago
Wow. Never thought about this approach to leveraging RAM for cameras. Awesome guide! Can anybody chime in regarding the resolution limit (640x64) -- If this is for memory reasons, could we pick alternate values such as 200x200 as long as the total pixel count is the same?
- penagwin 7y ago
  > Never thought about this approach to leveraging RAM for cameras. This is normal for high refresh rate cameras. They store straight to ram then dump to the disk (SSD) after. Especially the higher end cameras, they literally can't store data fast enough to go anywhere else.
- makomk 7y ago
  I think you're stuck with the oddball aspect ratio. Part of the CMOS readout process is done an entire line at a time, and there just isn't a way to increase the speed this much without dropping most of the lines. Actual high-speed cameras have similar limitations at the higher speeds.
- 78907s98gf 7y ago
  >Can anybody chime in regarding the resolution limit (640x64) Light exposure on the sensor, the faster the frame rate, the darker the image as its a fixed aperture. Photographic buffs could explain it better, but my take is the faster the exposure the bigger the aperture that is needed to let more light in to make the photo look well lit. I don't know if parts of the sensor also work faster due to its circuit design but I would imagine this is also a factor just like its possible to read more data form the outside of a spin disk than the tracks close to the centre.
- HermannSW 7y ago
  Deal is "halve vertical resolution and get roughly double framerate", see here for more information: https://news.ycombinator.com/item?id=20633504 https://news.ycombinator.com/item?id=20633504
vortico 7y ago
Are there any cameras at a higher price with better stats, but still lower than $2k for a used Chronos 1.4GP/s? I wouldn't mind buying a $1k camera to get 720p 1000+ fps.
- makomk 7y ago
  There are a handful of flagship smartphones that cost under a grand and technically support 960fps at 720p - the Samsung Galaxy S9, Note 9 and S10, along with Sony's Xperia XZ series. They come with a bunch of caveats including very limited recording time of less than half a second. A number of other smartphones advertise 960fps but fake it using software interpolation. The Chronos 1.4 is very aggressively priced for what it offers, even if it does cost several grand.
tinix 7y ago
if you put the sensor on a really fast step motor, you can pivot the sensor a tiny bit in two dimensions, to get way more resolution out of it, at a loss of framerate, and you'll need to do some post processing. something like this: https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-om-d-e-m1-mark-ii/ https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...
- Avery3R 7y ago
  The scrolljacking on that site makes it unusable
  - tripzilch 7y ago
    Literally unusable. I found that reader mode works fine.
- saidajigumi 7y ago
  Pixel-shifting is prone to some really terrible artifacting, particularly in shooting situations with movement (incl. any in-frame motion due to wind, etc.) Interestingly, whatever computational photography magic Panasonic is doing in the S1/S1R with their "multi-shot hi-res mode" appears to be far more robust in real-world shooting situations, even handheld. For those who care more about this, Lloyd Chambers has written extensive discussion and analysis of these systems at https://diglloyd.com/ https://diglloyd.com/ (fair warning: access to the detailed analysis is part of the paid site, a few high-level notes are in the public blog).
londons_explore 7y ago
The demo videos don't have a good black level. Is this processing method perhaps not using a correct dark image, or not correctly subtracting off the shaded calibration rows on the sensor? If thats the case, it's probably screwing with the gamma conversion, making it look even worse.
- robertelder 7y ago
  Nope, I didn't do any black level/gamma adjustments. I just used the first values that would produce what looked like a fairly high-quality result with ffmpeg and dcraw. There is probably tons of low hanging fruit to improve these results from someone who really knows what the're doing in terms of image processing.
Noctem 7y ago
What was the lighting setup like for the demo videos? Did you do any processing to reduce flicker? I get a lot of flickering from CFL and incandescent bulbs in my 240 FPS videos, though I’ve had better results with some LEDs. Considering how much light I’ve needed for good videos at 240 FPS, I’d imagine the lighting requirements at 660-1000 are fairly immense.
- robertelder 7y ago
  Usually, I would try to wait until evening when the sunlight shines directly into my apartment. Firlming under a 60 watt lamp was still quite dark. I didn't do any experiments with post-processing to reduce flicker, but I know there some advanced video editors do provide such features.
- HermannSW 7y ago
  Easiest way to avoid flicker is to light the scene with 1000lm or 5000lm cheap leds with led driver (they power the led with 38V/1.5A DC for 5000lm). Total cost of led+driver is less than 9$ with free shipping on aliexpress.
robertelder 7y ago
Hi Everyone, I do have it already linked in the article, but I should point out that the work I did is based upon the work of (at least) several other key individuals: A lot of the low-level sensor interaction code is found in: https://github.com/Hermann-SW/fork-raspiraw https://github.com/Hermann-SW/fork-raspiraw Also, check out his web site with lots of other high speed camera hacking stuff: https://stamm-wilbrandt.de/en/Raspberry_camera.html https://stamm-wilbrandt.de/en/Raspberry_camera.html The original fork of raspiraw is (I think) found here: https://github.com/6by9/raspiraw https://github.com/6by9/raspiraw And the code for converting RAW images into standard formats uses dcraw: https://en.wikipedia.org/wiki/Dcraw https://en.wikipedia.org/wiki/Dcraw I didn't personally write much code to put this together. I would say that my contribution here was mainly to pull together some impressive work done by others that wasn't getting much attention and market it well. Hopefully some of you get ideas and push this even further.
- geerlingguy 7y ago
  Thanks for posting your work! I’d like to see if I can get much more (if any) out of a Pi 4 which has more memory bandwidth and up to 2-4x more RAM than the 3 B+ it seems you tested with! Have you tried a 4 yet?
  - robertelder 7y ago
    I haven't yet. It was actually quite a bit if work to get these videos recorded and document/test the process. I suspect there is tons of low-hanging fruit left in terms of improvements that could be made. Hopefully the steps I documented will be reproducible enough that others can make improvements and create even more impressive demos.
- rumanator 7y ago
  Thank you for putting together such an inspiring and interesting project!
- HermannSW 7y ago
  Thanks Robert for the youtube videos and the blog, hopefully many more people will play with high framerate video capturing by that. But 660fps is not the end for v1 camera, I added section "kFPS videos from kEPS multiple exposure frames" to my github "v1 camera global external shutter" repo: https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter#kfps-videos-from-keps-multiple-exposure-frames https://github.com/Hermann-SW/Raspberry_v1_camera_global_ext... In that section a 3000fps and a 9000fps animation are shown (and the tools needed to create them)! The repo introduction now contains comparison of 20$ diy highspeed flashlight with commercially available one (and with the very costly one you cannot produce kEPS multiple exposures and kFPS framerate videos): https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter#user-content-diycomparison https://github.com/Hermann-SW/Raspberry_v1_camera_global_ext...
sytelus 7y ago
Very cool. Can this camera also be hacked for amplified sensor reading for night vision? Infrared or UV photography? Long exposure photography?
- HermannSW 7y ago
  Yes, there are NoIR versions of v1 camera, they just miss the infrared filter normal cameras have (around 9$ on aliexpress.com). And there are 12$ versions of v1 camera with (hardware) IR-cut filter, depending on photo sensor on camera IR filter is enabled at day, and removed at night.
  - HermannSW 7y ago
    I was wrong, the 9$ are for NoIR camera with M12 lens mount and lens. I just ordered 4 new NoIR cameras for 2.84$(!) including shipping! https://www.aliexpress.com/item/32946093276.html https://www.aliexpress.com/item/32946093276.html
Amendeson530 7y ago
Um, could something similar to this be done with better hardware to achieve better results at a slightly higher price point? 1000 fps on a 560p or 1080p res for a few hundred dollas would be huge... compared to Red/Phantom/name extremely expensive cameras...
person_of_color 7y ago
Could you stream the frames over 10G LAN to record longer videos?
- HermannSW 7y ago
  There is no Pi with 10G ethernet. New Pi4 has only Gigabit ethernet, Pi 3B+ has Gigabit Ethernet over USB 2.0 (maximum throughput 300 Mbps). The idea is interesting, did some calculation. For v1 camera capturing 640xH frames can be done at framerate "42755/H+13". Lets forget about the 13. raspiraw transfers raw10 Bayer data, that is 640xHx1.25 bytes, with maximal framerate that gives 42755x640x1.25=34204000 bytes/s=261Mbit/s(!). So you definitely need Gigabit ethernet of the Pi4, not sure whether Pi4 can actually stream out 261Mbit without losing stuff though ... Just asked on Raspberry networking forum (261Mbps for v1, 464Mbps for v2): https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=248254 https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=2482...
  - person_of_color 7y ago
    So it's feasible for Pi4?
    - HermannSW 7y ago
      No answer on the networking forum yet. I cannot try because I don't have a Pi4 yet. It seems possible. An easy test would be to test whether this command will work on Pi4: raspividyuv -md 7 -w 640 -h 480 -t 0 -fps 90 -o - | nc someIp somePort and on someIp server service listeing on somePort just storing the data received. Because yuv has 12bits/pixel which is a bit more than the 10bit/pixel of raw Bayer, the command produces a 316Mbps stream of data for v1 camera (732Mbps for v2 camera when recording with "-fps 180"). For testing the service on someIp can be as easy as: nc -l somePort > foobar
      - rasz 7y ago
        You should be fine with RPi3B+ internal gigabit NIC https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=208512&start=50#p1293094 https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=2085... pi@raspberrypi:~ $ sudo iperf3 -c XXX.XXX.X.XXX [ 4] 0.00-1.00 sec 37.9 MBytes 318 Mbits/sec 0 257 KBytes

445 points by robertelder 7y ago

This is amazing. These framerates are 1/10th of what a cursory google search says I can get out of a top of the line high-speed camera, but for probably 1/100th the price. You could make so many awesome instructional videos with a tool like this.

ortusdux 7y ago
The chronos 1.4 I have will do 640x96 @ 21,649fps for $3k. So that's 20 times faster for 50 times the price; pretty close to your estimate. The chronos does have a few other amenities that make it worth the price.
- ekr 7y ago
  That's 500 times the price (unless you're taking into account the raspberry pi itself).
  - hoseja 7y ago
    ... which you should
- kragen 7y ago
  Is it made in the US? Is it export-controlled? Here in Argentina getting export-controlled gear is a pain.
  - ortusdux 7y ago
    They are made in and shipped from Canada. I doubt they are export-controlled, but I would email them. They were quite responsive to most of my inquires. Just don't ask when they might implement features. I'm still waiting on some stuff that are listed on the 2 year old spec sheet as "*features (that) are fully supported in the camera's hardware, but are not yet supported in software. They will be added in a free software update after the camera's initial release."
bufferoverflow 7y ago
You can shoot 960fps at 720p/1080p with some smartphones these days. If you decide to go that route, make sure to get one that doesn't interpolate the frames, but does true 960fps.
- CobrastanJorji 7y ago
  Could you suggest such a phone because that sounds wicked awesome to experiment with.
  - CarVac 7y ago
    The Galaxy S10 series (including the S10e).
  - deleted 7y ago
    [deleted]
ktpsns 7y ago
Which brings me to this Gedankenexperiment: If you buy/build 10 of the raspberry pi based high speed cameras and run them in parallel and could make sure each camera sees the same (we don't consider details here for the time being), is it possible to merge the videos in order to reach the frame rate of a "top of the line high-speed camera"? I could imagine two scenarios: 1. Start recording with time delay. Probably requires excellent clock synchronisation, a real time kernel, reliable/comparable hardware (gatter timings differ at small timescales). If this works, this would provide a high quality equidistand framerate. or 2. Start recording randomly, synchronize by time or with an video event, and merge the frames. This will result in a stochastic distribution of frame rates. The usefulness of such a movie depends on what it is supposed to be used for (thinking of scientific usage). Will probably not give nice slow-mo movies. But here comes the catch: With interpolation, the shortest reachable frame rate dictates the quality of this "superposition camera". How many cheap cameras do we need to archieve, on average, the top line high speed camera? I leave the questions open for friends of Gedankenexperiments ;-)
- rasz 7y ago
  pee cameras are clocked externally(EMI made them include 25MHz clock gen on camera module itself), so in theory you could connect them all to a single clock distribution source with appropriately shifted phases
  - sansnomme 7y ago
    How do you disable onboard clock or is it automatic?
    - rasz 7y ago
      its already external, on the back of each camera
      - sansnomme 7y ago
        But it's only selectively enabled no?
HermannSW 7y ago
You can get Raspberry v1 camera clone now for less than 4$(!) with free shipping on aliexpress: https://www.aliexpress.com/wholesale?SearchText=raspberry+5MP&SortType=price_asc&g=y&isFreeShip=y&switch_new_app=y https://www.aliexpress.com/wholesale?SearchText=raspberry+5M... And unlike v2 camera, the cheap v1 camera allows (besides high framerate capturing) for global external shutter capturing as well(!). I did capture an in flight 109m/s airgun pellet 10 times in a single multiple exposure frame! https://stamm-wilbrandt.de/en/Raspberry_camera.html#global_external_shutter https://stamm-wilbrandt.de/en/Raspberry_camera.html#global_e...

yeahdef 7y ago

very impressive.

bavell 7y ago

Resolution: 640x64

penagwin 7y ago
I thought that was pretty low - but it certainly enough for some impressive shots! [0] Video from article: https://www.youtube.com/watch?v=-gMy8k4nHtw https://www.youtube.com/watch?v=-gMy8k4nHtw
msie 7y ago
Then get 10 Pi's to get 640x640?
- azhenley 7y ago
  6400x640?
  - jefft255 7y ago
    Nope, that’s a 100 pies...
  - bduerst 7y ago
    You're stacking pis and sensors side-by-side, not multiplying them.

alvern 7y ago

1107 fps on the V2 IMX219 camera. I wonder how fast or how large of a frame it could do on a Pi4 or odroid (article mentions the 640x64 frame is limited by memory bandwidth).

rasz 7y ago
I think Author meant mipi CSI-2 2 lane bandwidth limit = 1.1 Gbps, ~somewhere around 120MB/s
HermannSW 7y ago
You can do 640x64@665fps with v1 camera, and 640x128@667fps with v2 camera: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212518&p=1310445#p1320034 https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=2125... In tools directory you can use *_s tools to get double FOV vertically while keeping the framerate. So you can get 640x128_s@665fps/640x256_s@667fps for v1/v2 camera. There real limit for v2 camera is 1007fps, with can be achieved with 640x75 or 640x150_s tools. You can reduce vertical resolution below 75, but you will not get more than 1007fps (if you find out how to raise that limit, please shout! Deal for raspiraw high framerate capturing basically is "halve vertical resolution and get double framerate").

canada_dry 7y ago

> processor registers on the image sensor are set so that raw data is sent continuously (to be processed later) Very slick. Make me wonder what crazy high-speed imaging hacks might be possible with a top-of-the-line sensor used in iPhone/Pixel phones.

Gcfcow 7y ago
The Samsung Galaxy S9 & S10 can record short bursts of 960fps video at 720p out of the box. I'd imagine a hack could extract similar numbers from the pixel and iPhone.

novaRom 7y ago

Everything is doable if you have detailed description of the sensor. Do you know any sensor to get a better resolution with comparable frame rate and detailed specs? Can RAM limits be improved with a simple compression?

penagwin 7y ago
He did his testing on a raspberry pi 3 which means only 1GB of LPDDR2. I'd love to see how the RAM limits improve with 4GB LPDDR4, especially with usb 3.0 and real ethernet....
- novaRom 7y ago
  Camera interface is not via USB on Raspberry Pi. There's a lot of redundancy in high frame rate, so the best way to improve memory footprint is compression.
  - penagwin 7y ago
    My comment on the USB part is more asking the increased bandwidth make it possible to store it on a disk/network.
    - londons_explore 7y ago
      Most new devices have realised that storing RAW image data in realtime is impractical - it can be gigabytes per second! Instead, one needs to use a GPU, DSP, or other dedicated hardware to process/compress the data into a common video or image format.
      - 420codebro 7y ago
        I hope I don't sound like a retard asking this: where does one read more about detailed [hardware] image processing?
      - codetrotter 7y ago

jermaustin1 7y ago

I read through, and I didn't find a reason that the resolution is 640x64@660 other than that is what was input into the raspiraw program. Could this record at a higher resolution?

sp332 7y ago
It's probably bottlenecked by how fast the sensor data can be read.
- djsumdog 7y ago
  Yea they mention in the video, but don't mention what happens with a higher aspect ratio. I suspect it will cause artifacts, like part of one frame appearing on the previous or tearing or other distortions. Remember these sensors read a single row at a time. There's a Smarter Every Day video where they show how cellphones get those weird effects when recording plane propellers due to the way they scan images.
  - HermannSW 7y ago
    Yes, for objects moving with high speed vertically, rolling shutter capturing produces weird frames. But v1 camera can do "global external shutter" as well. See examples on "extreme rolling/global external" shutter for propellers here: https://github.com/Hermann-SW/Raspberry_camera_gallery/blob/master/propeller/README.md https://github.com/Hermann-SW/Raspberry_camera_gallery/blob/...
bufferoverflow 7y ago
Limited by the chip that reads the data off the sensor - ASIC of FPGA. Or maybe by the sensor design itself.
pfranz 7y ago
I haven't read through everything yet, but in the examples video[1] it's stated that they can only read a portion of the sensor to keep a high frame rate and that the camera returns one row at a time. A square image requires reading more rows and takes more time. [1] https://youtu.be/-gMy8k4nHtw?t=154 https://youtu.be/-gMy8k4nHtw?t=154

an4rchy 7y ago

Wow. Never thought about this approach to leveraging RAM for cameras. Awesome guide! Can anybody chime in regarding the resolution limit (640x64) -- If this is for memory reasons, could we pick alternate values such as 200x200 as long as the total pixel count is the same?

penagwin 7y ago
> Never thought about this approach to leveraging RAM for cameras. This is normal for high refresh rate cameras. They store straight to ram then dump to the disk (SSD) after. Especially the higher end cameras, they literally can't store data fast enough to go anywhere else.
makomk 7y ago
I think you're stuck with the oddball aspect ratio. Part of the CMOS readout process is done an entire line at a time, and there just isn't a way to increase the speed this much without dropping most of the lines. Actual high-speed cameras have similar limitations at the higher speeds.
78907s98gf 7y ago
>Can anybody chime in regarding the resolution limit (640x64) Light exposure on the sensor, the faster the frame rate, the darker the image as its a fixed aperture. Photographic buffs could explain it better, but my take is the faster the exposure the bigger the aperture that is needed to let more light in to make the photo look well lit. I don't know if parts of the sensor also work faster due to its circuit design but I would imagine this is also a factor just like its possible to read more data form the outside of a spin disk than the tracks close to the centre.
HermannSW 7y ago
Deal is "halve vertical resolution and get roughly double framerate", see here for more information: https://news.ycombinator.com/item?id=20633504 https://news.ycombinator.com/item?id=20633504

vortico 7y ago

Are there any cameras at a higher price with better stats, but still lower than $2k for a used Chronos 1.4GP/s? I wouldn't mind buying a $1k camera to get 720p 1000+ fps.

makomk 7y ago
There are a handful of flagship smartphones that cost under a grand and technically support 960fps at 720p - the Samsung Galaxy S9, Note 9 and S10, along with Sony's Xperia XZ series. They come with a bunch of caveats including very limited recording time of less than half a second. A number of other smartphones advertise 960fps but fake it using software interpolation. The Chronos 1.4 is very aggressively priced for what it offers, even if it does cost several grand.

tinix 7y ago

if you put the sensor on a really fast step motor, you can pivot the sensor a tiny bit in two dimensions, to get way more resolution out of it, at a loss of framerate, and you'll need to do some post processing. something like this: https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-om-d-e-m1-mark-ii/ https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...

Avery3R 7y ago
The scrolljacking on that site makes it unusable
- tripzilch 7y ago
  Literally unusable. I found that reader mode works fine.
saidajigumi 7y ago
Pixel-shifting is prone to some really terrible artifacting, particularly in shooting situations with movement (incl. any in-frame motion due to wind, etc.) Interestingly, whatever computational photography magic Panasonic is doing in the S1/S1R with their "multi-shot hi-res mode" appears to be far more robust in real-world shooting situations, even handheld. For those who care more about this, Lloyd Chambers has written extensive discussion and analysis of these systems at https://diglloyd.com/ https://diglloyd.com/ (fair warning: access to the detailed analysis is part of the paid site, a few high-level notes are in the public blog).

londons_explore 7y ago

The demo videos don't have a good black level. Is this processing method perhaps not using a correct dark image, or not correctly subtracting off the shaded calibration rows on the sensor? If thats the case, it's probably screwing with the gamma conversion, making it look even worse.

robertelder 7y ago
Nope, I didn't do any black level/gamma adjustments. I just used the first values that would produce what looked like a fairly high-quality result with ffmpeg and dcraw. There is probably tons of low hanging fruit to improve these results from someone who really knows what the're doing in terms of image processing.

Noctem 7y ago

What was the lighting setup like for the demo videos? Did you do any processing to reduce flicker? I get a lot of flickering from CFL and incandescent bulbs in my 240 FPS videos, though I’ve had better results with some LEDs. Considering how much light I’ve needed for good videos at 240 FPS, I’d imagine the lighting requirements at 660-1000 are fairly immense.

robertelder 7y ago
Usually, I would try to wait until evening when the sunlight shines directly into my apartment. Firlming under a 60 watt lamp was still quite dark. I didn't do any experiments with post-processing to reduce flicker, but I know there some advanced video editors do provide such features.
HermannSW 7y ago
Easiest way to avoid flicker is to light the scene with 1000lm or 5000lm cheap leds with led driver (they power the led with 38V/1.5A DC for 5000lm). Total cost of led+driver is less than 9$ with free shipping on aliexpress.

robertelder 7y ago

Hi Everyone, I do have it already linked in the article, but I should point out that the work I did is based upon the work of (at least) several other key individuals: A lot of the low-level sensor interaction code is found in: https://github.com/Hermann-SW/fork-raspiraw https://github.com/Hermann-SW/fork-raspiraw Also, check out his web site with lots of other high speed camera hacking stuff: https://stamm-wilbrandt.de/en/Raspberry_camera.html https://stamm-wilbrandt.de/en/Raspberry_camera.html The original fork of raspiraw is (I think) found here: https://github.com/6by9/raspiraw https://github.com/6by9/raspiraw And the code for converting RAW images into standard formats uses dcraw: https://en.wikipedia.org/wiki/Dcraw https://en.wikipedia.org/wiki/Dcraw I didn't personally write much code to put this together. I would say that my contribution here was mainly to pull together some impressive work done by others that wasn't getting much attention and market it well. Hopefully some of you get ideas and push this even further.

geerlingguy 7y ago
Thanks for posting your work! I’d like to see if I can get much more (if any) out of a Pi 4 which has more memory bandwidth and up to 2-4x more RAM than the 3 B+ it seems you tested with! Have you tried a 4 yet?
- robertelder 7y ago
  I haven't yet. It was actually quite a bit if work to get these videos recorded and document/test the process. I suspect there is tons of low-hanging fruit left in terms of improvements that could be made. Hopefully the steps I documented will be reproducible enough that others can make improvements and create even more impressive demos.
rumanator 7y ago
Thank you for putting together such an inspiring and interesting project!
HermannSW 7y ago
Thanks Robert for the youtube videos and the blog, hopefully many more people will play with high framerate video capturing by that. But 660fps is not the end for v1 camera, I added section "kFPS videos from kEPS multiple exposure frames" to my github "v1 camera global external shutter" repo: https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter#kfps-videos-from-keps-multiple-exposure-frames https://github.com/Hermann-SW/Raspberry_v1_camera_global_ext... In that section a 3000fps and a 9000fps animation are shown (and the tools needed to create them)! The repo introduction now contains comparison of 20$ diy highspeed flashlight with commercially available one (and with the very costly one you cannot produce kEPS multiple exposures and kFPS framerate videos): https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter#user-content-diycomparison https://github.com/Hermann-SW/Raspberry_v1_camera_global_ext...

sytelus 7y ago

Very cool. Can this camera also be hacked for amplified sensor reading for night vision? Infrared or UV photography? Long exposure photography?

HermannSW 7y ago
Yes, there are NoIR versions of v1 camera, they just miss the infrared filter normal cameras have (around 9$ on aliexpress.com). And there are 12$ versions of v1 camera with (hardware) IR-cut filter, depending on photo sensor on camera IR filter is enabled at day, and removed at night.
- HermannSW 7y ago
  I was wrong, the 9$ are for NoIR camera with M12 lens mount and lens. I just ordered 4 new NoIR cameras for 2.84$(!) including shipping! https://www.aliexpress.com/item/32946093276.html https://www.aliexpress.com/item/32946093276.html

Amendeson530 7y ago

Um, could something similar to this be done with better hardware to achieve better results at a slightly higher price point? 1000 fps on a 560p or 1080p res for a few hundred dollas would be huge... compared to Red/Phantom/name extremely expensive cameras...

person_of_color 7y ago

Could you stream the frames over 10G LAN to record longer videos?

HermannSW 7y ago
There is no Pi with 10G ethernet. New Pi4 has only Gigabit ethernet, Pi 3B+ has Gigabit Ethernet over USB 2.0 (maximum throughput 300 Mbps). The idea is interesting, did some calculation. For v1 camera capturing 640xH frames can be done at framerate "42755/H+13". Lets forget about the 13. raspiraw transfers raw10 Bayer data, that is 640xHx1.25 bytes, with maximal framerate that gives 42755x640x1.25=34204000 bytes/s=261Mbit/s(!). So you definitely need Gigabit ethernet of the Pi4, not sure whether Pi4 can actually stream out 261Mbit without losing stuff though ... Just asked on Raspberry networking forum (261Mbps for v1, 464Mbps for v2): https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=248254 https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=2482...
- person_of_color 7y ago
  So it's feasible for Pi4?
  - HermannSW 7y ago
    No answer on the networking forum yet. I cannot try because I don't have a Pi4 yet. It seems possible. An easy test would be to test whether this command will work on Pi4: raspividyuv -md 7 -w 640 -h 480 -t 0 -fps 90 -o - | nc someIp somePort and on someIp server service listeing on somePort just storing the data received. Because yuv has 12bits/pixel which is a bit more than the 10bit/pixel of raw Bayer, the command produces a 316Mbps stream of data for v1 camera (732Mbps for v2 camera when recording with "-fps 180"). For testing the service on someIp can be as easy as: nc -l somePort > foobar
    - rasz 7y ago
      You should be fine with RPi3B+ internal gigabit NIC https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=208512&start=50#p1293094 https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=2085... pi@raspberrypi:~ $ sudo iperf3 -c XXX.XXX.X.XXX [ 4] 0.00-1.00 sec 37.9 MBytes 318 Mbits/sec 0 257 KBytes