5 ms·
This is no simple thing to do. The biggest issue is that "raw" Linux is not an RTOS. When the photographer presses the shutter, you have a few milliseconds to
by ChrisMarshallNY 7y ago
This is no simple thing to do.
The biggest issue is that "raw" Linux is not an RTOS.
When the photographer presses the shutter, you have a few milliseconds to capture the image. You can't have the OS decide to prioritize receiving a tweet before getting to the shutter release.
It's even more intense with video, and most still cameras are quickly morphing into high-quality video cameras.
Also, image processing is REAL intense; especially with Computational Photography. It can suck down your battery in seconds.
There's a few ways to deal with it. You could use something like Xenomai or RTLinux, or establish some kind of SoC for the realtime-sensitive processing (what most phone manufacturers probably do -Apple sure does).
It's a non-trivial task.
- amelius 7y agoCurious, how does V4L handle these things?
- ChrisMarshallNY 7y agoNot sure. Never played with it. My experience is with a custom Android build, on some standard Snapdragon stuff (won't get any more detailed than that). I have played with ffmpeg, which is not suitable for mobile, unless heavily modified. The VideoLAN folks seem to have gotten VLC working for mobile, but it's a real pig.
- joshvm 7y agoIt gets a bit messy. In order to capture an image, most image sensors are triggered. Some sensors will operate in "free-run" mode, but in reality something is sending a capture signal to the camera. In sensors where you have very low level control over things, usually what that does is clear the pixels (dump the accumulated charge) and start exposing. Then, when the exposure is complete, the camera makes the data available for read-out (for a phone sensor analogue-digital conversion is likely done in circuitry on the sensor, not in the phone). That trigger is the important bit. Using triggering you can synchronise multiple cameras (e.g. the sensors in multi-camera phones are almost certainly co-triggered to avoid ghosting), illumination units, etc. In fact many camera chips can also be set to output flash signals when they're exposing, this eliminates latency in sending separate "capture" and "flash" signals. When I've deployed machine vision systems, often we would use an external system to trigger everything (e.g. a microcontroller). Interfacing hardware and software triggering is a nightmare. Then you've got whatever low level interface is actually communicating to the camera. On a phone, that's probably MIPI/CSI-2. Modern SoCs have dedicated (hardware) camera interfaces (the ISP) that are designed specifically to load images from camera sensors and do some minimal processing. You need kernel-level drivers for whatever your SoC is. These drivers will (in theory) let you set up arbitrary image sensors with whatever resolution/timing they require. But the key is you need two drivers: one to control the ISP and another to send commands to the camera sensor. A lot of the time (cough Broadcom) you only get a blob for the ISP and you almost never get full datasheets for image sensors without being a big customer. Rockchip have an open source driver for their ISP: http://opensource.rock-chips.com/wiki_Rockchip-isp1 http://opensource.rock-chips.com/wiki_Rockchip-isp1 See Rockchip again for an open-source example of an actual sensor driver. https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/i2c/imx219.c https://github.com/rockchip-linux/kernel/blob/release-4.4/dr... Want to make an educated guess why libcamera supports the [Rockchip] RK3399? They make it comparatively easy. In the end V4L acts as a mediator between this low level interface and other things which want to talk to the camera. Rockchip have provided what amounts to a plugin for V4L. Then on top of V4L you have libraries like OpenCV which give you nice interfaces like `imread` to capture images and do stuff with them. To answer your question you probably need to be more specific. The performance of what V4L does for a specific ISP/sensor combination very much depends on those underlying drivers. If you can afford it, a safe strategy is to leave the sensor in free-running mode in a separate thread that's dedicated to capturing images. Then in your application you pick off the most recent frame and do stuff with it in parallel. For a 30fps camera you therefore have 30ms or so to do your stuff before the next frame comes in.
- ChrisMarshallNY 7y agoThe other part of this, is that the entire image processing/video arena is a swamp of patents. They are quite enforceable ones, too. Video has been around quite a while, and used to be the domain of big fish corporations that could easily afford to file patents.
- emmelaich 7y agoLinux has real time scheduling available though.
- dingaling 7y ago> you have a few milliseconds to capture the image Most retail cameras are quite laggy. Even a top-spec 'pro' Canon 1DX II costing $6000 has a shutter response lag of 36ms at best ( and 55ms normally )
- rcoveson 7y agoIs that including some mechanical latency? If that’s all software/electrical latency it’s unbelievably bad.
- maxnoe 7y agoThe mirror has to move out of the way and the aperture must close to the chosen value.
- franga2000 7y agoAny idea how mirrorless cameras compare in this regard?
- ChrisMarshallNY 7y agoMirrorless is a challenge. You need to have something like <50ms frame lag, generally. I know there's some studies out there about this. It's been a while since I've done this stuff, but there's an "uncanny valley" of lag that needs to be avoided. Also, you need to present an image that is useful to the photographer. For example, if you are applying vignette correction, that can be a processor-intensive step, and can be difficult to apply at 24 frames per second on a full HD image (the sensor may be returning considerably higher than that, but the camera back may just have a 1080p display). If you are doing things like image enhancement/CP (for example smart HDR, selective enhancement, or stochastic NR), then how will they be displayed? Do you show a grainy image in the preview, even though the final will look much better? What about color space conversion? The display is likely to have a limited gamut, but the image may be saved at full Adobe, and you want to have perceptual correction, as opposed to clipped. This is an exciting time to be looking at this stuff, but the challenges are real.
- simias 7y agoI'm not sure I understand what you mean. I've done my share of video and photo programming on Linux (in particular I've written non-trivial v4l2 drivers for use in things like car rear view cameras, where obviously low latency is absolutely critical and you have to resort to tricks like processing the image by slice before you've read the full sensor) but I'm not sure I see how that's relevant for a "point and shoot" type camera. For one thing I don't think any reasonable camera would have the CPU "bitbang" the reading of the sensor's pixel, it will be read and buffered in hardware and set to the system RAM through DMA. We're talking about hundreds of megabytes of raw data per picture here, it's not something you want to memcpy(), especially on an embedded processor. If receiving a tweet is enough to make you miss some pixels then you haven't prioritized your RAM controller correctly or your system is severely overloaded already. The shutter release is independent of course, but it's not something where a few microseconds are going to change anything, especially for a mechanical shutter where you need to move physical parts. If your CPU stalls for 1ms processing a tweet then, again, you have bigger problems. You don't need an RTOS to control a camera shutter. So that leaves us with the problem of latency but even there I fail to see the problem. My girlfriend is a professional photographer and when she shoots she can either get the preview on the camera's screen (but that involves switching to the preview which already usually takes a pair of seconds) or getting the image on a PC connected through USB, which also takes a significant amount of time. Adding or removing 100ms of latency won't change a lot in practice as far as I can tell.
- ChrisMarshallNY 7y agoThe "receiving a tweet" thing was really just a strawman. It's more likely to be issues of OS maintenance that could cause the problems. I've had some really relevant experience, here, myself. I won't go into detail (N of the D-A). > it will be read and buffered in hardware and set to the system RAM through DMA. Moving memory, even on the fastest processor, is always a cost. This is especially true, when you look at the pixel count of most cameras (not just DSLRs), these days. Also, you don't just get the memory at once in a big multi-megabyte dump. The sensor needs to be polled. Each cell is queried for its charge state in a linear fashion. There may be ways to access sections in parallel, but I haven't encountered it (It would make sense -I have tended to be on the software side of things, leaving the hardware to the electrical engineering wizards). This is usually dealt with by an ASIC connected directly to the sensor. If you want to have a "software-only" camera (as opposed to one with a dedicated SoC), then you are looking at some really heavy-duty challenges. The idea of a "software-only" camera, is to completely do away with the dedicated SoC, or replace it with one that can be re-flashed. Not so simple. Computational Photography is something that is necessary for consumer-grade cameras, and is increasingly becoming relevant to pros, as well. It takes some real horsepower. The newer iPhones have awesome cameras. I suspect that some of the higher-end Androids are just as good, or better. If so, I will bet that you'll find a custom ASIC in the woodpile. Apple has custom ASICs for image handling, as well as CP. They have an ML chip built into their phones. I am confident that these challenges will be overcome (there's money to be made, and that's always a great driver), but the challenges are real. There are some very, very smart folks that have beat their brains out on this.