15 ms·

Libcamera – A complex camera support library for Linux, Android, and ChromeOS

inetknght 7y ago
Interesting. Does it work with Raspberry Pis and the CSI cameras? What about my Logitech USB camera? Would it support receiving a picture from eg a flatbed scanner over a parallel port?
- newnewpdro 7y ago
  I'd expect it to be complementary to the existing libraries for scanning/cameras, not mutually exclusive. So something like sane or libgphoto would presumably link in libcamera to access those "complicated" devices.
- omtinez 7y ago
  > Does it work with Raspberry Pis and the CSI cameras? What about my Logitech USB camera? I don't know details about the Raspberry Pi cameras, but if it's a UVC camera (quite likely) then it should be supported by this library according to the documentation. A USB Logitech camera should also be supported. The gotcha is that unless those cameras expose advanced functionality (3A, multiple stream support, etc) then you don't get any benefit over using the standard V4L drivers. > Would it support receiving a picture from eg a flatbed scanner over a parallel port? I don't think scanners identify themselves as camera devices, but someone can correct me if I'm wrong.
  - mpol 7y ago
    For flatbed scanners you would use something like SANE or Vuescan. Most devices have a different driver.
vzaliva 7y ago
I was not been able to find list of supported cameras. Could anybody please point to it?
- CharlesW 7y ago
  Found in a recent presentation by Jacopo Mondi of the libcamera team (PDF): https://static.sched.com/hosted_files/osseu19/21/libcamera.pdf https://static.sched.com/hosted_files/osseu19/21/libcamera.p... - Intel IPU3 - Rockchip RK3399 In progress: - RaspberryPi 3 and 4 - UVC cameras - VIMC test driver
teleclimber 7y ago
If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs. Given how hard other manufacturers try and fail to make equally good cameras, it's going to be a steep climb for the open source community too. So far I am not seeing much work being done in that space. Maybe this is a start? I want an open non-spying phone but it's going to be hard to let go of the amazing camera in my pocket now (Pixel 3a).
- catalogia 7y ago
  > "If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs." Couldn't they just find another niche? I think relatively few consumers know anything about photography. Even a proper professional DSLR would be useless to me because I know next to nothing about photography and tools don't make the artist. No picture I've ever taken has any artistic merit or technical merit. Of course, good camera software is a good thing and for the segment of society that's enthused about photography it's going to be essential. I'm just not convinced it's the secret sauce to mass appeal. Edit for response: > Well, okay, what problem would you rather see them tackle? I don't have such a preference. I expect volunteer developers will work on matters they find personally compelling. Those that find photography software compelling should pursue their passion.
  - jonas21 7y ago
    > I think relatively few consumers know anything about photography. That's precisely why you need good image processing software that works automatically. Nearly everybody wants to take great photos with their phone, but few people know how to configure everything themselves, and even those who do probably don't want to fiddle with every shot. > Couldn't they just find another niche? Well, okay, what problem would you rather see them tackle?
  - teleclimber 7y ago
    The reason for a good camera is not to take artistic photography (although you can do that too and many do). It's because people use that camera to record memories that they cherish. Just ask any new parent what happens to their phone's camera roll once number 1 pops out. Or ask young people to show you where they went on their last vacation: many times it'll be a bunch of cool shots taken on a phone. These memories are priceless and having to record them with a device that is open but has the photo quality of a 5 year old device is not an attractive trade-off.
catern 7y ago
>...collaboration with the industry to develop a camera stack... protecting vendor core IP. Can't say I have any interest in protecting "vendor IP" in their crappy camera drivers. Let vendors open their blobs! Closed source camera drivers will just reduce quality and increase bugginess.
- Jonnax 7y ago
  What's the point of this all or nothing attitude? This looks like a move to getting better support. The alternative is getting nothing.
  - matheusmoreira 7y ago
    The point is to get high quality software that actually belongs to us all. Accepting closed source drivers means accepting the Windows model where companies release buggy drivers for one or two combinations of processor architecture and operating system version before ending all support. The company gets to open wash their reputation while still keeping us dependent on them for scraps of support. Giving us nothing at all would've been more respectful than this. Letting companies protect their intellectual property benefits absolutely nobody but them. They don't have our best interests in mind, only their own. The ideal solution is to reverse engineer their hardware's interface and produce a competing implementation which will no doubt be better than their minimum viable product. My laptop has a backlit keyboard with RGB LEDs and I managed to figure out how it works and make my own program to replace the manufacturer's. I'm so happy I don't have to boot into Windows just to use their slow software anymore. They don't even have to give us source code. Documentation would eliminate the need to actually reverse engineer the hardware. They could just release that data and all interested parties would start hacking. The problem would take care of itself at no cost to them.
  - shmerl 7y ago
    We don't need more blobs. Such kind of approach is not helping.
  - serf 7y ago
    > What's the point of this all or nothing attitude? the point is to try to enact change in corporate behavior with regards to consumer devices. >The alternative is getting nothing. goes both ways. If I don't buy a product that uses binary blobs all over the place, they don't sell the product to me. Supporting products that support DRM systems natively does a lot to support the underlying DRM system, too. It sets a native expectation of DRM among developers, fostering the use. And worst of all, I can't try to fix blobs without labor that is so intensive that it renders the very idea moot.
mschuster91 7y ago
Is this aimed for smartphones or for real cameras? I mean, the Sony Alpha series do run Linux, even with an (ancient) Android runtime...
omtinez 7y ago
I'm surprised to see such big emphasis on support for Android. Most camera modules come with their own drivers that already provide Android support. One benefit could be the licensing, but after a quick inspection it is unclear to me what license this library is under -- there is a licenses folder with 4 different licenses in addition to a developer agreement. The design seems to be heavily inspired by the Android camera API: per-frame configuration, 3A, multiple stream support, device enumeration, etc. > The HAL will implement internally features required by Android and missing from libcamera, such as JPEG encoding support. That is interesting, since most camera modules will have a hardware accelerated path to encode frames directly to JPEG. If it's done internally, it will be much slower than all other implementations I'm aware of.
ChrisMarshallNY 7y ago
This is no simple thing to do. The biggest issue is that "raw" Linux is not an RTOS. When the photographer presses the shutter, you have a few milliseconds to capture the image. You can't have the OS decide to prioritize receiving a tweet before getting to the shutter release. It's even more intense with video, and most still cameras are quickly morphing into high-quality video cameras. Also, image processing is REAL intense; especially with Computational Photography. It can suck down your battery in seconds. There's a few ways to deal with it. You could use something like Xenomai or RTLinux, or establish some kind of SoC for the realtime-sensitive processing (what most phone manufacturers probably do -Apple sure does). It's a non-trivial task.
- amelius 7y ago
  Curious, how does V4L handle these things?
  - ChrisMarshallNY 7y ago
    Not sure. Never played with it. My experience is with a custom Android build, on some standard Snapdragon stuff (won't get any more detailed than that). I have played with ffmpeg, which is not suitable for mobile, unless heavily modified. The VideoLAN folks seem to have gotten VLC working for mobile, but it's a real pig.
  - joshvm 7y ago
    It gets a bit messy. In order to capture an image, most image sensors are triggered. Some sensors will operate in "free-run" mode, but in reality something is sending a capture signal to the camera. In sensors where you have very low level control over things, usually what that does is clear the pixels (dump the accumulated charge) and start exposing. Then, when the exposure is complete, the camera makes the data available for read-out (for a phone sensor analogue-digital conversion is likely done in circuitry on the sensor, not in the phone). That trigger is the important bit. Using triggering you can synchronise multiple cameras (e.g. the sensors in multi-camera phones are almost certainly co-triggered to avoid ghosting), illumination units, etc. In fact many camera chips can also be set to output flash signals when they're exposing, this eliminates latency in sending separate "capture" and "flash" signals. When I've deployed machine vision systems, often we would use an external system to trigger everything (e.g. a microcontroller). Interfacing hardware and software triggering is a nightmare. Then you've got whatever low level interface is actually communicating to the camera. On a phone, that's probably MIPI/CSI-2. Modern SoCs have dedicated (hardware) camera interfaces (the ISP) that are designed specifically to load images from camera sensors and do some minimal processing. You need kernel-level drivers for whatever your SoC is. These drivers will (in theory) let you set up arbitrary image sensors with whatever resolution/timing they require. But the key is you need two drivers: one to control the ISP and another to send commands to the camera sensor. A lot of the time (cough Broadcom) you only get a blob for the ISP and you almost never get full datasheets for image sensors without being a big customer. Rockchip have an open source driver for their ISP: http://opensource.rock-chips.com/wiki_Rockchip-isp1 http://opensource.rock-chips.com/wiki_Rockchip-isp1 See Rockchip again for an open-source example of an actual sensor driver. https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/i2c/imx219.c https://github.com/rockchip-linux/kernel/blob/release-4.4/dr... Want to make an educated guess why libcamera supports the [Rockchip] RK3399? They make it comparatively easy. In the end V4L acts as a mediator between this low level interface and other things which want to talk to the camera. Rockchip have provided what amounts to a plugin for V4L. Then on top of V4L you have libraries like OpenCV which give you nice interfaces like `imread` to capture images and do stuff with them. To answer your question you probably need to be more specific. The performance of what V4L does for a specific ISP/sensor combination very much depends on those underlying drivers. If you can afford it, a safe strategy is to leave the sensor in free-running mode in a separate thread that's dedicated to capturing images. Then in your application you pick off the most recent frame and do stuff with it in parallel. For a 30fps camera you therefore have 30ms or so to do your stuff before the next frame comes in.
shmerl 7y ago
> the Linux media community has very recently started collaboration with the industry to develop a camera stack that will be open-source-friendly while still protecting vendor core IP What does that mean? Open source should be open source. Is it going to work on top of v4l or it's independent? https://en.wikipedia.org/wiki/Video4Linux https://en.wikipedia.org/wiki/Video4Linux
sandGorgon 7y ago
This is very interesting. How does this compare with CameraX for which Google has built an entire building with hundreds of cameras as a testlab ? https://www.extremetech.com/computing/301298-camerax-googles-new-weapon-in-the-photography-wars https://www.extremetech.com/computing/301298-camerax-googles... https://developer.android.com/training/camerax https://developer.android.com/training/camerax
- lovelearning 7y ago
  This sits much lower in the stack than CameraX. This is more of a HAL for cameras standardized across multiple OSes. http://libcamera.org/docs.html#android-camera-hal-v3-compatibility http://libcamera.org/docs.html#android-camera-hal-v3-compati...
me551ah 7y ago
I don't get who is going to be using this library. Most android developers would prefer to use native Java APIs instead of a C++ library ( with a JNI wrapper around it?) since it's easier. Linux phones barely have any market share so this library would be used mainly on computers where most of the cameras are USB anyway.

Interesting. Does it work with Raspberry Pis and the CSI cameras? What about my Logitech USB camera? Would it support receiving a picture from eg a flatbed scanner over a parallel port?

newnewpdro 7y ago
I'd expect it to be complementary to the existing libraries for scanning/cameras, not mutually exclusive. So something like sane or libgphoto would presumably link in libcamera to access those "complicated" devices.
omtinez 7y ago
> Does it work with Raspberry Pis and the CSI cameras? What about my Logitech USB camera? I don't know details about the Raspberry Pi cameras, but if it's a UVC camera (quite likely) then it should be supported by this library according to the documentation. A USB Logitech camera should also be supported. The gotcha is that unless those cameras expose advanced functionality (3A, multiple stream support, etc) then you don't get any benefit over using the standard V4L drivers. > Would it support receiving a picture from eg a flatbed scanner over a parallel port? I don't think scanners identify themselves as camera devices, but someone can correct me if I'm wrong.
- mpol 7y ago
  For flatbed scanners you would use something like SANE or Vuescan. Most devices have a different driver.

vzaliva 7y ago

I was not been able to find list of supported cameras. Could anybody please point to it?

CharlesW 7y ago
Found in a recent presentation by Jacopo Mondi of the libcamera team (PDF): https://static.sched.com/hosted_files/osseu19/21/libcamera.pdf https://static.sched.com/hosted_files/osseu19/21/libcamera.p... - Intel IPU3 - Rockchip RK3399 In progress: - RaspberryPi 3 and 4 - UVC cameras - VIMC test driver

teleclimber 7y ago

If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs. Given how hard other manufacturers try and fail to make equally good cameras, it's going to be a steep climb for the open source community too. So far I am not seeing much work being done in that space. Maybe this is a start? I want an open non-spying phone but it's going to be hard to let go of the amazing camera in my pocket now (Pixel 3a).

catalogia 7y ago
> "If linux phones are going to be a thing (pinephone, librem) they'll need great cameras and excellent image processing software t be even somewhat competitive with Google Pixels and iPhones and Samsungs." Couldn't they just find another niche? I think relatively few consumers know anything about photography. Even a proper professional DSLR would be useless to me because I know next to nothing about photography and tools don't make the artist. No picture I've ever taken has any artistic merit or technical merit. Of course, good camera software is a good thing and for the segment of society that's enthused about photography it's going to be essential. I'm just not convinced it's the secret sauce to mass appeal. Edit for response: > Well, okay, what problem would you rather see them tackle? I don't have such a preference. I expect volunteer developers will work on matters they find personally compelling. Those that find photography software compelling should pursue their passion.
- jonas21 7y ago
  > I think relatively few consumers know anything about photography. That's precisely why you need good image processing software that works automatically. Nearly everybody wants to take great photos with their phone, but few people know how to configure everything themselves, and even those who do probably don't want to fiddle with every shot. > Couldn't they just find another niche? Well, okay, what problem would you rather see them tackle?
- teleclimber 7y ago
  The reason for a good camera is not to take artistic photography (although you can do that too and many do). It's because people use that camera to record memories that they cherish. Just ask any new parent what happens to their phone's camera roll once number 1 pops out. Or ask young people to show you where they went on their last vacation: many times it'll be a bunch of cool shots taken on a phone. These memories are priceless and having to record them with a device that is open but has the photo quality of a 5 year old device is not an attractive trade-off.

catern 7y ago

>...collaboration with the industry to develop a camera stack... protecting vendor core IP. Can't say I have any interest in protecting "vendor IP" in their crappy camera drivers. Let vendors open their blobs! Closed source camera drivers will just reduce quality and increase bugginess.

Jonnax 7y ago
What's the point of this all or nothing attitude? This looks like a move to getting better support. The alternative is getting nothing.
- matheusmoreira 7y ago
  The point is to get high quality software that actually belongs to us all. Accepting closed source drivers means accepting the Windows model where companies release buggy drivers for one or two combinations of processor architecture and operating system version before ending all support. The company gets to open wash their reputation while still keeping us dependent on them for scraps of support. Giving us nothing at all would've been more respectful than this. Letting companies protect their intellectual property benefits absolutely nobody but them. They don't have our best interests in mind, only their own. The ideal solution is to reverse engineer their hardware's interface and produce a competing implementation which will no doubt be better than their minimum viable product. My laptop has a backlit keyboard with RGB LEDs and I managed to figure out how it works and make my own program to replace the manufacturer's. I'm so happy I don't have to boot into Windows just to use their slow software anymore. They don't even have to give us source code. Documentation would eliminate the need to actually reverse engineer the hardware. They could just release that data and all interested parties would start hacking. The problem would take care of itself at no cost to them.
- shmerl 7y ago
  We don't need more blobs. Such kind of approach is not helping.
- serf 7y ago
  > What's the point of this all or nothing attitude? the point is to try to enact change in corporate behavior with regards to consumer devices. >The alternative is getting nothing. goes both ways. If I don't buy a product that uses binary blobs all over the place, they don't sell the product to me. Supporting products that support DRM systems natively does a lot to support the underlying DRM system, too. It sets a native expectation of DRM among developers, fostering the use. And worst of all, I can't try to fix blobs without labor that is so intensive that it renders the very idea moot.

mschuster91 7y ago

Is this aimed for smartphones or for real cameras? I mean, the Sony Alpha series do run Linux, even with an (ancient) Android runtime...

omtinez 7y ago

I'm surprised to see such big emphasis on support for Android. Most camera modules come with their own drivers that already provide Android support. One benefit could be the licensing, but after a quick inspection it is unclear to me what license this library is under -- there is a licenses folder with 4 different licenses in addition to a developer agreement. The design seems to be heavily inspired by the Android camera API: per-frame configuration, 3A, multiple stream support, device enumeration, etc. > The HAL will implement internally features required by Android and missing from libcamera, such as JPEG encoding support. That is interesting, since most camera modules will have a hardware accelerated path to encode frames directly to JPEG. If it's done internally, it will be much slower than all other implementations I'm aware of.

ChrisMarshallNY 7y ago

This is no simple thing to do. The biggest issue is that "raw" Linux is not an RTOS. When the photographer presses the shutter, you have a few milliseconds to capture the image. You can't have the OS decide to prioritize receiving a tweet before getting to the shutter release. It's even more intense with video, and most still cameras are quickly morphing into high-quality video cameras. Also, image processing is REAL intense; especially with Computational Photography. It can suck down your battery in seconds. There's a few ways to deal with it. You could use something like Xenomai or RTLinux, or establish some kind of SoC for the realtime-sensitive processing (what most phone manufacturers probably do -Apple sure does). It's a non-trivial task.

amelius 7y ago
Curious, how does V4L handle these things?
- ChrisMarshallNY 7y ago
  Not sure. Never played with it. My experience is with a custom Android build, on some standard Snapdragon stuff (won't get any more detailed than that). I have played with ffmpeg, which is not suitable for mobile, unless heavily modified. The VideoLAN folks seem to have gotten VLC working for mobile, but it's a real pig.
- joshvm 7y ago
  It gets a bit messy. In order to capture an image, most image sensors are triggered. Some sensors will operate in "free-run" mode, but in reality something is sending a capture signal to the camera. In sensors where you have very low level control over things, usually what that does is clear the pixels (dump the accumulated charge) and start exposing. Then, when the exposure is complete, the camera makes the data available for read-out (for a phone sensor analogue-digital conversion is likely done in circuitry on the sensor, not in the phone). That trigger is the important bit. Using triggering you can synchronise multiple cameras (e.g. the sensors in multi-camera phones are almost certainly co-triggered to avoid ghosting), illumination units, etc. In fact many camera chips can also be set to output flash signals when they're exposing, this eliminates latency in sending separate "capture" and "flash" signals. When I've deployed machine vision systems, often we would use an external system to trigger everything (e.g. a microcontroller). Interfacing hardware and software triggering is a nightmare. Then you've got whatever low level interface is actually communicating to the camera. On a phone, that's probably MIPI/CSI-2. Modern SoCs have dedicated (hardware) camera interfaces (the ISP) that are designed specifically to load images from camera sensors and do some minimal processing. You need kernel-level drivers for whatever your SoC is. These drivers will (in theory) let you set up arbitrary image sensors with whatever resolution/timing they require. But the key is you need two drivers: one to control the ISP and another to send commands to the camera sensor. A lot of the time (cough Broadcom) you only get a blob for the ISP and you almost never get full datasheets for image sensors without being a big customer. Rockchip have an open source driver for their ISP: http://opensource.rock-chips.com/wiki_Rockchip-isp1 http://opensource.rock-chips.com/wiki_Rockchip-isp1 See Rockchip again for an open-source example of an actual sensor driver. https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/i2c/imx219.c https://github.com/rockchip-linux/kernel/blob/release-4.4/dr... Want to make an educated guess why libcamera supports the [Rockchip] RK3399? They make it comparatively easy. In the end V4L acts as a mediator between this low level interface and other things which want to talk to the camera. Rockchip have provided what amounts to a plugin for V4L. Then on top of V4L you have libraries like OpenCV which give you nice interfaces like `imread` to capture images and do stuff with them. To answer your question you probably need to be more specific. The performance of what V4L does for a specific ISP/sensor combination very much depends on those underlying drivers. If you can afford it, a safe strategy is to leave the sensor in free-running mode in a separate thread that's dedicated to capturing images. Then in your application you pick off the most recent frame and do stuff with it in parallel. For a 30fps camera you therefore have 30ms or so to do your stuff before the next frame comes in.

shmerl 7y ago

> the Linux media community has very recently started collaboration with the industry to develop a camera stack that will be open-source-friendly while still protecting vendor core IP What does that mean? Open source should be open source. Is it going to work on top of v4l or it's independent? https://en.wikipedia.org/wiki/Video4Linux https://en.wikipedia.org/wiki/Video4Linux

sandGorgon 7y ago

This is very interesting. How does this compare with CameraX for which Google has built an entire building with hundreds of cameras as a testlab ? https://www.extremetech.com/computing/301298-camerax-googles-new-weapon-in-the-photography-wars https://www.extremetech.com/computing/301298-camerax-googles... https://developer.android.com/training/camerax https://developer.android.com/training/camerax

lovelearning 7y ago
This sits much lower in the stack than CameraX. This is more of a HAL for cameras standardized across multiple OSes. http://libcamera.org/docs.html#android-camera-hal-v3-compatibility http://libcamera.org/docs.html#android-camera-hal-v3-compati...

me551ah 7y ago

I don't get who is going to be using this library. Most android developers would prefer to use native Java APIs instead of a C++ library ( with a JNI wrapper around it?) since it's easier. Linux phones barely have any market share so this library would be used mainly on computers where most of the cameras are USB anyway.