5 ms·

The reason why modern smartphones have such good camera picture quality is primarily thanks to the ISP profiles developed by teams of people for the particular

The reason why modern smartphones have such good camera picture quality is primarily thanks to the ISP profiles developed by teams of people for the particular sensor in use. ISP is an imaging signal processor, a component of most mobile SOCs that sits between the camera and the rest of the system. It is a proprietary component that is essentially a black box controlled with a "profile" for specific camera chip. The profile specifies how to debayer, denoise, sharpen, color correct, adjust brightness and autofocus of the chip. Without a very good profile even a good camera sensor output will look like crap. I've spent few weeks of my (spare) time improving a Sony imx219 driver for a Rockchip rk3566 cpu that is used in few pine devices I own. I needed high fps video (I managed to get 120fps 720p - I think I could push it further to 140 perhaps, but this was OK for now) and as low latency as I could get. I wanted <20ms as I could encode 720p h264 frames in 7ms,unfortunately I got stuck with ISP (image signal processor) adding 40ms of latency even when in pass through mode. The modern ISP is a "black box" even with source code for the drivers. In addition every single ISP has its own proprietary format for those profiles. There are OS projects that try to make universal camera sensor profiles that work with multiple ISPs (libcamera for one), but it is still early days. What we need is a truly programmable/flexible isp. As it stands the only way to get sub 40ms of latency from a camera for a hobbyist is to use an fpga...

pluc 4y ago
megous 4y ago
The Pinephone Pro ISP is documented in RK3399 TRM (part3), and in various PDFs available online. It's not really a huge blackbox. It does not make the calibration (or preparation of profiles for various light conditions) easy though. There's a need for a bunch of HW tooling that is somewhat expensive for a hobbyist. And there's a lot of work and algorithm design that needs to be done outside of ISP for controling the autofocus, autoexpsoure and detection of type lighting in the scene and profile switching. None of this is particularly trivial.
- abxytg 4y ago
  It's very clear he is talking about black boxes from the perspective of one trying to reverse one to fix pinephones isp...
  - kop316 4y ago
    Respectfully, the person you are responding to knows what he is talking about: https://xnux.eu/log/ https://xnux.eu/log/
    - Roark66 4y ago
      So they write a bunch of drivers and stuff for pine phone. As a pine fan I thank them, but this doesn't mean we can't have a difference in opinion what does it mean for some component to be a black box. If you compare with some other manufacturer that gives us absolutely no docs, then yes having register level docs and user side source is really nice. But if you want to do something unconventional (like switch the whole damn thing off and bypass its latency) not knowing how it works internally is still a black box. For example there is an irq that fires when the first component of the isp(MI-memory interface) sees a vsync signal for a frame. We know it exists. We know how to enable/disable that irq. We can write code that does stuff when it fires. But what we don't know is is this vsync signal fired truly at the input of the ISP? It's output? Or perhaps between some specific components? It turns out that vsync fires not when the isp sees the very first vsync what the location of the registers suggests, but after all the processing is done at the output buffer (40ms later usually), but one wouldn't be able to tell without a lot of profiling. This amongst other things is what I call a black box. However, compare it with let's say a raspberry pi, and at register level the documentation is much better (for the isp).
      - megous 4y ago
        https://megous.com/dl/tmp/3add845b7e814de9.png https://megous.com/dl/tmp/3add845b7e814de9.png There are bunch of interrupt statuses around frame capture: RIS_FRAME_IN, RIS_FRAME, RIS_V_START. Not sure why you'd decide to use V_START for deciding when the frame is in memory. VSYNC start is sent at the start of a frame and it's documented as just a interrupt that is directly triggered from the MIPI CSI sync signal in the datasheet. quote: "The incoming hsync and vsync control signals from the camera are connected to interrupt logic. It is possible to trigger on these signals for an event triggered configuration during processing."
rjzzleep 4y ago
Can someone dive more deeply into this? I'm asking because it seems to me that there is more to this than what you just described. Sony AOSP is constantly upgrading the kernel, but the camera is always mediocre at best. If it was just the ISP there should have been a way in the past to take the ISP blob from similar chipset phones in the past and just use it with an updated base no? But that by itself doesn't seem to be enough to get the cam working decently.
- WithinReason 4y ago
  From the pictures linked in the article it seems to be about more than just the ISP.
  - Roark66 4y ago
    People don't realise how much stuff the ISP does. It took me over a week to make an extremely basic profile for a new camera chip. And I had all the tools and the documents OEMs use. There is a 100 page long procedure I had to follow taking multiple pictures of calibration standards illuminated with various calibrated lights. Then at the end it still looks like crap, but it is pretty true looking. Then the work to make it look good really starts. I had no idea before I started. There are simple things like debayer, but even just talking about exposure and gain control based on illumination is a pretty complex subject in itself. First you need to define parameters when to change exposure and when to leave it alone and just use camera gain. Then you need a set of parameters for when to use analog gain and when to use digital gain as communicated to the sensor. Then you have another two sets of parameters one for adjusting gain on the ISP (as opossed to all the previous stuff that feeds in to the camera chip) and some ISPs I'm told also have more advanced builtin correction based on curves. All this and we haven't even touched on how you measure the Illumination of the scene. At the very least you need one profile for back-lit and one for front-lit scenes. Then you need parameters for how the isp decides which profile to apply automatically and so on. Mind that this is just brightness, we haven't even started talking about color correction, white balance, denoise, sharpen and 13 other ISP blocks that are there. BTW, you need another profile for every resolution your camera outputs too. I don't know a lot about different ISPs. I only worked with one, but it gave me enough of an understanding to know why there is no way one profile will work for two camera chips. Or one profile will not work with two different versions of ISP hw. That's why I hope libcamera succeeds. They try to standardise all the isp control, but it is a huge task especially when every single isp is different.
hedora 4y ago
People used to make similar arguments about injket printer drivers (where the computer does dithering, head scheduling, etc). Over time the open source crowd implemented the state of the art image processing techniques once, then pointed them at reverse engineered backend drivers for each model. The result was better than what the manufacturers (especially HP, but also canon, etc) could do. I'm hoping history repeats itself with these camera sensors.
MartijnBraam 4y ago
A pine rk3566 device with a camera? It's great to have more people working on imx219 though:)
- Roark66 4y ago
  Well, with a camera port. Both the soquartz and Quartz64-A have csi connector for the camera. The pinout is such that raspberry pi v2 camera (imx219) can be connected directly to soquartz. However for Quartz64-A it requires an adapter. Regarding my imx219 code that pushes it to 120fps I haven't submitted it upstream because the way fps is set changed in v5 Linux kernels. I used a v4 driver as a starting point because many other features only worked on v4(such as full gpu encode/decode etc). Also I preferred my userland side to not have to calculate timing parameters as it would require a per device implementation. So my 120fps imx219 driver is for linux kernel 4.19 named rk3566_bsp_kernel-4.19, but I do have plans to upstream my code when I make them compatible with newest.

pluc 4y ago

megous 4y ago

The Pinephone Pro ISP is documented in RK3399 TRM (part3), and in various PDFs available online. It's not really a huge blackbox. It does not make the calibration (or preparation of profiles for various light conditions) easy though. There's a need for a bunch of HW tooling that is somewhat expensive for a hobbyist. And there's a lot of work and algorithm design that needs to be done outside of ISP for controling the autofocus, autoexpsoure and detection of type lighting in the scene and profile switching. None of this is particularly trivial.

abxytg 4y ago
It's very clear he is talking about black boxes from the perspective of one trying to reverse one to fix pinephones isp...
- kop316 4y ago
  Respectfully, the person you are responding to knows what he is talking about: https://xnux.eu/log/ https://xnux.eu/log/
  - Roark66 4y ago
    So they write a bunch of drivers and stuff for pine phone. As a pine fan I thank them, but this doesn't mean we can't have a difference in opinion what does it mean for some component to be a black box. If you compare with some other manufacturer that gives us absolutely no docs, then yes having register level docs and user side source is really nice. But if you want to do something unconventional (like switch the whole damn thing off and bypass its latency) not knowing how it works internally is still a black box. For example there is an irq that fires when the first component of the isp(MI-memory interface) sees a vsync signal for a frame. We know it exists. We know how to enable/disable that irq. We can write code that does stuff when it fires. But what we don't know is is this vsync signal fired truly at the input of the ISP? It's output? Or perhaps between some specific components? It turns out that vsync fires not when the isp sees the very first vsync what the location of the registers suggests, but after all the processing is done at the output buffer (40ms later usually), but one wouldn't be able to tell without a lot of profiling. This amongst other things is what I call a black box. However, compare it with let's say a raspberry pi, and at register level the documentation is much better (for the isp).
    - megous 4y ago
      https://megous.com/dl/tmp/3add845b7e814de9.png https://megous.com/dl/tmp/3add845b7e814de9.png There are bunch of interrupt statuses around frame capture: RIS_FRAME_IN, RIS_FRAME, RIS_V_START. Not sure why you'd decide to use V_START for deciding when the frame is in memory. VSYNC start is sent at the start of a frame and it's documented as just a interrupt that is directly triggered from the MIPI CSI sync signal in the datasheet. quote: "The incoming hsync and vsync control signals from the camera are connected to interrupt logic. It is possible to trigger on these signals for an event triggered configuration during processing."

rjzzleep 4y ago

Can someone dive more deeply into this? I'm asking because it seems to me that there is more to this than what you just described. Sony AOSP is constantly upgrading the kernel, but the camera is always mediocre at best. If it was just the ISP there should have been a way in the past to take the ISP blob from similar chipset phones in the past and just use it with an updated base no? But that by itself doesn't seem to be enough to get the cam working decently.

WithinReason 4y ago
From the pictures linked in the article it seems to be about more than just the ISP.
- Roark66 4y ago
  People don't realise how much stuff the ISP does. It took me over a week to make an extremely basic profile for a new camera chip. And I had all the tools and the documents OEMs use. There is a 100 page long procedure I had to follow taking multiple pictures of calibration standards illuminated with various calibrated lights. Then at the end it still looks like crap, but it is pretty true looking. Then the work to make it look good really starts. I had no idea before I started. There are simple things like debayer, but even just talking about exposure and gain control based on illumination is a pretty complex subject in itself. First you need to define parameters when to change exposure and when to leave it alone and just use camera gain. Then you need a set of parameters for when to use analog gain and when to use digital gain as communicated to the sensor. Then you have another two sets of parameters one for adjusting gain on the ISP (as opossed to all the previous stuff that feeds in to the camera chip) and some ISPs I'm told also have more advanced builtin correction based on curves. All this and we haven't even touched on how you measure the Illumination of the scene. At the very least you need one profile for back-lit and one for front-lit scenes. Then you need parameters for how the isp decides which profile to apply automatically and so on. Mind that this is just brightness, we haven't even started talking about color correction, white balance, denoise, sharpen and 13 other ISP blocks that are there. BTW, you need another profile for every resolution your camera outputs too. I don't know a lot about different ISPs. I only worked with one, but it gave me enough of an understanding to know why there is no way one profile will work for two camera chips. Or one profile will not work with two different versions of ISP hw. That's why I hope libcamera succeeds. They try to standardise all the isp control, but it is a huge task especially when every single isp is different.

hedora 4y ago

People used to make similar arguments about injket printer drivers (where the computer does dithering, head scheduling, etc). Over time the open source crowd implemented the state of the art image processing techniques once, then pointed them at reverse engineered backend drivers for each model. The result was better than what the manufacturers (especially HP, but also canon, etc) could do. I'm hoping history repeats itself with these camera sensors.

MartijnBraam 4y ago

A pine rk3566 device with a camera? It's great to have more people working on imx219 though:)

Roark66 4y ago
Well, with a camera port. Both the soquartz and Quartz64-A have csi connector for the camera. The pinout is such that raspberry pi v2 camera (imx219) can be connected directly to soquartz. However for Quartz64-A it requires an adapter. Regarding my imx219 code that pushes it to 120fps I haven't submitted it upstream because the way fps is set changed in v5 Linux kernels. I used a v4 driver as a starting point because many other features only worked on v4(such as full gpu encode/decode etc). Also I preferred my userland side to not have to calculate timing parameters as it would require a per device implementation. So my 120fps imx219 driver is for linux kernel 4.19 named rk3566_bsp_kernel-4.19, but I do have plans to upstream my code when I make them compatible with newest.