8 ms·

A Billion Pixels a Second: Inside Apple's iPhone 16 Camera Labs

123 points by thm 2y ago

shrubble 2y ago
This is really a puff piece - for instance, Nokia had a 1 billion pixels per second processing pipeline on their 808 PureView phone. In 2012.
- dylan604 2y ago
  what on cnet is not a puff piece?
- CharlesW 2y ago
  Even so, the "camera first" Nokia 808 PureView took notably worse photos (DxOMark Mobile Photo score: 60) than the iPhone 7 (score: 86) and other flagship phones (Samsung Galaxy S6 Edge score: 86) of that time.
  - gregoriol 2y ago
    Size is not the only metric that matters
  - n144q 2y ago
    iPhone 7 was released in 2016, 4 years after 808 was released. Can't see what your point is.
- rob_c 2y ago
  Unless apple invented it, no they didn't and you're wrong... The cult of apple kinda says everything wrong about the valley to me. This isn't magicians at work with secret spells, this is just cutting edge tech that is at the point of being commodity. Making something appealing to idiots has drawbacks, less repairability, less durability etc and now even apple is slowly reverting even this under govt changes...
  - deleted 2y ago
    [deleted]
  - scarface_74 2y ago
    If iOS has a majority market share in the US, wouldn’t anyone who is not using an iPhone be part of the “cult”? And I bet if I drop my phone, it would be a lot easier to drive to one of the five Apple Stores in the metro area to get it repaired than it would be for you to get your camera repaired. That’s not even considering the fact that it could service a drop better than your camera and is water resistant
    - n144q 2y ago
      > That’s not even considering the fact that it could service a drop better than your camera and is water resistant Definitely need citations for this.
      - scarface_74 2y ago
        https://kenrockwell.com/nikon/18200/18200-drop-test.htm https://kenrockwell.com/nikon/18200/18200-drop-test.htm https://www.imaging-resource.com/news/2013/05/23/camera-meet-concrete-will-canon-or-nikon-triumph-in-the-DSLR-drop-test https://www.imaging-resource.com/news/2013/05/23/camera-meet... Can you imagine any modern phone not surviving a 3 foot drop?
- ilrwbwrkhv 2y ago
  Yes, it definitely is. Once you notice the signs of PR, you can see this was being written because Apple's privacy story fell apart. This is really about hiding their entire SIRI recording your conversations without your permission.
- londons_explore 2y ago
  Indeed - it can mean simply a 1GHz data bus thats 32 bits wide. Pipeline all your image processing so it's 1 cycle per pixel and you're done.
- dagmx 2y ago
  Your comment is somewhat misleading. The 808, while a great high resolution sensor, processed stills at 5MP after pixel binning, compared to the 48/24/12MP of the iPhones. It did have a non-binned mode but again for stills. https://en.wikipedia.org/wiki/Nokia_808_PureView https://en.wikipedia.org/wiki/Nokia_808_PureView The majority of the post is actually about video and the comparison is a lot more dramatic there. A 4k frame from a modern phone ,like the iPhone 16 pro, is ~8MP. The 808 could “only” do 1080p at 30fps (62M pixels a second). Compare that to 4k at 120fps (995M pixels a second). That’s a far cry from 1Bn per second that you’re claiming. Impressive for the time though. There’s a significant difference here. The sensor on the 808 was amazing. The real meat of the tech here is the image processing pipeline that can process everything off the sensor for further processing.
  - shrubble 2y ago
    The issue is not what Nokia did with it, but the presence of the functionality. The binning and other features were enabled (poorly or not isn’t part of the discussion) by a chip that could do about 1 billion pixels per second; that’s the issue; Apple doing it 12 years later and hyping it is what I was commenting on.
    - dagmx 2y ago
      No, the issue is that you’re misrepresenting the facts of the device. The processing you’re talking about could not handle 1Bn pixels a second except for maybe the very frontend of the pipeline. It could not handle it beyond that, which is where the meat of any ISP is. It’s easy to dismiss the technology if you allow yourself to dismiss the majority of what needs to be done past the sensor read out.
      - shrubble 2y ago
        It's the Broadcom BCM2763, also known as the VideoCore IV. The datasheet is still available. This is happening well beyond the sensor read-out and the fact you are mentioning that, tells me you don't actually understand the architecture at the necessary level. Sorry.
JSR_FDED 2y ago
The title doesn’t do it justice - everything with images quickly adds up. Doing 120 fps video at 4K so that any chosen frame looks amazing without artifacts is really quite an achievement. The microphones were actually more interesting to me, that you can get lavalier performance from the tiny mics in the phone that are physically far from the person being recorded is seriously clever. Getting this to work some of the time is already an achievement but I think people underestimate how much work goes into making it work across all different scenarios.
- jval43 2y ago
  Mirrorless cameras nowadays can push as much as 8k60 (twice the amount of data over 4k120), every frame at full quality. All that with better optics/sensors and less "perceptual testing", thus less of an overprocessed look.
  - zuhsetaqi 2y ago
    What do they cost and how big are they? Nobody is claiming a world record from Apple …
  - Almondsetat 2y ago
    Heat dissipation
  - dylan604 2y ago
    can that mirrorless camera make a phone call, send a text, edit the photo it takes, browse the web, upload the picture to an app, run other apps, or any of the many many other things a phone does that a single function dedicated camera cannot? what trade offs do you accept for that mirrorless camera to do all of the same functionality while still taking an above decent looking image for the vast majority of its users without all of those fancy lenses fixed or interchangeable? As the saying goes, the best came you have is the one you have on you. Only snooty photo/video types care about your comment. Most videos used by phone people are only ever viewed on that device or other similar devices of others viewing it on whatever app they shared it. To even consider them comparable is just not an honest take in the slightest. I say this as someone with several DSLRs and a couple of cinema camera bodies in the next room.
mannyv 2y ago
Two unanswered question: why not put the camera in the middle of the phone? And why do the cameras keep moving around?
- kaonwarb 2y ago
  The space in the middle is dominated by a big battery in any smartphone I've seen. I suspect it would be less efficient to break that up.
  - makeitdouble 2y ago
    FWIW Xiaomi has flagships with split batteries for faster charge. e.g.: https://www.mi.com/global/product/xiaomi-11t-pro-120w-xiaomi-hypercharge/ https://www.mi.com/global/product/xiaomi-11t-pro-120w-xiaomi...
- klipt 2y ago
  I think I'd accidentally stick my thumb/finger in front of a central camera even more often than I already do with corner cameras!
- snowwrestler 2y ago
  > why not put the camera in the middle of the phone? This is one of those things that doesn't seem like it should matter, but it does. If the lens is mounted in the exact center of the body, the images come out looking unbalanced. To produce balanced images, you have to offset the lens. Even very expensive pro mirrorless bodies are offset; that is, if you look directly down the center of the lens, you'll notice there is more camera body sticking out on one side than the other. This is called the chirality of the optical path and it is surprisingly difficult to predict analytically. Companies will typically design the optical path, prototype it, and mount it on a jig to precisely measure the chirality. From this, they design the body with the proper offset. Chirality is more noticeable the smaller the sensor and the shorter the lens. So on smart phones, which put tiny sensors behind wide-angle lenses, they have to get the offset just right. This explains why the lenses are in slightly different places on the body every time Apple updates their cameras.
TazeTSchnitzel 2y ago
The body text is really uncomfortably sycophantic.
- mrandish 2y ago
  Yes, and it's maddeningly detail-free. Just vague claims that something wonderful is being accomplished with zero data or specifics.
  - pierrefermat1 2y ago
    The reality is journalists don't really have the knowledge to explain any specifics at all, so you just get this fluff. But hey at least there's pretty pictures
cynicalsecurity 2y ago
Photos on my wife's camera look slightly deformed. The people don't look like themselves, their faces are just slightly off. I'm blaming the AI inside of the photos app on iPhone, but I'm not sure. People's face look perfect on my mediocre Android though. I'm never going to buy or use an iPhone. Even the questionable advantage which was supposed to be the iPhone's camera is fake.
- gazchop 2y ago
  This is probably a combination of the lens corrections, the pretty awful auto white balance (warmth), the terrible oversharpening and also a bit of True Tone. Portrait mode also wrecks a lot of photos due to the crappy emulation of DOF. Output is clever but shit. Due to the general flatness of the lens there is a lot of distortion around the edges which is digitally removed after the photo is taken. This isn't 100% perfect and causes some rather uncanny looks in some of the photos. You can use this for artistic effect but it looks crap mostly. Generally if you're using a proper camera there's a big chunk of glass in front of it so the main part of a portrait is well outside the distorted edges of the frame so it's not noticeable. Even new cameras use minimal lens corrections in body as well to eliminate this. As for the white balance, Apple never seem to get this right. The colours are always slightly too orange / warm and vivid and never quite match reality in experience. You can crank the warmth down a bit after in photos.app to kill some of it. Oversharpening - everything is too sharp. This makes the image pop out but nothing more. It's a terrible curse on smartphones. Not much you can do about it. Even shooting ProRAW on mine is oversharpened. If you turn the True Tone feature off in Display/Settings it looks a bit better as well. That seems to completely mung viewing any photos later, giving them sometimes an over-blue tone. Urgh all this is why I bought a mirrorless. Smartphones are really not very good. Even good ones (mine is a 15 Pro). Mine gets mostly used to take photos of an AirBnB when I leave it now or where the car got parked.
  - amluto 2y ago
    My bonus pet peeve about portrait mode is that the internet is full of portrait mode photos, which means AI gets trained on portrait mode photos, which means AI generates pictures that look like portrait mode. Garbage in, garbage out.
harha_ 2y ago
I wonder how much that wire mesh floor distorts the recording result? I guess it must be insignificant, since the walls, ceiling and floor absorb almost all reflected sound waves?
- jdietrich 2y ago
  Sound waves are really long. 20kHz is about the limit of human hearing, so the shortest wave we can hear is about 17cm; the longest wave we can hear is about 17 meters. A suitably designed mesh will be effectively transparent across that frequency range.
orev 2y ago
> 38% of people said that better cameras are a main motivation for buying a new phone This strikes me as just a reflection of the ad campaigns. Apple promotes “better cameras” for every new iPhone, almost exclusively in their ads, so it’s not surprising that’s what people would say. With every new phone being just an incremental upgrade, hyping up the camera is the only way to get people to drop a $1000 on a new one. Most of these 38% won’t be able to tell the difference between a phone pic taken 5 or more years ago.
- esafak 2y ago
  Better images got people to buy even more expensive digital cameras that do less than your cell phone. It turns out people like to take pictures.
  - giancarlostoro 2y ago
    Do less in terms of what exactly? Because I have photos from old digital cameras from nearly 20 years ago that look drastically better and more detailed than Android or iPhone photos from several years back. I would be surprised if the quality is down. In terms of features the quality is key.
    - esafak 2y ago
      Do less in terms of not being able to do anything beside take a picture. And they don't do any computational photography, either, so their pictures are not necessarily better. Try any low light photography with moving objects on your camera? Or adjust the focal point? Cell phones can do it. If digital cameras were so good they would not be disappearing.
      - orev 2y ago
        Phone cameras have come a long way, no doubt, but DSLRs really do look amazing. Phones are winning because “the best camera is the one that’s with you”. Since people have the phone camera in their pocket all the time, they usually don’t bother to carry a separate bulky one except for special occasions.
a1o 2y ago
One thing about the iPhone microphone. I had a beautiful day the other day here with some light drizzle and the sea waves were rocking nicely it was a great sound and the beach was empty. Great feeling. I wanted to record this but I only had my iPhone - was just walking in the beach. So neither the default recording app of the iphone and neither any of the ones I tried, could capture the ocean waves or the sound of the drizzle. I had previously (intentionally) set out to the beach with the intent to capture audio and did so with my Laptop + external mic. My conclusion is that unfortunately that is the way it has to be, and that it's not possible to capture audio on a whim as is with photos.
- pryelluw 2y ago
  Which iPhone model?
  - a1o 2y ago
    The experience I described was with the 14 but I also tried in the 15 pro something similar and had same experience. For now my approach is to use a external microphone with a longer cable and a notebook, and this works, but it would be nice to make this work with the iPhone.
    - pryelluw 2y ago
      I daily an (newest model) SE and got some wireless lavalier mics that plug into the lighting port. They work well and were cheap. Have to carry them around. Not name brand or anything. Though I hope to get some from rode. What do you use for storage?
      - a1o 2y ago
        I don't use the phone productively, everything I tried I endup concluding it can't do? So it's only use is complaining about things in hacker news for killing time. The most recent idea was trying to use it for sounds but I had the mentioned issue. Does the lavalier works for environment sounds or you only use for voice?

123 points by thm 2y ago

This is really a puff piece - for instance, Nokia had a 1 billion pixels per second processing pipeline on their 808 PureView phone. In 2012.

dylan604 2y ago
what on cnet is not a puff piece?
CharlesW 2y ago
Even so, the "camera first" Nokia 808 PureView took notably worse photos (DxOMark Mobile Photo score: 60) than the iPhone 7 (score: 86) and other flagship phones (Samsung Galaxy S6 Edge score: 86) of that time.
- gregoriol 2y ago
  Size is not the only metric that matters
- n144q 2y ago
  iPhone 7 was released in 2016, 4 years after 808 was released. Can't see what your point is.
rob_c 2y ago
Unless apple invented it, no they didn't and you're wrong... The cult of apple kinda says everything wrong about the valley to me. This isn't magicians at work with secret spells, this is just cutting edge tech that is at the point of being commodity. Making something appealing to idiots has drawbacks, less repairability, less durability etc and now even apple is slowly reverting even this under govt changes...
- deleted 2y ago
  [deleted]
- scarface_74 2y ago
  If iOS has a majority market share in the US, wouldn’t anyone who is not using an iPhone be part of the “cult”? And I bet if I drop my phone, it would be a lot easier to drive to one of the five Apple Stores in the metro area to get it repaired than it would be for you to get your camera repaired. That’s not even considering the fact that it could service a drop better than your camera and is water resistant
  - n144q 2y ago
    > That’s not even considering the fact that it could service a drop better than your camera and is water resistant Definitely need citations for this.
    - scarface_74 2y ago
      https://kenrockwell.com/nikon/18200/18200-drop-test.htm https://kenrockwell.com/nikon/18200/18200-drop-test.htm https://www.imaging-resource.com/news/2013/05/23/camera-meet-concrete-will-canon-or-nikon-triumph-in-the-DSLR-drop-test https://www.imaging-resource.com/news/2013/05/23/camera-meet... Can you imagine any modern phone not surviving a 3 foot drop?
ilrwbwrkhv 2y ago
Yes, it definitely is. Once you notice the signs of PR, you can see this was being written because Apple's privacy story fell apart. This is really about hiding their entire SIRI recording your conversations without your permission.
londons_explore 2y ago
Indeed - it can mean simply a 1GHz data bus thats 32 bits wide. Pipeline all your image processing so it's 1 cycle per pixel and you're done.
dagmx 2y ago
Your comment is somewhat misleading. The 808, while a great high resolution sensor, processed stills at 5MP after pixel binning, compared to the 48/24/12MP of the iPhones. It did have a non-binned mode but again for stills. https://en.wikipedia.org/wiki/Nokia_808_PureView https://en.wikipedia.org/wiki/Nokia_808_PureView The majority of the post is actually about video and the comparison is a lot more dramatic there. A 4k frame from a modern phone ,like the iPhone 16 pro, is ~8MP. The 808 could “only” do 1080p at 30fps (62M pixels a second). Compare that to 4k at 120fps (995M pixels a second). That’s a far cry from 1Bn per second that you’re claiming. Impressive for the time though. There’s a significant difference here. The sensor on the 808 was amazing. The real meat of the tech here is the image processing pipeline that can process everything off the sensor for further processing.
- shrubble 2y ago
  The issue is not what Nokia did with it, but the presence of the functionality. The binning and other features were enabled (poorly or not isn’t part of the discussion) by a chip that could do about 1 billion pixels per second; that’s the issue; Apple doing it 12 years later and hyping it is what I was commenting on.
  - dagmx 2y ago
    No, the issue is that you’re misrepresenting the facts of the device. The processing you’re talking about could not handle 1Bn pixels a second except for maybe the very frontend of the pipeline. It could not handle it beyond that, which is where the meat of any ISP is. It’s easy to dismiss the technology if you allow yourself to dismiss the majority of what needs to be done past the sensor read out.
    - shrubble 2y ago
      It's the Broadcom BCM2763, also known as the VideoCore IV. The datasheet is still available. This is happening well beyond the sensor read-out and the fact you are mentioning that, tells me you don't actually understand the architecture at the necessary level. Sorry.

JSR_FDED 2y ago

The title doesn’t do it justice - everything with images quickly adds up. Doing 120 fps video at 4K so that any chosen frame looks amazing without artifacts is really quite an achievement. The microphones were actually more interesting to me, that you can get lavalier performance from the tiny mics in the phone that are physically far from the person being recorded is seriously clever. Getting this to work some of the time is already an achievement but I think people underestimate how much work goes into making it work across all different scenarios.

jval43 2y ago
Mirrorless cameras nowadays can push as much as 8k60 (twice the amount of data over 4k120), every frame at full quality. All that with better optics/sensors and less "perceptual testing", thus less of an overprocessed look.
- zuhsetaqi 2y ago
  What do they cost and how big are they? Nobody is claiming a world record from Apple …
- Almondsetat 2y ago
  Heat dissipation
- dylan604 2y ago
  can that mirrorless camera make a phone call, send a text, edit the photo it takes, browse the web, upload the picture to an app, run other apps, or any of the many many other things a phone does that a single function dedicated camera cannot? what trade offs do you accept for that mirrorless camera to do all of the same functionality while still taking an above decent looking image for the vast majority of its users without all of those fancy lenses fixed or interchangeable? As the saying goes, the best came you have is the one you have on you. Only snooty photo/video types care about your comment. Most videos used by phone people are only ever viewed on that device or other similar devices of others viewing it on whatever app they shared it. To even consider them comparable is just not an honest take in the slightest. I say this as someone with several DSLRs and a couple of cinema camera bodies in the next room.

mannyv 2y ago

Two unanswered question: why not put the camera in the middle of the phone? And why do the cameras keep moving around?

kaonwarb 2y ago
The space in the middle is dominated by a big battery in any smartphone I've seen. I suspect it would be less efficient to break that up.
- makeitdouble 2y ago
  FWIW Xiaomi has flagships with split batteries for faster charge. e.g.: https://www.mi.com/global/product/xiaomi-11t-pro-120w-xiaomi-hypercharge/ https://www.mi.com/global/product/xiaomi-11t-pro-120w-xiaomi...
klipt 2y ago
I think I'd accidentally stick my thumb/finger in front of a central camera even more often than I already do with corner cameras!
snowwrestler 2y ago
> why not put the camera in the middle of the phone? This is one of those things that doesn't seem like it should matter, but it does. If the lens is mounted in the exact center of the body, the images come out looking unbalanced. To produce balanced images, you have to offset the lens. Even very expensive pro mirrorless bodies are offset; that is, if you look directly down the center of the lens, you'll notice there is more camera body sticking out on one side than the other. This is called the chirality of the optical path and it is surprisingly difficult to predict analytically. Companies will typically design the optical path, prototype it, and mount it on a jig to precisely measure the chirality. From this, they design the body with the proper offset. Chirality is more noticeable the smaller the sensor and the shorter the lens. So on smart phones, which put tiny sensors behind wide-angle lenses, they have to get the offset just right. This explains why the lenses are in slightly different places on the body every time Apple updates their cameras.

TazeTSchnitzel 2y ago

The body text is really uncomfortably sycophantic.

mrandish 2y ago
Yes, and it's maddeningly detail-free. Just vague claims that something wonderful is being accomplished with zero data or specifics.
- pierrefermat1 2y ago
  The reality is journalists don't really have the knowledge to explain any specifics at all, so you just get this fluff. But hey at least there's pretty pictures

cynicalsecurity 2y ago

Photos on my wife's camera look slightly deformed. The people don't look like themselves, their faces are just slightly off. I'm blaming the AI inside of the photos app on iPhone, but I'm not sure. People's face look perfect on my mediocre Android though. I'm never going to buy or use an iPhone. Even the questionable advantage which was supposed to be the iPhone's camera is fake.

gazchop 2y ago
This is probably a combination of the lens corrections, the pretty awful auto white balance (warmth), the terrible oversharpening and also a bit of True Tone. Portrait mode also wrecks a lot of photos due to the crappy emulation of DOF. Output is clever but shit. Due to the general flatness of the lens there is a lot of distortion around the edges which is digitally removed after the photo is taken. This isn't 100% perfect and causes some rather uncanny looks in some of the photos. You can use this for artistic effect but it looks crap mostly. Generally if you're using a proper camera there's a big chunk of glass in front of it so the main part of a portrait is well outside the distorted edges of the frame so it's not noticeable. Even new cameras use minimal lens corrections in body as well to eliminate this. As for the white balance, Apple never seem to get this right. The colours are always slightly too orange / warm and vivid and never quite match reality in experience. You can crank the warmth down a bit after in photos.app to kill some of it. Oversharpening - everything is too sharp. This makes the image pop out but nothing more. It's a terrible curse on smartphones. Not much you can do about it. Even shooting ProRAW on mine is oversharpened. If you turn the True Tone feature off in Display/Settings it looks a bit better as well. That seems to completely mung viewing any photos later, giving them sometimes an over-blue tone. Urgh all this is why I bought a mirrorless. Smartphones are really not very good. Even good ones (mine is a 15 Pro). Mine gets mostly used to take photos of an AirBnB when I leave it now or where the car got parked.
- amluto 2y ago
  My bonus pet peeve about portrait mode is that the internet is full of portrait mode photos, which means AI gets trained on portrait mode photos, which means AI generates pictures that look like portrait mode. Garbage in, garbage out.

harha_ 2y ago

I wonder how much that wire mesh floor distorts the recording result? I guess it must be insignificant, since the walls, ceiling and floor absorb almost all reflected sound waves?

jdietrich 2y ago
Sound waves are really long. 20kHz is about the limit of human hearing, so the shortest wave we can hear is about 17cm; the longest wave we can hear is about 17 meters. A suitably designed mesh will be effectively transparent across that frequency range.

orev 2y ago

> 38% of people said that better cameras are a main motivation for buying a new phone This strikes me as just a reflection of the ad campaigns. Apple promotes “better cameras” for every new iPhone, almost exclusively in their ads, so it’s not surprising that’s what people would say. With every new phone being just an incremental upgrade, hyping up the camera is the only way to get people to drop a $1000 on a new one. Most of these 38% won’t be able to tell the difference between a phone pic taken 5 or more years ago.

esafak 2y ago
Better images got people to buy even more expensive digital cameras that do less than your cell phone. It turns out people like to take pictures.
- giancarlostoro 2y ago
  Do less in terms of what exactly? Because I have photos from old digital cameras from nearly 20 years ago that look drastically better and more detailed than Android or iPhone photos from several years back. I would be surprised if the quality is down. In terms of features the quality is key.
  - esafak 2y ago
    Do less in terms of not being able to do anything beside take a picture. And they don't do any computational photography, either, so their pictures are not necessarily better. Try any low light photography with moving objects on your camera? Or adjust the focal point? Cell phones can do it. If digital cameras were so good they would not be disappearing.
    - orev 2y ago
      Phone cameras have come a long way, no doubt, but DSLRs really do look amazing. Phones are winning because “the best camera is the one that’s with you”. Since people have the phone camera in their pocket all the time, they usually don’t bother to carry a separate bulky one except for special occasions.

a1o 2y ago

One thing about the iPhone microphone. I had a beautiful day the other day here with some light drizzle and the sea waves were rocking nicely it was a great sound and the beach was empty. Great feeling. I wanted to record this but I only had my iPhone - was just walking in the beach. So neither the default recording app of the iphone and neither any of the ones I tried, could capture the ocean waves or the sound of the drizzle. I had previously (intentionally) set out to the beach with the intent to capture audio and did so with my Laptop + external mic. My conclusion is that unfortunately that is the way it has to be, and that it's not possible to capture audio on a whim as is with photos.

pryelluw 2y ago
Which iPhone model?
- a1o 2y ago
  The experience I described was with the 14 but I also tried in the 15 pro something similar and had same experience. For now my approach is to use a external microphone with a longer cable and a notebook, and this works, but it would be nice to make this work with the iPhone.
  - pryelluw 2y ago
    I daily an (newest model) SE and got some wireless lavalier mics that plug into the lighting port. They work well and were cheap. Have to carry them around. Not name brand or anything. Though I hope to get some from rode. What do you use for storage?
    - a1o 2y ago
      I don't use the phone productively, everything I tried I endup concluding it can't do? So it's only use is complaining about things in hacker news for killing time. The most recent idea was trying to use it for sounds but I had the mentioned issue. Does the lavalier works for environment sounds or you only use for voice?