7 ms·

Smartphone Cameras Go Hyperspectral

152 points by voxadam 1y ago

fkyoureadthedoc 1y ago
> The new patent-pending technique > “Every photo carries hidden spectral information waiting to be uncovered. By extracting it, we can turn everyday photography into science.” And with our patent, extract rent from anyone who wants to do it!
- moritonal 1y ago
  That's a bit of a bad faith take. You were welcome to go spend the years(?) this chaps dedicated to putting together the research required to build this. If it works, let him enjoy the fruits of labour.
  - SequoiaHope 1y ago
    Patents do more than let you enjoy the fruits of your labor - the market already allows for that. Patents use the force of law to bar anyone else who might have discovered the same thing from building upon it.
    - spookie 1y ago
      Imagine you are just a dude, you did all this work, and go to "market". You are just a dude, therefore business grows slowly. You gather enough attention that some corporation with a lot of bling just goes and copies your thing. Your business fails.
      - fkyoureadthedoc 1y ago
        On the other hand those same corporations can generate, file, and litigate more patents than just a dude could ever hope to. It's 2007. Just-a-dude has a great idea, he notices customers to his website often buy just one item, so he'll let them do that with one simple click. What's this, he's just received a cease and desist? Sorry bro, Amazon patented that 10 years ago.
      - SequoiaHope 1y ago
        Once can just as trivially construct an argument demonstrating the issue with patents but the problem with this style of argument is that patents are not a simple thing. They have global far reaching effects. The government distributing a monopoly on information is a serious interference with the market, and due to patent harmonization efforts across the world, one person filing a patent in New Jersey affects even people in Kenya and Turkey and Thailand. The arguments for patents are often, as I see it, based on a deeply flawed understanding of the motivations of innovators and the affects of open information on innovation. For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.
        fluoridation 1y ago
        >For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong. Can you clarify this? Just curious about what you mean.
        brookst 1y ago
        I mean it sure sounds like the old “science can’t explain how bumblebees fly, therefore science is incomplete or outright wrong” argument. Which is of course just false.
        SequoiaHope 1y ago
        I replied so feel free to read that.
        SequoiaHope 1y ago
        I often speak to people who say that no one would innovate without government-secured monopoly on information, and this is clearly false as open source works without this monopoly. As such it seems to me that people who make that claim don’t understand how innovation works. Often I say that if we phased out patents and other IP restrictions, then investments would not stop but they would change from less frequent large investments to more frequent small investments. As long as designs can stay secret until release, there will always be first mover advantage and brand recognition. But you might get smaller investments to build out manufacturing for the next year, rather than bigger longer term investments. The flip side is that stagnant innovators who got lucky once will be subsumed by more agile competitors who can better deliver those innovations to market. Thus investment and innovation wouldn’t stop - markets still ensure certain advantages for innovators - but the nature of investment and innovation wouldn’t shift towards more incremental moves and more diverse actors. A major upside to this is that those best suited to scale an existing nascent technology would be free to compete at doing so. It should be noted that even die hard capitalists are against IP restrictions [1] as they are a massive government investment in the market. So proponents of IP restrictions must reconcile their arguments in favor of this government intervention with their potential interest in free markets. [1] https://youtu.be/GZgLJkj6m0A https://youtu.be/GZgLJkj6m0A
      - Forgeties79 1y ago
        I mean we’re basically getting the same result. Tons of businesses, not to mention patent trolls, constantly harass individuals and small businesses trying to get their foot in the door or just run a small, sustainable business. Hell forget my business failing, it’s possible I’ll never even get to try my idea out!
      - serf 1y ago
        you have it entirely backwards; patents dont protect just-a-dude, they protect the corporation. how? just-a-dude doesn't have a team of patent attorneys sitting in his back office waiting for work.
  - dtj1123 1y ago
    Sure, if he'd come up with this primarly using his own resources and time, but he discovered this whilst being paid to conduct research at a public university, a form of institution which is explicitly intended to disseminate knowledge. Society should enjoy the fruits of its investment.
    - brookst 1y ago
      Would the university have been willing to invest in the exotic printer and the labor to do the work without the potential upside of a patent? It’s very easy to declare that someone else “should” do a bunch of work and spend a bunch of money for the altruistic benefit of society.
  - OutOfHere 1y ago
    There is not much labor here. Anyone should be able to engineer a model by using deep learning over pictures that map raw images to their hyperspectral variants in various settings, including in adversarial settings that are intended to confuse. All you would need is a sufficiently large and diverse dataset.
- brookst 1y ago
  I hate the new rhetorical use of the word “rent”
  - reaperducer 1y ago
    "Rent-seeking" is square 3B on my HN bingo card. See also: Regulatory capture Late-stage capitalism Walled garden An/The unreasonable... So... Period. Full stop I Mean… Streisand effect orthogonal trivial non-trivial {$person}'s Law | Axiom | Razor | Paradox Objectively Ship of Theseus Gatekeeping
    - brookst 1y ago
      It’s like the opposite of traditional tech buzzword bingo where you raise VC for blockchain crypto AI agents. Instead, anything you don’t like is a rent-seeking late stage capitalism narrative.
  - fkyoureadthedoc 1y ago
    I seriously doubt using the word rent in reference to patent license royalties is new
slwvx 1y ago
I was hoping that someone came out with a camera that not only had not only sensors for visible light, but for infrared and UV. It's just another color to add to the sensors; I think we have enough megapixels, seems like going for other bands is reasonable.
- Scene_Cast2 1y ago
  I have a OnePlus 8 Pro with an IR camera. It's pretty nifty - nature photography looks cool, seeing through stovetops is neat (and seeing when they heat up), and VR things are also often playing around with IR (plastic transparent to IR, IR LEDs, etc). I ended up having to flash Lineage, as there was some outrage that in a highly limited set of circumstances, thin see-through T-shirts became slightly more see-through and OnePlus disabled that camera in their later firmware updates.
  - chankstein38 1y ago
    You have to love when amazing innovations disappear just in case the lowest-quality rung of our society might misuse something... I'm pretty sick of being ruled based on the lowest common denominator.
- sirtaknt 1y ago
  Some phones had near-IR camera (Pixel 4, Samsung S10) accessible via API. No "killer app" was found since then, 5+ years
  - ACCount37 1y ago
    iPhones also have a near-IR front camera, but that one is fully slaved to the FaceID system. Don't think anything in userland can access raw data from it.
    - grgergo 1y ago
      There are lots of 3D scanning apps using "Face ID", like Heges: https://hege.sh/ https://hege.sh/
sirtaknt 1y ago
I don't understand how from 3 independent values per pixel (RGB) they claim to derive 200+ independent values per pixel. Unless they are assuming a smooth "image" (all pixels the same RGB), perturbed only by the color card? Not exactly a camera then
- ladberg 1y ago
  They're not claiming to get that many values per pixel, they're getting that many values overall for the medium through which light passes between the card and the phone. The idea light comes from a source (e.g. sun), bounces off the various colors of the card and thus produces hundreds of different spectra, those all pass through a medium, and land on the phone camera. So you're getting one measurement consisting of hundreds of RGB values that each represent intensity of different spectra, and you combine it all together to get a single spectrogram.
- ted_dunning 1y ago
  Did you read the actual article. They go into the method they use at length.
esafak 1y ago
This could improve chromatic adaptation of captured images. In other words, better results when changing the white point.
- chaboud 1y ago
  Funny enough, that’s what photographers are doing when they shoot a color checker chart (e.g., Munsell, Macbeth, X-Rite). White balance is hard, in part, because the sensitivity bands of our vision and the camera sensors do not align. Take a look at fluorescent (or, better still, sodium vapor) light spectra for clarity on why this is a massive pain.
mvhv 1y ago
This doesn't really seem like "hyperspectral imaging". I think the idea is having a reference colour chart of known emission characteristics and photographing it through a transparent substance gives you an idea of how much that substance attenuates each wavelength. It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?
- crazygringo 1y ago
  In theory maybe you could build a version made of inks printed on a reflective mirror? And then you would hold the mirror so it reflected the object into the camera? But that seems far more difficult. Precisely combining and applying combinations of inks to a mirrored surface sounds like a helluva manufacturing challenge.
  - mvhv 1y ago
    I think ink and mirrors are kind of fundamentally incompatible. Probably closer to what you're thinking about would be putting a bunch of tiny bandpass filters infront of a mirror, but in that case you can ditch the mirror entirely and just point the camera through the filter array. A filter array right on-top of the sensor is how (the vast majority) of) CMOS cameras distinguish colour anyway.
- atwrk 1y ago
  On top of that it only works in the VIS range, thanks to the filters in front of the camera sensors - and most of the interesting information is in UV or IR. VIS only contains information about a few elements. (see Fraunhofer lines https://en.wikipedia.org/wiki/Fraunhofer_lines https://en.wikipedia.org/wiki/Fraunhofer_lines)
aDyslecticCrow 1y ago
This is really cool and very clever. But i want to raise one thing. > designed a special color reference chart that can be printed on a card My rudimentary understanding of physics makes me suspect this sentence is a simplification. A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do. So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands. I suspect alot of effort went into finding, mixing and algoritmically combining the right inks. I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp) Insanely clever. (Whatever they did)
- kelseyfrog 1y ago
  Not just that, but it would presumably be sensitive to light emission spectra too. As inks can only reflect wavelengths of light that hit them, if the emission spectra has spikes or gaps - think LED or florescent - the reflected spectra will be a function of the light source[1]. Perhaps there's some accounting for this, and I'm curious to learn what it is, because it's a phenomenally complex problem. 1. You might think the sun is a standard source, but it's usually modulated by the atmosphere[2]. 2. Unless you are in space.
  - aDyslecticCrow 1y ago
    > Perhaps there's some accounting for this, and I'm curious to learn what it is The slip itself is a calibration reference, so a clean photo of it could serve to compensate for the lamp and camera and calculate how accurate the readings is for different parts of the spectrum. (But good wide spectrum light would be ideal for high precision readout) You're also still limited to visible light because of the camera uv and ir filter, for which the sun is a decent reference.
    - kelseyfrog 1y ago
unglaublich 1y ago
Patent-pending... again someone trying to rent-seek a high-school physics fair idea. Measuring light absorption with a camera is almost as old as the camera. Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique. What "investment" is patent law protecting here?
- aDyslecticCrow 1y ago
  What you're referring to is color calibration. This is spectroscopy. This is likley more of a chemistry paper than a engineering paper because the ink in the reference chart is doing some heavy lifting.
linsomniac 1y ago
Someone needs to build a phone that is leaning towards a tricorder; I'd buy that for myself and my kids. My Pixel 10 has a temp sensor on it, which is cool, but I've had minimal use so far. I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).
- shrike 1y ago
  Take a look at the phyphox app - https://phyphox.org/ https://phyphox.org/
  - horacemorace 1y ago
    Wow! That app is amazing thanks for sharing
  - dtj1123 1y ago
    This is brilliant.
  - linsomniac 1y ago
    Also Arduino Science Journal: https://www.arduino.cc/education/science-journal https://www.arduino.cc/education/science-journal
- lawlessone 1y ago
  Caterpillar has a smartphone with a thermal camera. The price isn't far off the the price of the most expensive smartphones https://cat.smartwalkie.com/store/products/cats62pro https://cat.smartwalkie.com/store/products/cats62pro
  - dylan604 1y ago
    how does Cat's self repair policy compare to John Deere's? Then again, it's not far off from Apple's
  - eichin 1y ago
abeppu 1y ago
I notice the article doesn't say anything about accuracy. This is not my area, but I think the _other_ hacky way to try to do spectroscopy with a phone is with a diffraction grating (and maybe a box with a slit in it). Diffraction gratings are cheap, probably not so different from a specially-printed reference card. If you have a choice, which is better?
- csmoak 1y ago
  diffraction grating wouldnt give you a controlled lighting environment (illuminant). they seem to handle that issue here by using a known spectral reference chart which might let them handle any normal lighting environment.
  - abeppu 1y ago
    I would think in the same environment you would take images immediately before and after adding the sample.
- ted_dunning 1y ago
  The actual article does say quite a bit about accuracy.
docfort 1y ago
More specific info on the reference card is available in the paper's supplemental information. https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1-3597038.pdf?arnumber=11125864 https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1.... Basically, they used special paper with a pro-Canon inkjet, along with a special ICC color profile.
DoctorOetker 1y ago
Every picture contains noise. Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one). It makes one wonder how much of the noise is actually misinterpreted signal.
ChrisRR 1y ago
Hyperspectral imaging see things the human eye can't: Sure Custom processors to accelerate generation of AI text: Go ahead Slightly thicker to fit a bigger battery in: How dare you
danbruc 1y ago
I do not understand how this could possibly work. From a camera with RGB filters we get essentially three different integrals over the spectrum per pixel, how would we recover the spectrum from that? Even assuming you can account for the spectrum of the light sources and the color filters in the camera - which should be doable with a color chart with known spectra - I do not see how you could go from three data points to a full spectrum without making assumptions about the possible object spectra. EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.
mikewarot 1y ago
This is a combination of things that almost breaks my brain, but it works, and it's brilliant! First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time. Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence. The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable. Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure. [1] https://www.youtube.com/watch?v=_ezhdhHNku0 https://www.youtube.com/watch?v=_ezhdhHNku0 [2] https://en.wikipedia.org/wiki/Coded_aperture https://en.wikipedia.org/wiki/Coded_aperture [3] https://en.wikipedia.org/wiki/Fourier_transform https://en.wikipedia.org/wiki/Fourier_transform [4] https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllciB0cmFuc2Zvcm0%3D https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllc... [5] https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZmlsdGVy https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZ... [6] https://en.wikipedia.org/wiki/Bayer_filter https://en.wikipedia.org/wiki/Bayer_filter

152 points by voxadam 1y ago

> The new patent-pending technique > “Every photo carries hidden spectral information waiting to be uncovered. By extracting it, we can turn everyday photography into science.” And with our patent, extract rent from anyone who wants to do it!

moritonal 1y ago
That's a bit of a bad faith take. You were welcome to go spend the years(?) this chaps dedicated to putting together the research required to build this. If it works, let him enjoy the fruits of labour.
- SequoiaHope 1y ago
  Patents do more than let you enjoy the fruits of your labor - the market already allows for that. Patents use the force of law to bar anyone else who might have discovered the same thing from building upon it.
  - spookie 1y ago
    Imagine you are just a dude, you did all this work, and go to "market". You are just a dude, therefore business grows slowly. You gather enough attention that some corporation with a lot of bling just goes and copies your thing. Your business fails.
    - fkyoureadthedoc 1y ago
      On the other hand those same corporations can generate, file, and litigate more patents than just a dude could ever hope to. It's 2007. Just-a-dude has a great idea, he notices customers to his website often buy just one item, so he'll let them do that with one simple click. What's this, he's just received a cease and desist? Sorry bro, Amazon patented that 10 years ago.
    - SequoiaHope 1y ago
      Once can just as trivially construct an argument demonstrating the issue with patents but the problem with this style of argument is that patents are not a simple thing. They have global far reaching effects. The government distributing a monopoly on information is a serious interference with the market, and due to patent harmonization efforts across the world, one person filing a patent in New Jersey affects even people in Kenya and Turkey and Thailand. The arguments for patents are often, as I see it, based on a deeply flawed understanding of the motivations of innovators and the affects of open information on innovation. For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong.
      - fluoridation 1y ago
        >For example most arguments in favor of patents cannot explain how open source works, and so are clearly incomplete or outright wrong. Can you clarify this? Just curious about what you mean.
        brookst 1y ago
        I mean it sure sounds like the old “science can’t explain how bumblebees fly, therefore science is incomplete or outright wrong” argument. Which is of course just false.
        SequoiaHope 1y ago
        I replied so feel free to read that.
        SequoiaHope 1y ago
        I often speak to people who say that no one would innovate without government-secured monopoly on information, and this is clearly false as open source works without this monopoly. As such it seems to me that people who make that claim don’t understand how innovation works. Often I say that if we phased out patents and other IP restrictions, then investments would not stop but they would change from less frequent large investments to more frequent small investments. As long as designs can stay secret until release, there will always be first mover advantage and brand recognition. But you might get smaller investments to build out manufacturing for the next year, rather than bigger longer term investments. The flip side is that stagnant innovators who got lucky once will be subsumed by more agile competitors who can better deliver those innovations to market. Thus investment and innovation wouldn’t stop - markets still ensure certain advantages for innovators - but the nature of investment and innovation wouldn’t shift towards more incremental moves and more diverse actors. A major upside to this is that those best suited to scale an existing nascent technology would be free to compete at doing so. It should be noted that even die hard capitalists are against IP restrictions [1] as they are a massive government investment in the market. So proponents of IP restrictions must reconcile their arguments in favor of this government intervention with their potential interest in free markets. [1] https://youtu.be/GZgLJkj6m0A https://youtu.be/GZgLJkj6m0A
    - Forgeties79 1y ago
      I mean we’re basically getting the same result. Tons of businesses, not to mention patent trolls, constantly harass individuals and small businesses trying to get their foot in the door or just run a small, sustainable business. Hell forget my business failing, it’s possible I’ll never even get to try my idea out!
    - serf 1y ago
      you have it entirely backwards; patents dont protect just-a-dude, they protect the corporation. how? just-a-dude doesn't have a team of patent attorneys sitting in his back office waiting for work.
- dtj1123 1y ago
  Sure, if he'd come up with this primarly using his own resources and time, but he discovered this whilst being paid to conduct research at a public university, a form of institution which is explicitly intended to disseminate knowledge. Society should enjoy the fruits of its investment.
  - brookst 1y ago
    Would the university have been willing to invest in the exotic printer and the labor to do the work without the potential upside of a patent? It’s very easy to declare that someone else “should” do a bunch of work and spend a bunch of money for the altruistic benefit of society.
- OutOfHere 1y ago
  There is not much labor here. Anyone should be able to engineer a model by using deep learning over pictures that map raw images to their hyperspectral variants in various settings, including in adversarial settings that are intended to confuse. All you would need is a sufficiently large and diverse dataset.
brookst 1y ago
I hate the new rhetorical use of the word “rent”
- reaperducer 1y ago
  "Rent-seeking" is square 3B on my HN bingo card. See also: Regulatory capture Late-stage capitalism Walled garden An/The unreasonable... So... Period. Full stop I Mean… Streisand effect orthogonal trivial non-trivial {$person}'s Law | Axiom | Razor | Paradox Objectively Ship of Theseus Gatekeeping
  - brookst 1y ago
    It’s like the opposite of traditional tech buzzword bingo where you raise VC for blockchain crypto AI agents. Instead, anything you don’t like is a rent-seeking late stage capitalism narrative.
- fkyoureadthedoc 1y ago
  I seriously doubt using the word rent in reference to patent license royalties is new

slwvx 1y ago

I was hoping that someone came out with a camera that not only had not only sensors for visible light, but for infrared and UV. It's just another color to add to the sensors; I think we have enough megapixels, seems like going for other bands is reasonable.

Scene_Cast2 1y ago
I have a OnePlus 8 Pro with an IR camera. It's pretty nifty - nature photography looks cool, seeing through stovetops is neat (and seeing when they heat up), and VR things are also often playing around with IR (plastic transparent to IR, IR LEDs, etc). I ended up having to flash Lineage, as there was some outrage that in a highly limited set of circumstances, thin see-through T-shirts became slightly more see-through and OnePlus disabled that camera in their later firmware updates.
- chankstein38 1y ago
  You have to love when amazing innovations disappear just in case the lowest-quality rung of our society might misuse something... I'm pretty sick of being ruled based on the lowest common denominator.
sirtaknt 1y ago
Some phones had near-IR camera (Pixel 4, Samsung S10) accessible via API. No "killer app" was found since then, 5+ years
- ACCount37 1y ago
  iPhones also have a near-IR front camera, but that one is fully slaved to the FaceID system. Don't think anything in userland can access raw data from it.
  - grgergo 1y ago
    There are lots of 3D scanning apps using "Face ID", like Heges: https://hege.sh/ https://hege.sh/

sirtaknt 1y ago

I don't understand how from 3 independent values per pixel (RGB) they claim to derive 200+ independent values per pixel. Unless they are assuming a smooth "image" (all pixels the same RGB), perturbed only by the color card? Not exactly a camera then

ladberg 1y ago
They're not claiming to get that many values per pixel, they're getting that many values overall for the medium through which light passes between the card and the phone. The idea light comes from a source (e.g. sun), bounces off the various colors of the card and thus produces hundreds of different spectra, those all pass through a medium, and land on the phone camera. So you're getting one measurement consisting of hundreds of RGB values that each represent intensity of different spectra, and you combine it all together to get a single spectrogram.
ted_dunning 1y ago
Did you read the actual article. They go into the method they use at length.

esafak 1y ago

This could improve chromatic adaptation of captured images. In other words, better results when changing the white point.

chaboud 1y ago
Funny enough, that’s what photographers are doing when they shoot a color checker chart (e.g., Munsell, Macbeth, X-Rite). White balance is hard, in part, because the sensitivity bands of our vision and the camera sensors do not align. Take a look at fluorescent (or, better still, sodium vapor) light spectra for clarity on why this is a massive pain.

mvhv 1y ago

This doesn't really seem like "hyperspectral imaging". I think the idea is having a reference colour chart of known emission characteristics and photographing it through a transparent substance gives you an idea of how much that substance attenuates each wavelength. It's a cool trick if it works, but it seems very finicky and I guess would be limited to transparent/homogeneous liquids?

crazygringo 1y ago
In theory maybe you could build a version made of inks printed on a reflective mirror? And then you would hold the mirror so it reflected the object into the camera? But that seems far more difficult. Precisely combining and applying combinations of inks to a mirrored surface sounds like a helluva manufacturing challenge.
- mvhv 1y ago
  I think ink and mirrors are kind of fundamentally incompatible. Probably closer to what you're thinking about would be putting a bunch of tiny bandpass filters infront of a mirror, but in that case you can ditch the mirror entirely and just point the camera through the filter array. A filter array right on-top of the sensor is how (the vast majority) of) CMOS cameras distinguish colour anyway.
atwrk 1y ago
On top of that it only works in the VIS range, thanks to the filters in front of the camera sensors - and most of the interesting information is in UV or IR. VIS only contains information about a few elements. (see Fraunhofer lines https://en.wikipedia.org/wiki/Fraunhofer_lines https://en.wikipedia.org/wiki/Fraunhofer_lines)

aDyslecticCrow 1y ago

This is really cool and very clever. But i want to raise one thing. > designed a special color reference chart that can be printed on a card My rudimentary understanding of physics makes me suspect this sentence is a simplification. A normal printer use Cyan Magenta Yellow Black to print. A photo of such a print would already destroy alot of spectral information for the same reason the individual rgb sensors do. So i suspect those colored dots are a very careful and deliberate concoction of very particular inks with very specific spectral color bands. I suspect alot of effort went into finding, mixing and algoritmically combining the right inks. I'm guessing it works similarly to a how a narrow band florescent lamp makes only materials that reflect a very specific frequency be visible, which makea alot of prints and pigments look wierd. (If you do the opposite; use ink with very specific spectral band, you can instead measure the lamp) Insanely clever. (Whatever they did)

kelseyfrog 1y ago
Not just that, but it would presumably be sensitive to light emission spectra too. As inks can only reflect wavelengths of light that hit them, if the emission spectra has spikes or gaps - think LED or florescent - the reflected spectra will be a function of the light source[1]. Perhaps there's some accounting for this, and I'm curious to learn what it is, because it's a phenomenally complex problem. 1. You might think the sun is a standard source, but it's usually modulated by the atmosphere[2]. 2. Unless you are in space.
- aDyslecticCrow 1y ago
  > Perhaps there's some accounting for this, and I'm curious to learn what it is The slip itself is a calibration reference, so a clean photo of it could serve to compensate for the lamp and camera and calculate how accurate the readings is for different parts of the spectrum. (But good wide spectrum light would be ideal for high precision readout) You're also still limited to visible light because of the camera uv and ir filter, for which the sun is a decent reference.
  - kelseyfrog 1y ago

unglaublich 1y ago

Patent-pending... again someone trying to rent-seek a high-school physics fair idea. Measuring light absorption with a camera is almost as old as the camera. Using a known reflectance chart in-scene to recover spectral information is a standard calibration technique. What "investment" is patent law protecting here?

aDyslecticCrow 1y ago
What you're referring to is color calibration. This is spectroscopy. This is likley more of a chemistry paper than a engineering paper because the ink in the reference chart is doing some heavy lifting.

linsomniac 1y ago

Someone needs to build a phone that is leaning towards a tricorder; I'd buy that for myself and my kids. My Pixel 10 has a temp sensor on it, which is cool, but I've had minimal use so far. I've always wanted to build a tricorder with my son, was just thinking about it last week when he was putting together a digital compass (with RasPi Nano, magnetic sensor, GPS, and LED light ring + OLED).

shrike 1y ago
Take a look at the phyphox app - https://phyphox.org/ https://phyphox.org/
- horacemorace 1y ago
  Wow! That app is amazing thanks for sharing
- dtj1123 1y ago
  This is brilliant.
- linsomniac 1y ago
  Also Arduino Science Journal: https://www.arduino.cc/education/science-journal https://www.arduino.cc/education/science-journal
lawlessone 1y ago
Caterpillar has a smartphone with a thermal camera. The price isn't far off the the price of the most expensive smartphones https://cat.smartwalkie.com/store/products/cats62pro https://cat.smartwalkie.com/store/products/cats62pro
- dylan604 1y ago
  how does Cat's self repair policy compare to John Deere's? Then again, it's not far off from Apple's
- eichin 1y ago

abeppu 1y ago

I notice the article doesn't say anything about accuracy. This is not my area, but I think the _other_ hacky way to try to do spectroscopy with a phone is with a diffraction grating (and maybe a box with a slit in it). Diffraction gratings are cheap, probably not so different from a specially-printed reference card. If you have a choice, which is better?

csmoak 1y ago
diffraction grating wouldnt give you a controlled lighting environment (illuminant). they seem to handle that issue here by using a known spectral reference chart which might let them handle any normal lighting environment.
- abeppu 1y ago
  I would think in the same environment you would take images immediately before and after adding the sample.
ted_dunning 1y ago
The actual article does say quite a bit about accuracy.

docfort 1y ago

More specific info on the reference card is available in the paper's supplemental information. https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1-3597038.pdf?arnumber=11125864 https://ieeexplore.ieee.org/ielx8/83/10795784/11125864/supp1.... Basically, they used special paper with a pro-Canon inkjet, along with a special ICC color profile.

DoctorOetker 1y ago

Every picture contains noise. Every now and then we discover new traces left by the original light field (by using a fuller picture to model the image formation process, instead of an easier oversimplified one). It makes one wonder how much of the noise is actually misinterpreted signal.

ChrisRR 1y ago

Hyperspectral imaging see things the human eye can't: Sure Custom processors to accelerate generation of AI text: Go ahead Slightly thicker to fit a bigger battery in: How dare you

danbruc 1y ago

I do not understand how this could possibly work. From a camera with RGB filters we get essentially three different integrals over the spectrum per pixel, how would we recover the spectrum from that? Even assuming you can account for the spectrum of the light sources and the color filters in the camera - which should be doable with a color chart with known spectra - I do not see how you could go from three data points to a full spectrum without making assumptions about the possible object spectra. EDIT: Okay, after going to the actual paper I at least get transmission mode - you photograph the color chart through the sample and this will of course imprint the absorption spectrum onto the know spectrum of the color chart and you can then look at the difference to the color chart without the sample in between. But I do not get the logic behind their reflectance mode.

mikewarot 1y ago

This is a combination of things that almost breaks my brain, but it works, and it's brilliant! First, you need to understand coded masks[1,2]. This is a way to use an LCD or other array to mask off parts of a scene so that very specific parts of it are sampled, but the rest aren't, to get a single, high resolution analog value. Then you switch to other masks, and get more values. You can then work backwards in the math through the known mask shapes, to get the original image with far fewer samples that would be required one at a time. Think of the above as a 2d visual version of the Fourier transform[3,4]. This transform is used heavily to compress images throwing away most of the bits in an image without losing it's essence. The analysis they're talking about uses a very specially printed card. It isn't just something generated with a standard 4 ink printer, each "dot" is a separate unique ink with tightly controlled spectral curves, these form a virtual version of the above masks. When you view these through a sample, it can then give an idea of the spectral response of the camera, and the liquid, by using the many different known response curves of the "dots" to work backwards, and generate the 1 dimensional very tight response curve of a hyperspectral imager, by figuring out where each "dot" is in the scene, then averaging that dot's intensity across the RGB values of the picture taken by the camera. Today's cameras have sufficient resolution and bit depth that you get the original bit depth (usually 8 bits) and an additional bit for each doubling of the number of pixels in a given "dot". This is degraded by Bayer pattern filters[5,6], and the nature of cameras, but it's not unrecoverable. Like with Coded Masks, and Fourier transforms, you then take your high resolution analog values, and work backwards to get the things you want to measure. [1] https://www.youtube.com/watch?v=_ezhdhHNku0 https://www.youtube.com/watch?v=_ezhdhHNku0 [2] https://en.wikipedia.org/wiki/Coded_aperture https://en.wikipedia.org/wiki/Coded_aperture [3] https://en.wikipedia.org/wiki/Fourier_transform https://en.wikipedia.org/wiki/Fourier_transform [4] https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllciB0cmFuc2Zvcm0%3D https://www.youtube.com/watch?v=spUNpyF58BY&pp=ygURZm91cmllc... [5] https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZmlsdGVy https://www.youtube.com/watch?v=LWxu4rkZBLw&pp=ygUMYmF5ZXIgZ... [6] https://en.wikipedia.org/wiki/Bayer_filter https://en.wikipedia.org/wiki/Bayer_filter