7 ms·

New iPhone app identifies trees from photos of their leaves

180 points by gmac 16y ago

ataggart 16y ago
Having surpassed Star Trek's communicator, we seem to be well on our way towards the tricorder.
- Groxx 16y ago
  Nah, I think we'll have that as soon as someone wires up a camera to Mechanical Turk. "Looks like cancer." "Definitely Athlete's Foot." "It's not a tumor!" "Please state the nature of the medical emergency." Repeat 1000x, pull out the winners, and diagnose!
  - bh42222 16y ago
    Repeat 1000x, pull out the winners, and diagnose! 1. If consistent winners can be identified, I'd expect them to demand much higher then average pay. 2. Everyone of your turks might have to be licensed to practice medicine, or the government might put at least some of you in jail. I know of at least a few melanoma identification, using software, efforts which died due to regulation and the shocking resistance of doctors. But I wonder how free apps, good at diagnosing all kind of things, would do? With cellphone CPUs and cameras getting better you could catch quite a few things...
    - VladRussian 16y ago
      similar to already working scheme in the pathology - the hi-def microscope scans are send to India, analyzed by real professionals there, and the results are signed off by professional here >I know of at least a few melanoma identification, using software, efforts which died due to regulation and the shocking resistance of doctors. Just make sure that "For entertainment only" is clearly visible on the product label :) >But I wonder how free apps, good at diagnosing all kind of things, would do? With cellphone CPUs and cameras getting better you could catch quite a few things... and with portable ECG, EEG, ultrasound devices, DNA chips, etc easy available and connectable ...
- epochwolf 16y ago
  I don't think we quite have Star Trek communicators right. At least not the ones from Next Generation. Those seemed to be plot activated. You didn't always need to hit a button for them to make a call. Though they did drop calls like our modern smart phones.
phlux 16y ago
It would be great if you could take a pic of a tree leaf, and i will capture the geoloc of the pic as well and plot where that tree is on the map - and start tracking where all types of trees live.
- lsb 16y ago
  Even greater to use hi-res satnav for that! We can assess water health that way, it'd be fascinating to do similarly for forests.
  - phlux 16y ago
    Get kids/students to participate in snapping a pic of a leaf any opportunity they get to feed the data into the system. Reward them somehow. Ask one simple question with each pic take "How many trees of this type would you estimate are in your vacinity, that you can see" Map those trees all over and compare with climate/weather. The estimates just increase the size of the circle on the map that shows where those trees live. Eventually you'll get a very clear idea of what trees thrive/live where.
- apu 16y ago
  This is exactly what we are doing. We're working with botanists at the Smithsonian Institution in DC who are very interested in this kind of data.
  - andrewgleave 16y ago
    Great app. Next week I'm releasing a project for the Isle of Man which crowdsources the recording of location and health of Elm trees (via iPhone and Android app). The Island is one of the very few locations in the UK which still have a healthy population but has recently suffered funding cuts meaning Gov. surveys are no longer being done, and hence the need for a public solution. If the app could recognise more species of elm (English and American seem to be there, but no Wych or Dutch for example), would be a valuable tool for users trying to contribute to the project. The source for the Django app and PhoneGap-based mobile app will be up on GitHub next week, and the data is publicly available via our CouchOne instance (http://redrobot.couchone.com/_utils/database.html?openelm http://redrobot.couchone.com/_utils/database.html?openelm) – the project isn't live yet so there's only a small amount of data.
zumbojo 16y ago
A NYT article during development in 2009: https://www.nytimes.com/2009/05/10/business/10novel.html https://www.nytimes.com/2009/05/10/business/10novel.html
xutopia 16y ago
That's funny I wanted something similar to determine it based on the trunk of each tree. In Canada there are no leafs on trees half of the year and you can make some interesting edible syrups with certain trees.
- morganpyne 16y ago
  Being into woodworking (lutherie specifically), I would kill for something that could identify species by samples of wood. I think this is a very different task and extremely difficult to achieve though. Sometimes even 2 experts disagree on the pedigree of a given piece of wood - visual inspection is simply not enough and nothing short of a detailed lab analysis can confirm some identifications. Variation of the wood from tree to tree of exactly the same species is large, at least you are dealing with relative consistency in the leaves.
apu 16y ago
This is my project =) Leafsnap is a project I've been working on for about 2 years and we finally launched this week! It's a free app for the iPhone (with an iPad version coming next week and an Android version later in the summer) that contains very high-quality images and descriptions of the tree species of the Northeast US. It also has an automatic-recognition feature that uses computer vision to help users identify trees by taking a photo of a leaf. This project is a joint collaboration between the computer science departments at Columbia University and the University of Maryland, and botanists at the Smithsonian Institution. Please let me know what you think! Also, I'd be happy to answer any questions about the app or the recognition technology behind it.
- phlux 16y ago
  Can you just tell us what the recognition tech behind it is?
  - apu 16y ago
    Sure! There are 3 main pieces: 1. Segment the leaf from the background. Even though we require users to place the leaf on a white background, this is still a very tough problem because of lighting, shadows, viewpoint, blur, focus, etc. We've tried all published segmentation algorithms and nothing works. We wrote our own, using EM on the HSV colorspace as the basis for it (plus lots of heuristics for leaves). It works okay, but it definitely needs the most work. 2. Extract some features from the contour of the leaf. We use histograms of curvature over scale. The idea is that some leaves have the same rough shape, but differ at a fine scale (e.g., serrated vs. smooth leaves). Others are similar at a fine-scale but look quite different overall. To distinguish all of these cases, we estimate "curvature at a scale" at each point on the contour and build a histogram of curvature values. We then repeat this at coarser and coarser scales and concatenate all histograms together to get a large feature vector. 3. Recognition/matching: We use a simple nearest-neighbors classifier to match the feature vectors to our ground-truth database of labeled leaves. We return the top 20 ranked species or so.
    - 6ren 16y ago
superchink 16y ago
I can't seem to get to the site. Requests are timing out. HN effect?
- apu 16y ago
  I wish -- but it was actually a Guardian article and an intomobile article that knocked us out. We're back now! http://www.guardian.co.uk/environment/blog/2011/may/06/iphone-app-wildlife-species-recognition http://www.guardian.co.uk/environment/blog/2011/may/06/iphon... http://www.intomobile.com/2011/05/06/leafsnap-iphone-app-lets-you-id-trees-camera/ http://www.intomobile.com/2011/05/06/leafsnap-iphone-app-let...
ksolanki 16y ago
Very nice to see a good computer vision app make it to the market. This has certainly brought the future closer to us :)
eiji 16y ago
Maybe a stupid question: Why isn't it open source?
- deleted 16y ago
  [deleted]
eekfuh 16y ago
Does this use any type of location lookup? I wonder how beneficial that would be, either look at the persons location if they use the cam or grab it out of the exif data, so that you can better determine the tree by where it of could be grown.
- apu 16y ago
  Not right now, but we store the GPS with all images. Once we expand our coverage to more of the world, we will probably start using the GPS to help narrow down the list of possibilities. However, the app does let you filter the list of species shown to "New York" or "Washington DC" or "The Northeast".
kirpekar 16y ago
Superb -- fun app while hiking with my kid. I grew up in another country, so a lot of trees where I live are unknown to me. Thanks
wallflower 16y ago
Love the About page. Nice personal connection. http://leafsnap.com/about/ http://leafsnap.com/about/ Ah memories of my 10th grade biology class (bug or leaf collection). The trusty Audubon Guide to North American Trees. It is really shocking that the HGTTG is coming true...
willwagner 16y ago
Awesome app. I can't wait to use it on our next hike. I really wish there were more apps like this (and Shazam). Just off of the top of my head, i'd pay a reasonable amount of money to identify: bugs, fish, seashells, animal footprints, mushrooms, and I'm sure a whole lot more. Being part of a hiking family with two small kids, the questions are endless and even if it were crowdsourced in some way (e.g. please identify this bug for karma), it could be handy learning tool for kids, getting them more excited about science at an early age.
mooneater 16y ago
This is a fairly technical crowd, can you tell us more about how the algorithm works?
- apu 16y ago
  http://news.ycombinator.com/item?id=2522523 http://news.ycombinator.com/item?id=2522523
jenniart 15y ago
I really adore your about page with all of the photos of your team with leaves. Very fun and personal. :)
coldarchon 15y ago
I'm just happy you didn't make one for insects. Just imagine an app that doesn't recognize a running bug and the owner of the iPhone kills it after 3rd try to get a better result ..
pshapiro 15y ago
Wow, I was just thinking about how to accomplish this idea yesterday.

180 points by gmac 16y ago

ataggart 16y ago

Having surpassed Star Trek's communicator, we seem to be well on our way towards the tricorder.

Groxx 16y ago
Nah, I think we'll have that as soon as someone wires up a camera to Mechanical Turk. "Looks like cancer." "Definitely Athlete's Foot." "It's not a tumor!" "Please state the nature of the medical emergency." Repeat 1000x, pull out the winners, and diagnose!
- bh42222 16y ago
  Repeat 1000x, pull out the winners, and diagnose! 1. If consistent winners can be identified, I'd expect them to demand much higher then average pay. 2. Everyone of your turks might have to be licensed to practice medicine, or the government might put at least some of you in jail. I know of at least a few melanoma identification, using software, efforts which died due to regulation and the shocking resistance of doctors. But I wonder how free apps, good at diagnosing all kind of things, would do? With cellphone CPUs and cameras getting better you could catch quite a few things...
  - VladRussian 16y ago
    similar to already working scheme in the pathology - the hi-def microscope scans are send to India, analyzed by real professionals there, and the results are signed off by professional here >I know of at least a few melanoma identification, using software, efforts which died due to regulation and the shocking resistance of doctors. Just make sure that "For entertainment only" is clearly visible on the product label :) >But I wonder how free apps, good at diagnosing all kind of things, would do? With cellphone CPUs and cameras getting better you could catch quite a few things... and with portable ECG, EEG, ultrasound devices, DNA chips, etc easy available and connectable ...
epochwolf 16y ago
I don't think we quite have Star Trek communicators right. At least not the ones from Next Generation. Those seemed to be plot activated. You didn't always need to hit a button for them to make a call. Though they did drop calls like our modern smart phones.

phlux 16y ago

It would be great if you could take a pic of a tree leaf, and i will capture the geoloc of the pic as well and plot where that tree is on the map - and start tracking where all types of trees live.

lsb 16y ago
Even greater to use hi-res satnav for that! We can assess water health that way, it'd be fascinating to do similarly for forests.
- phlux 16y ago
  Get kids/students to participate in snapping a pic of a leaf any opportunity they get to feed the data into the system. Reward them somehow. Ask one simple question with each pic take "How many trees of this type would you estimate are in your vacinity, that you can see" Map those trees all over and compare with climate/weather. The estimates just increase the size of the circle on the map that shows where those trees live. Eventually you'll get a very clear idea of what trees thrive/live where.
apu 16y ago
This is exactly what we are doing. We're working with botanists at the Smithsonian Institution in DC who are very interested in this kind of data.
- andrewgleave 16y ago
  Great app. Next week I'm releasing a project for the Isle of Man which crowdsources the recording of location and health of Elm trees (via iPhone and Android app). The Island is one of the very few locations in the UK which still have a healthy population but has recently suffered funding cuts meaning Gov. surveys are no longer being done, and hence the need for a public solution. If the app could recognise more species of elm (English and American seem to be there, but no Wych or Dutch for example), would be a valuable tool for users trying to contribute to the project. The source for the Django app and PhoneGap-based mobile app will be up on GitHub next week, and the data is publicly available via our CouchOne instance (http://redrobot.couchone.com/_utils/database.html?openelm http://redrobot.couchone.com/_utils/database.html?openelm) – the project isn't live yet so there's only a small amount of data.

zumbojo 16y ago

A NYT article during development in 2009: https://www.nytimes.com/2009/05/10/business/10novel.html https://www.nytimes.com/2009/05/10/business/10novel.html

xutopia 16y ago

That's funny I wanted something similar to determine it based on the trunk of each tree. In Canada there are no leafs on trees half of the year and you can make some interesting edible syrups with certain trees.

morganpyne 16y ago
Being into woodworking (lutherie specifically), I would kill for something that could identify species by samples of wood. I think this is a very different task and extremely difficult to achieve though. Sometimes even 2 experts disagree on the pedigree of a given piece of wood - visual inspection is simply not enough and nothing short of a detailed lab analysis can confirm some identifications. Variation of the wood from tree to tree of exactly the same species is large, at least you are dealing with relative consistency in the leaves.

apu 16y ago

This is my project =) Leafsnap is a project I've been working on for about 2 years and we finally launched this week! It's a free app for the iPhone (with an iPad version coming next week and an Android version later in the summer) that contains very high-quality images and descriptions of the tree species of the Northeast US. It also has an automatic-recognition feature that uses computer vision to help users identify trees by taking a photo of a leaf. This project is a joint collaboration between the computer science departments at Columbia University and the University of Maryland, and botanists at the Smithsonian Institution. Please let me know what you think! Also, I'd be happy to answer any questions about the app or the recognition technology behind it.

phlux 16y ago
Can you just tell us what the recognition tech behind it is?
- apu 16y ago
  Sure! There are 3 main pieces: 1. Segment the leaf from the background. Even though we require users to place the leaf on a white background, this is still a very tough problem because of lighting, shadows, viewpoint, blur, focus, etc. We've tried all published segmentation algorithms and nothing works. We wrote our own, using EM on the HSV colorspace as the basis for it (plus lots of heuristics for leaves). It works okay, but it definitely needs the most work. 2. Extract some features from the contour of the leaf. We use histograms of curvature over scale. The idea is that some leaves have the same rough shape, but differ at a fine scale (e.g., serrated vs. smooth leaves). Others are similar at a fine-scale but look quite different overall. To distinguish all of these cases, we estimate "curvature at a scale" at each point on the contour and build a histogram of curvature values. We then repeat this at coarser and coarser scales and concatenate all histograms together to get a large feature vector. 3. Recognition/matching: We use a simple nearest-neighbors classifier to match the feature vectors to our ground-truth database of labeled leaves. We return the top 20 ranked species or so.
  - 6ren 16y ago

superchink 16y ago

I can't seem to get to the site. Requests are timing out. HN effect?

apu 16y ago
I wish -- but it was actually a Guardian article and an intomobile article that knocked us out. We're back now! http://www.guardian.co.uk/environment/blog/2011/may/06/iphone-app-wildlife-species-recognition http://www.guardian.co.uk/environment/blog/2011/may/06/iphon... http://www.intomobile.com/2011/05/06/leafsnap-iphone-app-lets-you-id-trees-camera/ http://www.intomobile.com/2011/05/06/leafsnap-iphone-app-let...

ksolanki 16y ago

Very nice to see a good computer vision app make it to the market. This has certainly brought the future closer to us :)

eiji 16y ago

Maybe a stupid question: Why isn't it open source?

deleted 16y ago
[deleted]

eekfuh 16y ago

Does this use any type of location lookup? I wonder how beneficial that would be, either look at the persons location if they use the cam or grab it out of the exif data, so that you can better determine the tree by where it of could be grown.

apu 16y ago
Not right now, but we store the GPS with all images. Once we expand our coverage to more of the world, we will probably start using the GPS to help narrow down the list of possibilities. However, the app does let you filter the list of species shown to "New York" or "Washington DC" or "The Northeast".

kirpekar 16y ago

Superb -- fun app while hiking with my kid. I grew up in another country, so a lot of trees where I live are unknown to me. Thanks

wallflower 16y ago

Love the About page. Nice personal connection. http://leafsnap.com/about/ http://leafsnap.com/about/ Ah memories of my 10th grade biology class (bug or leaf collection). The trusty Audubon Guide to North American Trees. It is really shocking that the HGTTG is coming true...

willwagner 16y ago

Awesome app. I can't wait to use it on our next hike. I really wish there were more apps like this (and Shazam). Just off of the top of my head, i'd pay a reasonable amount of money to identify: bugs, fish, seashells, animal footprints, mushrooms, and I'm sure a whole lot more. Being part of a hiking family with two small kids, the questions are endless and even if it were crowdsourced in some way (e.g. please identify this bug for karma), it could be handy learning tool for kids, getting them more excited about science at an early age.

mooneater 16y ago

This is a fairly technical crowd, can you tell us more about how the algorithm works?

apu 16y ago
http://news.ycombinator.com/item?id=2522523 http://news.ycombinator.com/item?id=2522523

jenniart 15y ago

I really adore your about page with all of the photos of your team with leaves. Very fun and personal. :)

coldarchon 15y ago

I'm just happy you didn't make one for insects. Just imagine an app that doesn't recognize a running bug and the owner of the iPhone kills it after 3rd try to get a better result ..

pshapiro 15y ago

Wow, I was just thinking about how to accomplish this idea yesterday.