5 ms·

You can build the next generation bird app. This one requires uploading to the cloud to recognize bird calls. What if you used the AI hardware in the phone to

You can build the next generation bird app. This one requires uploading to the cloud to recognize bird calls. What if you used the AI hardware in the phone to do the audio recognition? For efficiency you could even use geolocation and figure out which species are found in a location and download a model just for those. Anything not matched could be uploaded as before.

gregsadetsky 5y ago
I asked a veterinarian friend who specializes in birds (and is a long time bird watcher), and she knew the app and said that it works pretty well (coming from her, that's definitely high praise) The only thing she'd improve was exactly what you mentioned: offline recognition! (and also keeping the bird sound recordings to export them later)
- marcdevo 5y ago
  see above thread for info on Merlin with offline sound ID capabilities...
  - gregsadetsky 5y ago
    Yep, that's great news. Thanks!
marcdevo 5y ago
I work at the Cornell Lab, and we also have an app that is more consumer-oriented, and which DOESN'T require uploading the recordings to the cloud -- see Merlin Bird ID app https://merlin.allaboutbirds.org/ https://merlin.allaboutbirds.org/ When the Lab’s researchers conceived of BirdNET, there were no reliable bird sound identification tools. BirdNET was built as a rapid prototype, engaging computer science students to build an app for that users to test the machine learning algorithms. BirdNET proved to be a research breakthrough and by 2020 was performing with far better accuracy than five other apps tested. That success opened the way to apply computer vision to sound identification in the Lab’s outreach and education app, Merlin. Merlin offers OFFLINE functionality, and multiple ways to help identify birds, including through a user describing the bird, taking a photo of the bird, and now recording a bird song or call. Merlin Bird ID is integrated with the Lab’s systems and resources, including updated taxonomy, bird information from eBird and Birds of the World, rich media from the Macaulay Library, life list building tools integrated with eBird, and more.
- justinlkarr 5y ago
  The off-line version of this (Merlin) worked great in the Boundary Waters in June. Detected visually-verified grouse, vireos, woodpeckers, trumpter swan and bald eagles on Nina Moose and Agnes lakes.
- hasmanean 5y ago
  Oh right, turns out I had it downloaded but hadn’t used it (the bird pack was too big to download over data). The benefit of using hardware accelerated ML built into the phone is that it’s much more lower power. It’s designed for continuous use cases (“Hey Alexa” or Hey Siri). So you don’t have to turn the recording on and off and miss the bird call. I don’t know if continuous monitoring can be used by third party apps. But having it on all the time, with geolocation would be amazing. You could set alerts etc. If you could use multiple phones to locate the bird in 3space that would be neat. Then you could tell people where to point their cameras. Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird. This is the future.
  - sdenton4 5y ago
    /Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird./ This is actually the 'real' research motivation behind the bird classification work: Slap a microphone to the side of a tree, pick it up in a month, and get some accurate picture of what species have been in the area. Birds are relatively easy to observe, thanks to their vocalizations, which makes them an indicator species. We have a good idea what many species eat, so they end up telling you quite a lot about the surrounding ecosystem. However, it turns out that the 'soundscape problem' where the microphone is just attached to a tree is a bit more difficult than identifying foreground birds only, using a device that can be pointed in the relevant direction by the user. We've been encouraging further work on the soundscape problem by hosting the BirdCLEF and Kaggle competitions, and have been seeing steady progress. Improvements in the 'hard' soundscape problem have been driving improvements in the 'consumer' identification algorithms. https://www.kaggle.com/c/birdclef-2021/overview https://www.kaggle.com/c/birdclef-2021/overview [source: I've been working with the BirdNet folks on and off for the last few years, and co-host the Kaggle competitions.]
dheera 5y ago
Oooh you can maybe geolocate users who don't give you location permissions by listening to the birds around you.

gregsadetsky 5y ago

I asked a veterinarian friend who specializes in birds (and is a long time bird watcher), and she knew the app and said that it works pretty well (coming from her, that's definitely high praise) The only thing she'd improve was exactly what you mentioned: offline recognition! (and also keeping the bird sound recordings to export them later)

marcdevo 5y ago
see above thread for info on Merlin with offline sound ID capabilities...
- gregsadetsky 5y ago
  Yep, that's great news. Thanks!

marcdevo 5y ago

I work at the Cornell Lab, and we also have an app that is more consumer-oriented, and which DOESN'T require uploading the recordings to the cloud -- see Merlin Bird ID app https://merlin.allaboutbirds.org/ https://merlin.allaboutbirds.org/ When the Lab’s researchers conceived of BirdNET, there were no reliable bird sound identification tools. BirdNET was built as a rapid prototype, engaging computer science students to build an app for that users to test the machine learning algorithms. BirdNET proved to be a research breakthrough and by 2020 was performing with far better accuracy than five other apps tested. That success opened the way to apply computer vision to sound identification in the Lab’s outreach and education app, Merlin. Merlin offers OFFLINE functionality, and multiple ways to help identify birds, including through a user describing the bird, taking a photo of the bird, and now recording a bird song or call. Merlin Bird ID is integrated with the Lab’s systems and resources, including updated taxonomy, bird information from eBird and Birds of the World, rich media from the Macaulay Library, life list building tools integrated with eBird, and more.

justinlkarr 5y ago
The off-line version of this (Merlin) worked great in the Boundary Waters in June. Detected visually-verified grouse, vireos, woodpeckers, trumpter swan and bald eagles on Nina Moose and Agnes lakes.
hasmanean 5y ago
Oh right, turns out I had it downloaded but hadn’t used it (the bird pack was too big to download over data). The benefit of using hardware accelerated ML built into the phone is that it’s much more lower power. It’s designed for continuous use cases (“Hey Alexa” or Hey Siri). So you don’t have to turn the recording on and off and miss the bird call. I don’t know if continuous monitoring can be used by third party apps. But having it on all the time, with geolocation would be amazing. You could set alerts etc. If you could use multiple phones to locate the bird in 3space that would be neat. Then you could tell people where to point their cameras. Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird. This is the future.
- sdenton4 5y ago
  /Maybe a standalone IoT monitoring device could be placed in forests to count each and every bird./ This is actually the 'real' research motivation behind the bird classification work: Slap a microphone to the side of a tree, pick it up in a month, and get some accurate picture of what species have been in the area. Birds are relatively easy to observe, thanks to their vocalizations, which makes them an indicator species. We have a good idea what many species eat, so they end up telling you quite a lot about the surrounding ecosystem. However, it turns out that the 'soundscape problem' where the microphone is just attached to a tree is a bit more difficult than identifying foreground birds only, using a device that can be pointed in the relevant direction by the user. We've been encouraging further work on the soundscape problem by hosting the BirdCLEF and Kaggle competitions, and have been seeing steady progress. Improvements in the 'hard' soundscape problem have been driving improvements in the 'consumer' identification algorithms. https://www.kaggle.com/c/birdclef-2021/overview https://www.kaggle.com/c/birdclef-2021/overview [source: I've been working with the BirdNet folks on and off for the last few years, and co-host the Kaggle competitions.]

dheera 5y ago

Oooh you can maybe geolocate users who don't give you location permissions by listening to the birds around you.