3 ms·

The underlying classifier, BirdNET, is a traditional neural network and not an LLM: https://doi.org/10.1016/j.ecoinf.2021.101236 https://doi.org/10.1016/j.ecoi

The underlying classifier, BirdNET, is a traditional neural network and not an LLM: https://doi.org/10.1016/j.ecoinf.2021.101236 https://doi.org/10.1016/j.ecoinf.2021.101236

tuvix 19d ago
Are there any LLMs being widely used for audio classification? I know VLMs are being used a lot in image stuff. It always seems kind of silly to me to throw everything at an LLM. I know they’re huge and can automatically handle a huge number of tasks but something in me finds it wasteful when we could be creating easily trainable, cheap to run bespoke models for a lot of stuff
- benob 19d ago
  https://github.com/earthspecies/NatureLM-audio https://github.com/earthspecies/NatureLM-audio
  - bix6 19d ago
    More info please. Can I add this into birdnet go for more species detection?
- lambda 19d ago
  There are LLMs that support audio input, similar to those with vision support. From my testing of open weights LLMs with audio support, they basically are only trained to recognize audio as an alternative to text input, they treat audio as basically equivalent to a transcript, and can't recognize or distinguish things like music, accents, background sounds, etc. So they're only really good for transcribing or summarizing or using audio input in place of text input for prompts, but not anything that requires distinguishing any information about the audio that would not be present in a transcript. It can be tempting to try to use an LLM for a variety of tasks; kind of the whole thing about an LLM is that you don't have to do a separate complex training run for every task, but can just provide instructions in natural language. But it only works as far as what the training data covers, if the training basically always treated audio and a text transcript as equivalent, the model has nothing causing it to learn other relevant features of the audio. If there's enough bird call identification in the training data of an LLM, it might be able to do that, but I think multimodal training data tends to be much more limited than the text training corpus
  - sbrother 19d ago
    Have you had any luck fine tuning one with musical data for classification or music-aware QA? I've been hacking on https://trebel.la/ https://trebel.la/ which I would like to be a music practice companion, and the biggest missing feature is actually useful audio-based feedback pipeline. My current approach, not yet validated, is trying to generate training data from masterclass recordings on Youtube, and then fine tuning MOSS-audio on a bunch of those. But I'm interested if there are better models, or large training sets I don't know about.
    - MisterMunchkin 18d ago
      You're selling a music feedback app and it doesn't do the music feedback part yet?
      - sbrother 18d ago
        I'm not selling anything; I'm developing it live and there is a landing page, but there's no payment hooked up since like you said, it's not functional yet.
        elictronic 17d ago
        It's the first thing you see when opening the page.
    - lambda 17d ago
      I have not. It's something that I've considered, but never actually have done it.
- radarsat1 19d ago
  wav2vec or similar approaches are used a lot these days, which is basically BERT with audio inputs. Whether that counts as an LLM or not, I am not sure. It's a transformer architecture in any case. People will often reach for "easily trainable, cheap" solutions when they can; the reason people reach for Transformers and LLMs is because when you throw more data at them, they get better.
- ninjalanternshk 18d ago
  The harness that connects to a chatbot, API or voice interaction is the place to route requests to different systems. If you remember the early days of ChatGPT it explicitly said it was routing image generation to Dall-E after embellishing your request itself first. Determining which tool to use should be a lightweight operation but I’m not expert enough to understand exactly how much lighter than a full LLM call just to recognize it needs a different tool or model.
  - tuvix 18d ago
    For sure, I just know it’s tempting given the power of large transformers to throw things at an existing model. For instance, OCR is something that can be done locally with no access to a GPU but people (including me) still often use cloud hosted multi-modal large language models for it.
- antonvs 18d ago
  There’s still plenty of commercial work on specialized models, for the simple reason that they’re typically much cheaper to train and run. Many systems now use LLMs in conjunction with specialized models.
ada1981 19d ago
Curious if there is a similar project for dog barks.
- lacunary 19d ago
  I wanted to say that seems like a stretch but then I often find myself visualizing my best guess at the appearance, including species, of an unknown dog. It seems like size of dog and pitch of bark are negatively correlated
  - nozzlegear 17d ago
    I think it's because bigger dogs have larger throats and a bigger "voice box," so to speak. They have more of the vocal folds than a small dog, plus they have a bigger chest and larger lungs to force more air through them. That's my bro-science understanding of it, anyway.
- pgreenwood 18d ago
  Yes, here's the classifier: def get_species(): return 'Canis familiaris'
rexxars 19d ago
Note that while the underlying birdnet-go project started as BirdNET only, it can now use Google Perch v2, BattyBirdNET (for bats!) and other models in the future. It's a really cool project!
cosmojg 18d ago
Why would anyone assume this uses an LLM? It classifies bird sounds, not human language. I don't mean this as an attack, I'm genuinely curious! This seemed obvious to me, and I want to know what line of thinking might lead one to believe that an LLM is the better (or more likely) tool for this job over a purpose-built classifier.
- WolfeReader 18d ago
  Most "Show HN" posts these days are vibe-coded garbage. It is a pleasant surprise to see one that isn't!
3abiton 16d ago
Few years ago, there was an android app (open source from fdroid) that basically provided the model and ability to listen and give you informafion about the birds. It was very fun during hikes with friends.

The underlying classifier, BirdNET, is a traditional neural network and not an LLM: https://doi.org/10.1016/j.ecoinf.2021.101236 https://doi.org/10.1016/j.ecoinf.2021.101236

tuvix 19d ago

Are there any LLMs being widely used for audio classification? I know VLMs are being used a lot in image stuff. It always seems kind of silly to me to throw everything at an LLM. I know they’re huge and can automatically handle a huge number of tasks but something in me finds it wasteful when we could be creating easily trainable, cheap to run bespoke models for a lot of stuff

benob 19d ago
https://github.com/earthspecies/NatureLM-audio https://github.com/earthspecies/NatureLM-audio
- bix6 19d ago
  More info please. Can I add this into birdnet go for more species detection?
lambda 19d ago
There are LLMs that support audio input, similar to those with vision support. From my testing of open weights LLMs with audio support, they basically are only trained to recognize audio as an alternative to text input, they treat audio as basically equivalent to a transcript, and can't recognize or distinguish things like music, accents, background sounds, etc. So they're only really good for transcribing or summarizing or using audio input in place of text input for prompts, but not anything that requires distinguishing any information about the audio that would not be present in a transcript. It can be tempting to try to use an LLM for a variety of tasks; kind of the whole thing about an LLM is that you don't have to do a separate complex training run for every task, but can just provide instructions in natural language. But it only works as far as what the training data covers, if the training basically always treated audio and a text transcript as equivalent, the model has nothing causing it to learn other relevant features of the audio. If there's enough bird call identification in the training data of an LLM, it might be able to do that, but I think multimodal training data tends to be much more limited than the text training corpus
- sbrother 19d ago
  Have you had any luck fine tuning one with musical data for classification or music-aware QA? I've been hacking on https://trebel.la/ https://trebel.la/ which I would like to be a music practice companion, and the biggest missing feature is actually useful audio-based feedback pipeline. My current approach, not yet validated, is trying to generate training data from masterclass recordings on Youtube, and then fine tuning MOSS-audio on a bunch of those. But I'm interested if there are better models, or large training sets I don't know about.
  - MisterMunchkin 18d ago
    You're selling a music feedback app and it doesn't do the music feedback part yet?
    - sbrother 18d ago
      I'm not selling anything; I'm developing it live and there is a landing page, but there's no payment hooked up since like you said, it's not functional yet.
      - elictronic 17d ago
        It's the first thing you see when opening the page.
  - lambda 17d ago
    I have not. It's something that I've considered, but never actually have done it.
radarsat1 19d ago
wav2vec or similar approaches are used a lot these days, which is basically BERT with audio inputs. Whether that counts as an LLM or not, I am not sure. It's a transformer architecture in any case. People will often reach for "easily trainable, cheap" solutions when they can; the reason people reach for Transformers and LLMs is because when you throw more data at them, they get better.
ninjalanternshk 18d ago
The harness that connects to a chatbot, API or voice interaction is the place to route requests to different systems. If you remember the early days of ChatGPT it explicitly said it was routing image generation to Dall-E after embellishing your request itself first. Determining which tool to use should be a lightweight operation but I’m not expert enough to understand exactly how much lighter than a full LLM call just to recognize it needs a different tool or model.
- tuvix 18d ago
  For sure, I just know it’s tempting given the power of large transformers to throw things at an existing model. For instance, OCR is something that can be done locally with no access to a GPU but people (including me) still often use cloud hosted multi-modal large language models for it.
antonvs 18d ago
There’s still plenty of commercial work on specialized models, for the simple reason that they’re typically much cheaper to train and run. Many systems now use LLMs in conjunction with specialized models.

ada1981 19d ago

Curious if there is a similar project for dog barks.

lacunary 19d ago
I wanted to say that seems like a stretch but then I often find myself visualizing my best guess at the appearance, including species, of an unknown dog. It seems like size of dog and pitch of bark are negatively correlated
- nozzlegear 17d ago
  I think it's because bigger dogs have larger throats and a bigger "voice box," so to speak. They have more of the vocal folds than a small dog, plus they have a bigger chest and larger lungs to force more air through them. That's my bro-science understanding of it, anyway.
pgreenwood 18d ago
Yes, here's the classifier: def get_species(): return 'Canis familiaris'

rexxars 19d ago

Note that while the underlying birdnet-go project started as BirdNET only, it can now use Google Perch v2, BattyBirdNET (for bats!) and other models in the future. It's a really cool project!

cosmojg 18d ago

Why would anyone assume this uses an LLM? It classifies bird sounds, not human language. I don't mean this as an attack, I'm genuinely curious! This seemed obvious to me, and I want to know what line of thinking might lead one to believe that an LLM is the better (or more likely) tool for this job over a purpose-built classifier.

WolfeReader 18d ago
Most "Show HN" posts these days are vibe-coded garbage. It is a pleasant surprise to see one that isn't!

3abiton 16d ago

Few years ago, there was an android app (open source from fdroid) that basically provided the model and ability to listen and give you informafion about the birds. It was very fun during hikes with friends.