7 ms·

Does anyone use llama as their primary model for any usecase? Maybe it's my fault for not spending much time with it, but I still couldn't find the applications

Does anyone use llama as their primary model for any usecase? Maybe it's my fault for not spending much time with it, but I still couldn't find the applications for which llama has an advantage over the competition.

philipkglass 1y ago
I recently needed to classify thousands of documents according to some custom criteria. I wanted to use LLM classification from these thousands of documents to train a faster, smaller BERT (well, ModernBERT) classifier to use across millions of documents. For my task, Llama 3.3 was still the best local model I could run. I tried newer ones (Phi4, Gemma3, Mistral Small) but they produced much worse results. Some larger local models are probably better if you have the hardware for them, but I only have a single 4090 GPU and 128 GB of system RAM.
- galeos 1y ago
  How did you find ModernBERT performance Vs prior BERT models?
  - philipkglass 1y ago
    I didn't try original BERT at all because I didn't get good results from any LLMs on small document excerpts, so I assumed that a substantial context was necessary for good results. Traditional BERT only accepts up to 512 tokens, while ModernBERT goes up to 8192. I ended up using a 2048 token limit.
    - stavros 1y ago
      Would you happen to know of any resources for how to distill a ModernBERT model out of a larger one? I'm interested in doing exactly what you did, but I don't know how to start.
      - philipkglass 1y ago
        I was trying to identify "evergreen" and "time-sensitive" kinds of writing -- basically, I wanted to figure out if web pages captured in 2016 would still have content that's interesting to read today or if the passage of time would have rendered them irrelevant. Here's the training code that I used to fine-tune ModernBERT from the ~5000 pages I had labeled with Llama 3.3. It should be a good starting point if you have your own fine-tuning task like this. If you can get away with a smaller context than I used here, it will be much faster and the batches can be larger (requires experimentation). https://pastebin.com/Saq1EyAB https://pastebin.com/Saq1EyAB
        stavros 1y ago
        Thank you!
marcodena 1y ago
Here https://research.atspotify.com/2024/12/contextualized-recommendations-through-personalized-narratives-using-llms/ https://research.atspotify.com/2024/12/contextualized-recomm... ;)
mvieira38 1y ago
It's pretty popular in the local LLM space
- littlestymaar 1y ago
  It used to, but Llama 4 is useless for local LLM for most people.
- nowittyusername 1y ago
  Is it? I am pretty active in those spaces and most folks it seems are using any number of the Chinese models like qwen, qwq, etc...
  - NitpickLawyer 1y ago
    If by those spaces you mean reddit, then yeah I've also noticed this trend. It has became more egregious with the duality of L4 vs Qwen3 reception. L4 was blamed, mocked and everyone was posting shit about it (well, some of it was relevant since the launch was rushed and many providers had bad implementations for ~2-3 days) in stark contrast to qwen3 which also had inferencing problems (related to 3rd party tools using wrong settings) and by all accounts has overinflated their benchmark scores. Don't take the "activity" of those places as gospel, try the models on your own stacks, with your own benchmarks for best results.
    - logicchains 1y ago
      L4 launched without a thinking model, making it an inferior choice for coding, one of the main LLM use cases. Even in benchmarks it wasn't even competitive at coding with 3-month-old Deepseek R1.
      - NitpickLawyer 1y ago
        Agreed, coding is not a strong point of L4. However the "hive mind" in some places thinks L4 is "a failure" and "useless". In reality it is an "ok" model, and most 3rd party benchmarks done after the inference lib updates were in line with whatever meta announced.
aprilthird2021 1y ago
The biggest advantage is that it's free and available on your phone already. But if you have an Android phone, Gemini on that phone is far superior. And if you have Apple, well maybe that's all the people who use it

philipkglass 1y ago

I recently needed to classify thousands of documents according to some custom criteria. I wanted to use LLM classification from these thousands of documents to train a faster, smaller BERT (well, ModernBERT) classifier to use across millions of documents. For my task, Llama 3.3 was still the best local model I could run. I tried newer ones (Phi4, Gemma3, Mistral Small) but they produced much worse results. Some larger local models are probably better if you have the hardware for them, but I only have a single 4090 GPU and 128 GB of system RAM.

galeos 1y ago
How did you find ModernBERT performance Vs prior BERT models?
- philipkglass 1y ago
  I didn't try original BERT at all because I didn't get good results from any LLMs on small document excerpts, so I assumed that a substantial context was necessary for good results. Traditional BERT only accepts up to 512 tokens, while ModernBERT goes up to 8192. I ended up using a 2048 token limit.
  - stavros 1y ago
    Would you happen to know of any resources for how to distill a ModernBERT model out of a larger one? I'm interested in doing exactly what you did, but I don't know how to start.
    - philipkglass 1y ago
      I was trying to identify "evergreen" and "time-sensitive" kinds of writing -- basically, I wanted to figure out if web pages captured in 2016 would still have content that's interesting to read today or if the passage of time would have rendered them irrelevant. Here's the training code that I used to fine-tune ModernBERT from the ~5000 pages I had labeled with Llama 3.3. It should be a good starting point if you have your own fine-tuning task like this. If you can get away with a smaller context than I used here, it will be much faster and the batches can be larger (requires experimentation). https://pastebin.com/Saq1EyAB https://pastebin.com/Saq1EyAB
      - stavros 1y ago
        Thank you!

marcodena 1y ago

Here https://research.atspotify.com/2024/12/contextualized-recommendations-through-personalized-narratives-using-llms/ https://research.atspotify.com/2024/12/contextualized-recomm... ;)

mvieira38 1y ago

It's pretty popular in the local LLM space

littlestymaar 1y ago
It used to, but Llama 4 is useless for local LLM for most people.
nowittyusername 1y ago
Is it? I am pretty active in those spaces and most folks it seems are using any number of the Chinese models like qwen, qwq, etc...
- NitpickLawyer 1y ago
  If by those spaces you mean reddit, then yeah I've also noticed this trend. It has became more egregious with the duality of L4 vs Qwen3 reception. L4 was blamed, mocked and everyone was posting shit about it (well, some of it was relevant since the launch was rushed and many providers had bad implementations for ~2-3 days) in stark contrast to qwen3 which also had inferencing problems (related to 3rd party tools using wrong settings) and by all accounts has overinflated their benchmark scores. Don't take the "activity" of those places as gospel, try the models on your own stacks, with your own benchmarks for best results.
  - logicchains 1y ago
    L4 launched without a thinking model, making it an inferior choice for coding, one of the main LLM use cases. Even in benchmarks it wasn't even competitive at coding with 3-month-old Deepseek R1.
    - NitpickLawyer 1y ago
      Agreed, coding is not a strong point of L4. However the "hive mind" in some places thinks L4 is "a failure" and "useless". In reality it is an "ok" model, and most 3rd party benchmarks done after the inference lib updates were in line with whatever meta announced.

aprilthird2021 1y ago

The biggest advantage is that it's free and available on your phone already. But if you have an Android phone, Gemini on that phone is far superior. And if you have Apple, well maybe that's all the people who use it