4 ms·

I would expect this, but it doesn't seem to be the case. If I ask for search.brave.com to give me a list of gini coefficients for the top ten countries by GDP,

by panny 1y ago

I would expect this, but it doesn't seem to be the case. If I ask for search.brave.com to give me a list of gini coefficients for the top ten countries by GDP, it can't do it. However, if I tell it the data is available on the CIA world factbook, it can then spit that info out promptly. However, if I close the context and ask again, it hasn't learned this information and once again is unable to provide the list. It didn't datamine me. It had no better idea where to find this information the second time I asked. This is the experience others have stated with other AIs as well. It does not seem special to brave.

Etheryte 1y ago
Data mining doesn't mean the model is instantly updated, that would be prohibitively expensive at scale. It's way easier to batch your data together with a bunch of other data and use it later on. That doesn't even mean it will know where to find the information eventually since models are not one to one with their inputs, because again, size and cost.
- panny 1y ago
  >Data mining doesn't mean the model is instantly updated I'm not expecting instant. Even next week it won't be there. It's like how AI never learned to count how many times the letter r appears in strawberry. Like sure, now if you ask brave, it will tell you three, but that is only because that question went viral. It didn't "learn" anything, it was just hard coded for that particular answer. Ask it how many times the letter l appears in smallville and it will get it wrong again.
  - qwertytyyuu 1y ago
    every week is still way to expensive to do at scale, at best they'll update training data with each model iteration.
  - simgt 1y ago
    I didn't think for a second you could be right, so I tried with Claude. L in smallville was correct, then it suggests it'd have gotten l in parallel wrong by answering 3 instead of 2 (buts gets it right in a new chat). Then it suggests it'd get n in millennium wrong by giving the right answer, and gets it wrong in a new chat. https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 Thanks, that was enlightening.
  - t0md4n 1y ago
    It wouldn’t be instant, next week or even next month. Pre-training doesn’t happen that frequently and varies between each model provider. As for the strawberry test, this is a tokenization issue that is fundamental to LLM’s, however, most models can now solve this type of question using thinking/code/tools to count the letters. https://imgur.com/a/NqIJEx6 https://imgur.com/a/NqIJEx6
  - ordersofmag 1y ago
    LLM aren't retrained and released on a weekly time-scale. The data mining may only be reflected in the training of the next generation of the model.
  - Etheryte 1y ago
    Both OpenAI and Claude average roughly one flagship release a year, and these are some of the best funded companies in the space. The bigger your model, the more expensive it is to train, so you want to do it as rarely as reasonably possible. Every other company will either work with smaller models and/or train even more rarely, aside from fine-tunes and customizations they put on top.
- astrange 1y ago
  What is "your data" here? Why would pretraining on "your data" be useful to AI providers? The AI isn't going to learn factual answers to questions by training on the text of questions! How would that work?
  - Etheryte 1y ago
    LLMs aren't only about knowing answers to questions, it's also understanding the question to begin with that's tricky. What questions you ask combined with what followup questions you asked is a very valuable dataset, because if there are common threads across your users, you can train your LLM to understand the first ambiguous question better. This applies to pretty much every other interaction you have with LLMs too, the quicker you can deduce what the majority of the users actually want, the less tokens you need to generate to make them happy, the wider you can make your profit margins.
add-sub-mul-div 1y ago
Brave isn't data mining you for your benefit, they're doing it for their benefit.
- panny 1y ago
  Likewise, I'm not teaching their AI where to find GINI coefficients for their benefit, but for mine. I'd like for their AI to learn something, if only to make my experience better. But there's no learning happening.
deleted 1y ago
[deleted]
hluska 1y ago
You’re expecting models to constantly retrain themselves based on riddles. That’s not very reasonable nor is it even economically feasible right now. At massive scale, I question whether it’s even technically feasible.

by panny 1y ago

Etheryte 1y ago

Data mining doesn't mean the model is instantly updated, that would be prohibitively expensive at scale. It's way easier to batch your data together with a bunch of other data and use it later on. That doesn't even mean it will know where to find the information eventually since models are not one to one with their inputs, because again, size and cost.

panny 1y ago
>Data mining doesn't mean the model is instantly updated I'm not expecting instant. Even next week it won't be there. It's like how AI never learned to count how many times the letter r appears in strawberry. Like sure, now if you ask brave, it will tell you three, but that is only because that question went viral. It didn't "learn" anything, it was just hard coded for that particular answer. Ask it how many times the letter l appears in smallville and it will get it wrong again.
- qwertytyyuu 1y ago
  every week is still way to expensive to do at scale, at best they'll update training data with each model iteration.
- simgt 1y ago
  I didn't think for a second you could be right, so I tried with Claude. L in smallville was correct, then it suggests it'd have gotten l in parallel wrong by answering 3 instead of 2 (buts gets it right in a new chat). Then it suggests it'd get n in millennium wrong by giving the right answer, and gets it wrong in a new chat. https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 https://claude.ai/share/93b46c3b-23a7-40ad-8a2b-ec2ed6c34a19 Thanks, that was enlightening.
- t0md4n 1y ago
  It wouldn’t be instant, next week or even next month. Pre-training doesn’t happen that frequently and varies between each model provider. As for the strawberry test, this is a tokenization issue that is fundamental to LLM’s, however, most models can now solve this type of question using thinking/code/tools to count the letters. https://imgur.com/a/NqIJEx6 https://imgur.com/a/NqIJEx6
- ordersofmag 1y ago
  LLM aren't retrained and released on a weekly time-scale. The data mining may only be reflected in the training of the next generation of the model.
- Etheryte 1y ago
  Both OpenAI and Claude average roughly one flagship release a year, and these are some of the best funded companies in the space. The bigger your model, the more expensive it is to train, so you want to do it as rarely as reasonably possible. Every other company will either work with smaller models and/or train even more rarely, aside from fine-tunes and customizations they put on top.
astrange 1y ago
What is "your data" here? Why would pretraining on "your data" be useful to AI providers? The AI isn't going to learn factual answers to questions by training on the text of questions! How would that work?
- Etheryte 1y ago
  LLMs aren't only about knowing answers to questions, it's also understanding the question to begin with that's tricky. What questions you ask combined with what followup questions you asked is a very valuable dataset, because if there are common threads across your users, you can train your LLM to understand the first ambiguous question better. This applies to pretty much every other interaction you have with LLMs too, the quicker you can deduce what the majority of the users actually want, the less tokens you need to generate to make them happy, the wider you can make your profit margins.

add-sub-mul-div 1y ago

Brave isn't data mining you for your benefit, they're doing it for their benefit.

panny 1y ago
Likewise, I'm not teaching their AI where to find GINI coefficients for their benefit, but for mine. I'd like for their AI to learn something, if only to make my experience better. But there's no learning happening.

deleted 1y ago

[deleted]

hluska 1y ago

You’re expecting models to constantly retrain themselves based on riddles. That’s not very reasonable nor is it even economically feasible right now. At massive scale, I question whether it’s even technically feasible.