7 ms·

> The ability to “talk to an expert” about any topic I’m curious about and ask very specific questions has been invaluable to me. It is dangerous to assume tha

> The ability to “talk to an expert” about any topic I’m curious about and ask very specific questions has been invaluable to me. It is dangerous to assume that LLMs are experts on any topic. With or without quotes. You are getting a super fast journalist intern with a huge memory but inability to reason critically, lacking understanding about anything and huge unreliability when it comes to answering questions (you can get completely different answers to the same question depending on how you answer it and sometimes even the same question can get you different answers). LLMs are very useful and are a true game changer. But calling that expertise is a disservice to the true experts.

pera 2y ago
It's scary to think that we are moving into this direction: I can see how in the next few years politicians and judges will use LLMs as neutral experts. And all in the hand of a few big tech corporations...
- SamPatt 2y ago
  They aren't just in the hands of big corporations though. The open source, local LLM community is absolutely buzzing right now. Yes, the big companies are making the models, but enough of them are open weights that they can be fine tuned and run however you like. I think LLMs genuinely do present an opportunity to be neutral experts, or at the least neutral third parties. If they're run in completely transparent ways, they may be preferable to humans in some circumstances.
  - svieira 2y ago
    The whole problem is that they are not neutral. They token-complete based on the corpus that was fed into them and the dimensions that were extracted out of those corpuses and the curve-fitting done to those dimensions. Being "completely transparent" means exposing _all_ of that, but that's too large for anyone to reasonably understand without becoming an expert in that particular model. And then we're right back to "trusting expert human beings" again.
    - SamPatt 2y ago
      Nothing is truly neutral. Humans all have a different corpus too. We roughly know what data has gone in, and what the RL process looks like, and how the models handle a given ethical situation. With good prompting, the SOTA models already act in ways I think most reasonable people would agree with, and that's without trying to build this specifically for that use case.
  - dns_snek 2y ago
    > Yes, the big companies are making the models, but enough of them are open weights that they can be fine tuned and run however you like. And how long is that going to last? This is a well known playbook at this point, we'd be better off if we didn't fall for it yet again - it's comical at this point. Sooner or later they'll lock the ecosystem down, take all the free stuff away and demand to extract the market value out of the work they used to "graciously" provide for free to build an audience and market share.
    - SamPatt 2y ago
      How will they do this? You can't take the free stuff away. It's on my hard drive. They can stop releasing them, but local models aren't going anywhere.
      - johnisgood 2y ago
        What are the hardware and software requirements for a self-hosted LLM that is akin to Claude?
        skirmish 2y ago
        Llama v3.3 70B after quantization runs reasonably well on a 24GB GPU (7900XTX or 4090) and 64GB of regular RAM. Software: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp .
      - dns_snek 2y ago
        They can't take the current open models away, but those will eventually (and I imagine, rather quickly) become obsolete for many areas of knowledge work that require relatively up to date information.
- pino82 2y ago
  The world was such a boring and dark place before everybody was constantly swiping on his smartphone in any situation, and before everysaid said basically got piped through a bigtech data center, where their algorithms control its way. Now we finally have a tool where all of you can prove every day how strong/smart/funny/foo you are (not actually). How was life even possible without? So, don't be so pessimistic. ;)
- ben_w 2y ago
  > I can see how in the next few years politicians and judges will use LLMs as neutral experts. While also noting that "neutral" is not well-defined, I agree. They will be used as if they were.
  - AgentOrange1234 2y ago
    Will they though? We humans are very good at rejecting any information that doesn’t confirm our priors or support our political goals. Like, if ChatGPT says (say) vaccines are good/bad, I expect the other side will simply attack and reject it as misinformation, conspiracy, and similar.
    - ben_w 2y ago
      From what I can see, LLMs default to being sychophants; acting as if a sychophant was neutral is entirely compatible with the cognitive bias you describe.
anshumankmr 2y ago
They have tried to address it with the help of o1 or o3 model at least to help it understand and reason better than before, but one of the quotes my manager says with regards to these is to trust it but verify it also.
- Biologist123 2y ago
  “Believe in God, but tie up your camels”.
Biologist123 2y ago
But hasn’t it become quite easy to deal with this issue simply by asking for the sources of the information and then validating? I quite like using the consensus app and then asking for specific academic paper references which I can then quickly check. However this has taught me also that academic claims must also be validated…
- netdevphoenix 2y ago
  If you need to validate the sources, you might as well go to the sources directly and bypass the LLM. The whole point of LLMs is not needing to go to the sources. The LLM consumes them for you. If you need to read and understand the sources yourself well enough to tell if the LLM is lying, the LLM is a wasteful middleman. It's like buying supermarket food and also buying the same food from the farmers themselves.
bilsbie 2y ago
IMO it’s dangerous to call experts experts as well. Possibly more dangerous.
- Angostura 2y ago
  No. Expertise isn’t a synonym for ‘infallible’ it denotes someone whose lived experience, learned knowledge and skill means that you should listen to their opinion in their area of expertise - and defer to it, unless you have direct and evidence-based reasons for thinking they are wrong.
  - bilsbie 2y ago
    By that definition an expert would be <more> trustworthy. (Usually they want you to look at credentials instead.) However that still ignores human nature to use that trust for personal gain. Nothing about expertise makes someone a saint.
gizmo 2y ago
People who are experts (PhD and 20 years of experience) often have very dumb opinions in their field of expertise. Experts make amateur mistakes too. Look at the books written by expert economists, expert psychologists, expert historians, expert philosophers, expert software engineers. Most books are not worth the paper they're written on, despite the authors being experts with decades of experience in their respective fields. I think you overestimate the ability of a typical 'expert'. You can earn a PhD without the ability to reason critically. You can testify as an expert in a courtroom without understanding conditional probability. Lawyers and accountants in real life also totally contradict themselves when they get asked the same question twice but phrased slightly differently.
- parineum 2y ago
  > People who are experts (PhD and 20 years of experience) often have very dumb opinions in their field of expertise. And the training data contains all those dumb opinions.
  - ulbu 2y ago
    and rehashes it unthinkingly, without an idea of what it means to consider and disagree with it.
- indeed30 2y ago
  I don't disagree with what you say, but one difference is that we generally hold these people accountable and often shift liability to them when they are wrong (though not always, admittedly), which is not something I have ever seen done with any AI system.
  - MacsHeadroom 2y ago
    This sounds like an argument in favor of AI personhood, not an argument against AI experts.
- FrustratedMonky 2y ago
  Right, but, then what? If you throw away all of the books from experts, what do you do, go out in your backyard and start running experiments to re-create all of science? Or start googling? What, some random person on the internet is going to be a better 'expert' than someone that wrote a book? Books might not be great, but they are at least some minimum bar to reach. You had to do some study and analysis. Seems like any critic of books, if you scratch the surface is just the whole anti-science/anti-education tropes again and again. What is the option? Don't like peer review science, fine, it has flaws, propose an option.
kevinsync 2y ago
I actually find LLMs lacking true expertise to be a feature, not a bug. Most of the time I'm starting from a place of no knowledge on a topic that's novel to me, I ask some questions, it replies with summaries, keywords, names of things, basic concepts. I enter with the assumption that it's really no different than googling phrases and sifting through results (except I don't know what phrases I'm supposed to be googling in the first place), so the summaries help a lot. I then ask a lot of questions and ask for examples and explanations, some of which of course turn out to be wrong, but the more I push back, re-generate, re-question, etc (while using traditional search engines in another tab), the better responses I can get it to provide. Come to think of it, it's really no different than walking into Home Depot and asking "the old guys" working in the aisles about stuff -- you can access some fantastic knowledge if you know the names of all the tools and techniques, and if not, can show them a picture or describe what you're trying to do and they'll at least point you in a starting direction with regards to names of tools needed, techniques to use, etc. Just like I don't expect Home Depot hourly worker Grandpa Bob to be the end-all-be-all expert (for free, as well!), neither do I expect ChatGPT to be an all-knowing-all-encompassing oracle of knowledge. It'll probably get you 95% of the way there though!
- fennecfoxy 2y ago
  TBF they're only truly useful when hooked up to RAG imo. I'm honestly surprised that we haven't yet built a digital seal of authenticity for truth that can be used by AI agents + RAG to conceivably give the most accurate answer possible. Scientists should be writing papers sealed digitally once they're peer reviewed and considered "truth", same thing with journalist/news articles - sealed once confirmed true or backed up by a solid source in the same way we trust root certificates. But then again, especially when it comes to journalism, cropping photos, chopping quotes, etc all to misrepresent etc. Turns out we're all the bad actors; it's in our DNA. And tbf, many people when presented with hard evidence to the contrary of the opinion that they cling onto like a babe to a breast, just plug their ears and cover their eyes. Okay so maybe there's no point seeking truth/factual correctness, our species doesn't want it 99% of the time, unless it affects them directly (eg people that shoot down public healthcare until they have an expensive illness themselves).
- 2y ago
wing-_-nuts 2y ago
Shrug I treat LLM answers about the same way I treat wikipedia articles. If it's critical I get it right, I go to the wiki sources referenced. Recent models have gotten good at 'showing their sources', which is helpful.
- netdevphoenix 2y ago
  > If it's critical I get it right, I go to the wiki sources referenced the problem with this is that humans will likely use it for low key stuff, see that it works (or that the errors don't affect them too badly) and start using it for more serious stuff. It will all be good until someone uses it in something more serious and some time later it ends badly. Human basic thinking is fairly primitive. If yesterday was sunny, the assumption is that today should too. The more this happens the higher your confidence. The problem is that this confidence emboldens people to gamble on that and when it is not sunny anymore, terrible things happen. A lot of hype driven behaviour is like that. Crypto was like that. The economic crisis of the late 00s was like that. And LLMs look set to be like that too. It is going to take a big event involving big critical damage or a high profile series of deaths via misuse of an LLM to give policymakers and business leaders around the world a reality check and get them looking at LLMs in a more critical way. An AI autumn if you wish. It is going to happen at some point. Maybe not in 2025 or 2026 but it will definitely happen. You may argue that it is the fault of the human using the LLM/crypto/giving out loans but it really doesn't matter when those decisions affect others.
jasondigitized 2y ago
It's dangerous to assume that the person you have access to is an expert either.

pera 2y ago

It's scary to think that we are moving into this direction: I can see how in the next few years politicians and judges will use LLMs as neutral experts. And all in the hand of a few big tech corporations...

SamPatt 2y ago
They aren't just in the hands of big corporations though. The open source, local LLM community is absolutely buzzing right now. Yes, the big companies are making the models, but enough of them are open weights that they can be fine tuned and run however you like. I think LLMs genuinely do present an opportunity to be neutral experts, or at the least neutral third parties. If they're run in completely transparent ways, they may be preferable to humans in some circumstances.
- svieira 2y ago
  The whole problem is that they are not neutral. They token-complete based on the corpus that was fed into them and the dimensions that were extracted out of those corpuses and the curve-fitting done to those dimensions. Being "completely transparent" means exposing _all_ of that, but that's too large for anyone to reasonably understand without becoming an expert in that particular model. And then we're right back to "trusting expert human beings" again.
  - SamPatt 2y ago
    Nothing is truly neutral. Humans all have a different corpus too. We roughly know what data has gone in, and what the RL process looks like, and how the models handle a given ethical situation. With good prompting, the SOTA models already act in ways I think most reasonable people would agree with, and that's without trying to build this specifically for that use case.
- dns_snek 2y ago
  > Yes, the big companies are making the models, but enough of them are open weights that they can be fine tuned and run however you like. And how long is that going to last? This is a well known playbook at this point, we'd be better off if we didn't fall for it yet again - it's comical at this point. Sooner or later they'll lock the ecosystem down, take all the free stuff away and demand to extract the market value out of the work they used to "graciously" provide for free to build an audience and market share.
  - SamPatt 2y ago
    How will they do this? You can't take the free stuff away. It's on my hard drive. They can stop releasing them, but local models aren't going anywhere.
    - johnisgood 2y ago
      What are the hardware and software requirements for a self-hosted LLM that is akin to Claude?
      - skirmish 2y ago
        Llama v3.3 70B after quantization runs reasonably well on a 24GB GPU (7900XTX or 4090) and 64GB of regular RAM. Software: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp .
    - dns_snek 2y ago
      They can't take the current open models away, but those will eventually (and I imagine, rather quickly) become obsolete for many areas of knowledge work that require relatively up to date information.
pino82 2y ago
The world was such a boring and dark place before everybody was constantly swiping on his smartphone in any situation, and before everysaid said basically got piped through a bigtech data center, where their algorithms control its way. Now we finally have a tool where all of you can prove every day how strong/smart/funny/foo you are (not actually). How was life even possible without? So, don't be so pessimistic. ;)
ben_w 2y ago
> I can see how in the next few years politicians and judges will use LLMs as neutral experts. While also noting that "neutral" is not well-defined, I agree. They will be used as if they were.
- AgentOrange1234 2y ago
  Will they though? We humans are very good at rejecting any information that doesn’t confirm our priors or support our political goals. Like, if ChatGPT says (say) vaccines are good/bad, I expect the other side will simply attack and reject it as misinformation, conspiracy, and similar.
  - ben_w 2y ago
    From what I can see, LLMs default to being sychophants; acting as if a sychophant was neutral is entirely compatible with the cognitive bias you describe.

anshumankmr 2y ago

They have tried to address it with the help of o1 or o3 model at least to help it understand and reason better than before, but one of the quotes my manager says with regards to these is to trust it but verify it also.

Biologist123 2y ago
“Believe in God, but tie up your camels”.

Biologist123 2y ago

But hasn’t it become quite easy to deal with this issue simply by asking for the sources of the information and then validating? I quite like using the consensus app and then asking for specific academic paper references which I can then quickly check. However this has taught me also that academic claims must also be validated…

netdevphoenix 2y ago
If you need to validate the sources, you might as well go to the sources directly and bypass the LLM. The whole point of LLMs is not needing to go to the sources. The LLM consumes them for you. If you need to read and understand the sources yourself well enough to tell if the LLM is lying, the LLM is a wasteful middleman. It's like buying supermarket food and also buying the same food from the farmers themselves.

bilsbie 2y ago

IMO it’s dangerous to call experts experts as well. Possibly more dangerous.

Angostura 2y ago
No. Expertise isn’t a synonym for ‘infallible’ it denotes someone whose lived experience, learned knowledge and skill means that you should listen to their opinion in their area of expertise - and defer to it, unless you have direct and evidence-based reasons for thinking they are wrong.
- bilsbie 2y ago
  By that definition an expert would be <more> trustworthy. (Usually they want you to look at credentials instead.) However that still ignores human nature to use that trust for personal gain. Nothing about expertise makes someone a saint.

gizmo 2y ago

People who are experts (PhD and 20 years of experience) often have very dumb opinions in their field of expertise. Experts make amateur mistakes too. Look at the books written by expert economists, expert psychologists, expert historians, expert philosophers, expert software engineers. Most books are not worth the paper they're written on, despite the authors being experts with decades of experience in their respective fields. I think you overestimate the ability of a typical 'expert'. You can earn a PhD without the ability to reason critically. You can testify as an expert in a courtroom without understanding conditional probability. Lawyers and accountants in real life also totally contradict themselves when they get asked the same question twice but phrased slightly differently.

parineum 2y ago
> People who are experts (PhD and 20 years of experience) often have very dumb opinions in their field of expertise. And the training data contains all those dumb opinions.
- ulbu 2y ago
  and rehashes it unthinkingly, without an idea of what it means to consider and disagree with it.
indeed30 2y ago
I don't disagree with what you say, but one difference is that we generally hold these people accountable and often shift liability to them when they are wrong (though not always, admittedly), which is not something I have ever seen done with any AI system.
- MacsHeadroom 2y ago
  This sounds like an argument in favor of AI personhood, not an argument against AI experts.
FrustratedMonky 2y ago
Right, but, then what? If you throw away all of the books from experts, what do you do, go out in your backyard and start running experiments to re-create all of science? Or start googling? What, some random person on the internet is going to be a better 'expert' than someone that wrote a book? Books might not be great, but they are at least some minimum bar to reach. You had to do some study and analysis. Seems like any critic of books, if you scratch the surface is just the whole anti-science/anti-education tropes again and again. What is the option? Don't like peer review science, fine, it has flaws, propose an option.

kevinsync 2y ago

I actually find LLMs lacking true expertise to be a feature, not a bug. Most of the time I'm starting from a place of no knowledge on a topic that's novel to me, I ask some questions, it replies with summaries, keywords, names of things, basic concepts. I enter with the assumption that it's really no different than googling phrases and sifting through results (except I don't know what phrases I'm supposed to be googling in the first place), so the summaries help a lot. I then ask a lot of questions and ask for examples and explanations, some of which of course turn out to be wrong, but the more I push back, re-generate, re-question, etc (while using traditional search engines in another tab), the better responses I can get it to provide. Come to think of it, it's really no different than walking into Home Depot and asking "the old guys" working in the aisles about stuff -- you can access some fantastic knowledge if you know the names of all the tools and techniques, and if not, can show them a picture or describe what you're trying to do and they'll at least point you in a starting direction with regards to names of tools needed, techniques to use, etc. Just like I don't expect Home Depot hourly worker Grandpa Bob to be the end-all-be-all expert (for free, as well!), neither do I expect ChatGPT to be an all-knowing-all-encompassing oracle of knowledge. It'll probably get you 95% of the way there though!

fennecfoxy 2y ago
TBF they're only truly useful when hooked up to RAG imo. I'm honestly surprised that we haven't yet built a digital seal of authenticity for truth that can be used by AI agents + RAG to conceivably give the most accurate answer possible. Scientists should be writing papers sealed digitally once they're peer reviewed and considered "truth", same thing with journalist/news articles - sealed once confirmed true or backed up by a solid source in the same way we trust root certificates. But then again, especially when it comes to journalism, cropping photos, chopping quotes, etc all to misrepresent etc. Turns out we're all the bad actors; it's in our DNA. And tbf, many people when presented with hard evidence to the contrary of the opinion that they cling onto like a babe to a breast, just plug their ears and cover their eyes. Okay so maybe there's no point seeking truth/factual correctness, our species doesn't want it 99% of the time, unless it affects them directly (eg people that shoot down public healthcare until they have an expensive illness themselves).
2y ago

wing-_-nuts 2y ago

Shrug I treat LLM answers about the same way I treat wikipedia articles. If it's critical I get it right, I go to the wiki sources referenced. Recent models have gotten good at 'showing their sources', which is helpful.

netdevphoenix 2y ago
> If it's critical I get it right, I go to the wiki sources referenced the problem with this is that humans will likely use it for low key stuff, see that it works (or that the errors don't affect them too badly) and start using it for more serious stuff. It will all be good until someone uses it in something more serious and some time later it ends badly. Human basic thinking is fairly primitive. If yesterday was sunny, the assumption is that today should too. The more this happens the higher your confidence. The problem is that this confidence emboldens people to gamble on that and when it is not sunny anymore, terrible things happen. A lot of hype driven behaviour is like that. Crypto was like that. The economic crisis of the late 00s was like that. And LLMs look set to be like that too. It is going to take a big event involving big critical damage or a high profile series of deaths via misuse of an LLM to give policymakers and business leaders around the world a reality check and get them looking at LLMs in a more critical way. An AI autumn if you wish. It is going to happen at some point. Maybe not in 2025 or 2026 but it will definitely happen. You may argue that it is the fault of the human using the LLM/crypto/giving out loans but it really doesn't matter when those decisions affect others.

jasondigitized 2y ago

It's dangerous to assume that the person you have access to is an expert either.