4 ms·

I get the sentiment, but I actually think some skepticism in the system is healthy. Billions are flowing towards LLMS, and Sam Altman will overpromise AGI is j

I get the sentiment, but I actually think some skepticism in the system is healthy. Billions are flowing towards LLMS, and Sam Altman will overpromise AGI is just around the corner and the days of jobs are gone to fill his coffers to anyone that will listen. Additionally if we begin to use these things in real production environments where mistakes matters, knowing the exact limitations is key. None of this takes away from the fact that these are exciting times.

antirez 2y ago
Yes there is another part of the community that overhypes everything. But I can expect that from a CEO of an AI company (especially if he is Altman), but from researches? Also the fact that LLMs may reach superhuman expertise in certain fields in a short timeframe (a few years), since reinforcement learning is starting to be applied to LLMs may no longer be a totally crazy position. If it is possible to extend considerably the same approach seen in R1-Zero there could be low hanging fruits around the corner.
comeonbro 2y ago
This article is about things which aren't limitations anymore! You are applauding it as pushback for pushback's sake, but it's an article about limitations in biplane construction, published after we'd already landed on the moon.
- suddenlybananas 2y ago
  Is there any evidence that these fundamental issues with compositionality have been resolved or are you just asserting it? Has the paper been replicated with a CoT model and had a positive result?
  - dr_dshiv 2y ago
    Well, yes — because modern models can solve all the examples in the article. The theory of compositionality is still an issue, but the evidence for it recedes. I think most of the issue comes from the challenge of informational coherence. Once incoherence enters the context, the intelligence drops massively. You can have a lot of context and LLMs can maintain coherence— but not if the context itself is incoherent. And, informationally, it is just a matter of time before a little incoherence gets into a thread. This is why agents have so much potential—being able to separate out separate threads of thought in different context windows reduces the likelihood of incoherence emerging (vs one long thread). Actually, maybe “cybernetic ecologies” are closer to what I mean than “agents.” See Anthropic’s “Building Effective Agents.” https://www.anthropic.com/research/building-effective-agents https://www.anthropic.com/research/building-effective-agents
    - anon84873628 2y ago
      >I think most of the issue comes from the challenge of informational coherence. Once incoherence enters the context, the intelligence drops massively. You can have a lot of context and LLMs can maintain coherence— but not if the context itself is incoherent. As a non-expert, part of my definition of intelligence is that the system can detect incoherence, a.k.a reject bullshit. LLMs today can't do that and will happily emit bullshit in response. Maybe the "gates" in the "workflows" discussed in the Anthropic article are a practical solution to that. But that still just seems like inserting human intelligence into the system for a specific engineering domain; not a general solution.
dr_dshiv 2y ago
I can’t communicate enough how the skepticism (“this is just hype” or “LLMs are stochastic parrots”) is the vastly dominant thought paradigm in European academic circles. So instead of everyone having some enthusiasm and some skepticism, you get a bifurcation where whole classes of people act as the skeptics and others as the enthusiasts. I view the strong skeptics as more “in the wrong” because they often don’t use LLMs much. If you are an actual enthusiastic user, you simply can’t get good performance without a very strong dose of skepticism towards everything LLMs output.
- _t9ow 2y ago
  > I can’t communicate enough how the skepticism (“this is just hype” or “LLMs are stochastic parrots”) is the vastly dominant thought paradigm in European academic circles. I'm very curious. If you don't mind taking the time to elaborate, will you give a few examples of such skepticism/naysaying? Thank you.
  - tessellated 2y ago
    Some "academia" types I meet seem to be struck in this "skeptic vs enthusiast" discussion. How do we harness LLMs’ potential while rigorously mitigating harm?
- fragmede 2y ago
  I don't think everyone shares those doubts. The first time you catch an LLM in a lie is sobering, but there are lots of areas, and thus lots of users, for whom it doesn't hallucinate for, because they're asking softball questions and it doesn't end up hallucinating, or hallucinations just really aren't aren't that big a deal. (eg an LLM horoscope generator or using it write sci fi.) so while we're on HN going back and forth about how outright lies by the system indight the whole thing for everybody, we should be careful to note that it's not for everybody, or rather, it's a known limitation so don't trust it to cite real cases for you as a lawyer, but using it to help you figure out what mens rea means in a practical sense by asking it questions about the concept, totally. Honestly, hallucinations happen so rarely for me because of the kinds of things I ask it, that it doesn't happen enough for me to not believe it's answers in low-stakes situations, or situations on the level of horoscope generation, and I'm sure I'm not alone in treating ChatGPT that way, despite evidence to the contrary.

antirez 2y ago

Yes there is another part of the community that overhypes everything. But I can expect that from a CEO of an AI company (especially if he is Altman), but from researches? Also the fact that LLMs may reach superhuman expertise in certain fields in a short timeframe (a few years), since reinforcement learning is starting to be applied to LLMs may no longer be a totally crazy position. If it is possible to extend considerably the same approach seen in R1-Zero there could be low hanging fruits around the corner.

comeonbro 2y ago

This article is about things which aren't limitations anymore! You are applauding it as pushback for pushback's sake, but it's an article about limitations in biplane construction, published after we'd already landed on the moon.

suddenlybananas 2y ago
Is there any evidence that these fundamental issues with compositionality have been resolved or are you just asserting it? Has the paper been replicated with a CoT model and had a positive result?
- dr_dshiv 2y ago
  Well, yes — because modern models can solve all the examples in the article. The theory of compositionality is still an issue, but the evidence for it recedes. I think most of the issue comes from the challenge of informational coherence. Once incoherence enters the context, the intelligence drops massively. You can have a lot of context and LLMs can maintain coherence— but not if the context itself is incoherent. And, informationally, it is just a matter of time before a little incoherence gets into a thread. This is why agents have so much potential—being able to separate out separate threads of thought in different context windows reduces the likelihood of incoherence emerging (vs one long thread). Actually, maybe “cybernetic ecologies” are closer to what I mean than “agents.” See Anthropic’s “Building Effective Agents.” https://www.anthropic.com/research/building-effective-agents https://www.anthropic.com/research/building-effective-agents
  - anon84873628 2y ago
    >I think most of the issue comes from the challenge of informational coherence. Once incoherence enters the context, the intelligence drops massively. You can have a lot of context and LLMs can maintain coherence— but not if the context itself is incoherent. As a non-expert, part of my definition of intelligence is that the system can detect incoherence, a.k.a reject bullshit. LLMs today can't do that and will happily emit bullshit in response. Maybe the "gates" in the "workflows" discussed in the Anthropic article are a practical solution to that. But that still just seems like inserting human intelligence into the system for a specific engineering domain; not a general solution.

dr_dshiv 2y ago

I can’t communicate enough how the skepticism (“this is just hype” or “LLMs are stochastic parrots”) is the vastly dominant thought paradigm in European academic circles. So instead of everyone having some enthusiasm and some skepticism, you get a bifurcation where whole classes of people act as the skeptics and others as the enthusiasts. I view the strong skeptics as more “in the wrong” because they often don’t use LLMs much. If you are an actual enthusiastic user, you simply can’t get good performance without a very strong dose of skepticism towards everything LLMs output.

_t9ow 2y ago
> I can’t communicate enough how the skepticism (“this is just hype” or “LLMs are stochastic parrots”) is the vastly dominant thought paradigm in European academic circles. I'm very curious. If you don't mind taking the time to elaborate, will you give a few examples of such skepticism/naysaying? Thank you.
- tessellated 2y ago
  Some "academia" types I meet seem to be struck in this "skeptic vs enthusiast" discussion. How do we harness LLMs’ potential while rigorously mitigating harm?
fragmede 2y ago
I don't think everyone shares those doubts. The first time you catch an LLM in a lie is sobering, but there are lots of areas, and thus lots of users, for whom it doesn't hallucinate for, because they're asking softball questions and it doesn't end up hallucinating, or hallucinations just really aren't aren't that big a deal. (eg an LLM horoscope generator or using it write sci fi.) so while we're on HN going back and forth about how outright lies by the system indight the whole thing for everybody, we should be careful to note that it's not for everybody, or rather, it's a known limitation so don't trust it to cite real cases for you as a lawyer, but using it to help you figure out what mens rea means in a practical sense by asking it questions about the concept, totally. Honestly, hallucinations happen so rarely for me because of the kinds of things I ask it, that it doesn't happen enough for me to not believe it's answers in low-stakes situations, or situations on the level of horoscope generation, and I'm sure I'm not alone in treating ChatGPT that way, despite evidence to the contrary.