4 ms·

Depending on definition, it is solved.

deleted 4y ago
[deleted]
chartpath 4y ago
Even if that were true, LLMs don't give any kind of "handles" on the semantics. You just get what you get and have to hope it is tuned for your domain. This is 100% fine for generic consumer-facing services where the training data is representative, but for specialized and jargon-filled domains where there has to be a very opinionated interpretation of words, classical NLU is really the only ethical choice IMHO.
gattilorenz 4y ago
You're using the wrong definition, then. /s Where is some evidence that NLP is 'solved'? What does it even mean? OpenAI itself acknowledges the fundamental limitations of ChatGPT and the method of training it, but apparently everybody is happily sweeping them under the rug: "ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows." (from https://openai.com/blog/chatgpt https://openai.com/blog/chatgpt ) Certainly ChatGPT/GPT-4 are impressive accomplishments, and it doesn't mean they won't be useful, but we were pretty sure in the past that we had "solved" AI or that we were just about to crack it, just give it a few years... except there's always a new rabbit hole to fall into waiting for you.
- gonzo41 4y ago
  It'd be great if GPT could provide it's sources for the text it generated. I've been asking it about lyrics from songs that I know of, but where I can't find the original artist listed. I was hoping chat gpt had consumed a stack of lyrics and I could just ask it, "What song has this chorus or one similar to X..." It didn't work. Instead it firmly stated the wrong answer. And when I gave it time ranges it just noped out of there. I think If I could ask it a question and it could go, I've used these 20-100 sources directly to synthesize this information, it'd be very helpful.
  - IanCal 4y ago
    Have you tried bing chat? That search & sourcing is exactly what it does.
    - gonzo41 4y ago
      Not yet, I'll try at work on my windows box. Thanks.
    - snowwrestler 4y ago
      Sure, but the sources list is generated by the same system that generated the text, so it’s equally subject to hallucinations. Some examples in here: https://dkb.blog/p/bing-ai-cant-be-trusted https://dkb.blog/p/bing-ai-cant-be-trusted To answer the question above, these systems cannot provide sources because they don’t work that way. Their source for everything is, basically, everything. They are trained on a huge corpus of text data and every output depends on that entire training. They have no way to distinguish or differentiate which piece of the training data was the “actual” or “true” source of what they generated. It’s like the old questions “which drop caused the flood” or “which pebble caused the landslide”.
      - IanCal 4y ago
        The issues are good to know about but > Their source for everything is, basically, everything. They are trained on a huge corpus of text data and every output depends on that entire training. Bing chat is explicitly taking in extra data. It's a distinctly different setup from chatgpt.
- throwaway4aday 4y ago
  Is the goal of NLP for the model to actually understand the language it is processing? By understand I mean having the ability to relate the language to the real world and reason about it the same way a human would. To me, that goes far beyond NLP into true AI territory where the "model" is at the least conscious of its environment and possesses a true memory of past experiences. Maybe it would not be consciously aware of its self but it would be damn close. I think LLMs have essentially solved the natural language processing problem but they have not solved reasoning or logical abilities including mathematics.
  - gattilorenz 4y ago
    LLMs have (maybe/probably) solved the language modeling problem, sure. That’s hardly NLP, right? NLG is more than “producing text with no semantics” and both NLG and NLU are only part of NLP. ChatGPT cannot even reason reliably on what it knows and doesn’t know… it’s the library of Babel, but every book is written in excellent English.
    - nl 4y ago
      Fluency = NLP Knowledge representation is a separate problem. NLP gives us some insights into what works here, but the multi-modal aspects of things like GPT4 show there is a lot more to knowledge presentation than just NLP.
- nl 4y ago
  Natural language processing/understanding/generation is solved (at least in English). LLMs produce perfectly fluent output and can understand natural language input as well as any human. However knowledge representation is not solved. We still don't know how to interface a perfect LLM to other systems in the same way a human does things like looking up facts we aren't confident of or using a calculator to do math we cant' do in our head. These are very significant problems and super important. But they are more adjacent to NLP in the same way tasks like something like Text-to-SQL [1] isn't a pure NLP task. [1] for example https://github.com/salesforce/WikiSQL https://github.com/salesforce/WikiSQL

deleted 4y ago

[deleted]

Even if that were true, LLMs don't give any kind of "handles" on the semantics. You just get what you get and have to hope it is tuned for your domain. This is 100% fine for generic consumer-facing services where the training data is representative, but for specialized and jargon-filled domains where there has to be a very opinionated interpretation of words, classical NLU is really the only ethical choice IMHO.

gattilorenz 4y ago

You're using the wrong definition, then. /s Where is some evidence that NLP is 'solved'? What does it even mean? OpenAI itself acknowledges the fundamental limitations of ChatGPT and the method of training it, but apparently everybody is happily sweeping them under the rug: "ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows." (from https://openai.com/blog/chatgpt https://openai.com/blog/chatgpt ) Certainly ChatGPT/GPT-4 are impressive accomplishments, and it doesn't mean they won't be useful, but we were pretty sure in the past that we had "solved" AI or that we were just about to crack it, just give it a few years... except there's always a new rabbit hole to fall into waiting for you.

gonzo41 4y ago
It'd be great if GPT could provide it's sources for the text it generated. I've been asking it about lyrics from songs that I know of, but where I can't find the original artist listed. I was hoping chat gpt had consumed a stack of lyrics and I could just ask it, "What song has this chorus or one similar to X..." It didn't work. Instead it firmly stated the wrong answer. And when I gave it time ranges it just noped out of there. I think If I could ask it a question and it could go, I've used these 20-100 sources directly to synthesize this information, it'd be very helpful.
- IanCal 4y ago
  Have you tried bing chat? That search & sourcing is exactly what it does.
  - gonzo41 4y ago
    Not yet, I'll try at work on my windows box. Thanks.
  - snowwrestler 4y ago
    Sure, but the sources list is generated by the same system that generated the text, so it’s equally subject to hallucinations. Some examples in here: https://dkb.blog/p/bing-ai-cant-be-trusted https://dkb.blog/p/bing-ai-cant-be-trusted To answer the question above, these systems cannot provide sources because they don’t work that way. Their source for everything is, basically, everything. They are trained on a huge corpus of text data and every output depends on that entire training. They have no way to distinguish or differentiate which piece of the training data was the “actual” or “true” source of what they generated. It’s like the old questions “which drop caused the flood” or “which pebble caused the landslide”.
    - IanCal 4y ago
      The issues are good to know about but > Their source for everything is, basically, everything. They are trained on a huge corpus of text data and every output depends on that entire training. Bing chat is explicitly taking in extra data. It's a distinctly different setup from chatgpt.
throwaway4aday 4y ago
Is the goal of NLP for the model to actually understand the language it is processing? By understand I mean having the ability to relate the language to the real world and reason about it the same way a human would. To me, that goes far beyond NLP into true AI territory where the "model" is at the least conscious of its environment and possesses a true memory of past experiences. Maybe it would not be consciously aware of its self but it would be damn close. I think LLMs have essentially solved the natural language processing problem but they have not solved reasoning or logical abilities including mathematics.
- gattilorenz 4y ago
  LLMs have (maybe/probably) solved the language modeling problem, sure. That’s hardly NLP, right? NLG is more than “producing text with no semantics” and both NLG and NLU are only part of NLP. ChatGPT cannot even reason reliably on what it knows and doesn’t know… it’s the library of Babel, but every book is written in excellent English.
  - nl 4y ago
    Fluency = NLP Knowledge representation is a separate problem. NLP gives us some insights into what works here, but the multi-modal aspects of things like GPT4 show there is a lot more to knowledge presentation than just NLP.
nl 4y ago
Natural language processing/understanding/generation is solved (at least in English). LLMs produce perfectly fluent output and can understand natural language input as well as any human. However knowledge representation is not solved. We still don't know how to interface a perfect LLM to other systems in the same way a human does things like looking up facts we aren't confident of or using a calculator to do math we cant' do in our head. These are very significant problems and super important. But they are more adjacent to NLP in the same way tasks like something like Text-to-SQL [1] isn't a pure NLP task. [1] for example https://github.com/salesforce/WikiSQL https://github.com/salesforce/WikiSQL