4 ms·

<rant> Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.c

<rant> Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando..) they are the ones with the least results. From my observations most SOTAs come from chineses researchers by far, followed by deepmind. BTW isn't that a sad truth that not even one of all major AI actors has a draft of an AGI architecture, something comparable to CYC or opencog. https://wiki.opencog.org/w/CogPrime_Overview https://wiki.opencog.org/w/CogPrime_Overview Two other observations I would like to share: Many important NLP tasks have almost nobody publicly working on them it seems, on paperswithcode.com or NLP-progress (from github) some tasks have only one or two papers... And many others have not evolved since 2016. Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before and hop, free state of the art for you! Yet researchers don't seems to chase those low hanging, high returns fruits. Also researchers seems to work a lot in isolation, many new generic improvements like new optimizers (RAdam for example) and new activation functions (Swish) allow to beat most of older state of the art on almost all task just by using them. Yet researchers will take years before using them because of an absurd inertia. Also unlike an open source program, BERT and XLnet have very low response and activity on github despite major open issues... </rant>

p1esk 7y ago
Many important NLP tasks have almost nobody publicly working on them Well, then perhaps you should go work on them, instead of ranting here.
- The_rationalist 7y ago
  Why the ad hominem? I am pointing a problem of allocation of ressources on the AI research field. It's not to me to fixe that, but yes I am actively working on a logical fallacies detector which is the first of human history and works for the 256 possible forms of syllogisms, I'm expanding it to other logical forms such as modus ponens/tollens.
  - p1esk 7y ago
    It's not to me to fixe that There's nothing to fix. People work on what they want to work on. Things that seem important to you are not important to me, and the opposite. I'm OK with that.
    - The_rationalist 7y ago
      "People work on what they want to work" ideally yes, but ultimately they work on something that please them AND that give them a decent salary. Funding should not go to fun (but useless in the real world) Nlp tasks. "Things that seem important to you are not important to me, and the opposite." and here's go relativism or the abandon of thought... It's indeed difficult to quantify cardinally the utility of an NLP task against an other, but we can agree on an ordinality (order of magnitude) E.g do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. Thus making them the most important NLP tasks as they enable other Nlp tasks and are the most used in practice? You think that what exactly is more important? Are you talking about text generation? Why is that important? Something important enable to solve important problems in the real world. How text generation solve any real world problem is beyond my knowledge. But if you rationally think that it's more important that angular Nlp tasks, you can probably explain why and give an example or two? Yes, an AGI will need to emit text just as humans do, indeed. But before that she needs to understund the natural language before emitting it. GPT-2 maybe capture an aesthetic of the initial input pretty well but it does not generate meaningful sentences or only by accident, so no GPT-2 does not advance the quest to create an intelligent agent mastering natural language.
      - p1esk 7y ago
        do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. I'm not sure. I rarely have to do that explicitly in my head. Perhaps a model should learn to infer/guess them implicitly, from context, just like I do. what exactly is more important? In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or from prior conversational history, or using some externally supplied conditioning) would be far more important. GPT-2 does not generate meaningful sentences or only by accident You think adding POS tags would help it generate meaningful sentences?
- Der_Einzige 7y ago
  Peer review is so shit at major AI conferences that his paper was most likely rejected for nonsensical reasons
  - The_rationalist 7y ago
    Why do you say that? Does it add anything to the conversation?
roseway4 7y ago
Inference, question-answering, NER detection/disambiguation are pretty important NLP tasks (at least from a practitioner's perspective). While GPT-2 has gained mindshare for its generative capabilities, BERT and other pre-trained Transformer Encoder models are used for production workloads (we use BERT for text classification and explored using it for clustering). It's useful to view the GitHub projects for these models as reference implementations. They're intended to provide a roadmap for reproducing the research and to aid in implementing production libraries. Regarding the latter, take a look at the work by HuggingFace, the Flair project, Spark-NLP and others.
- The_rationalist 7y ago
  "Inference, question-answering, NER detection/disambiguation are pretty important NLP tasks" Yes indeed. "While GPT-2 has gained mindshare for its generative capabilities, BERT and other pre-trained Transformer Encoder models are used for production workloads" You rephrased my point pretty well, while openAI search for "fun" tasks, deepmind and others allow progress on real world tasks. You use BERT which is nice but do you consider using it's successor: XLnet? "take a look at the work by HuggingFace, the Flair project, Spark-NLP and others." I was aware of Flair (from Zalando) but thank you for Huggingface and Spark-NLP, I will take a look!
  - deleted 7y ago
    [deleted]
LiamMcCalloway 7y ago
> are there any real use case? Writing Bloomberg's "market wrap" articles.
the8472 7y ago
The article mentions TabNine, which has made it to the HN frontpage before.
- dgellow 7y ago
  For reference, the blog article for the release of "Deep TabNine", the auto-completion engine based on GPT-2: https://www.tabnine.com/blog/deep/ https://www.tabnine.com/blog/deep/. And earlier this month they released a local version: https://www.tabnine.com/blog/local/ https://www.tabnine.com/blog/local/.
inimino 7y ago
> Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before If it was a high-return fruit somebody would be doing it. Not necessarily publishing papers about it or trying to beat useless artificial benchmarks on it.
- The_rationalist 7y ago
  If it was a high-return fruit somebody would be doing it." Not necessarily. Not necessarily publishing papers about it Yes. trying to beat useless artificial benchmarks on it. Wtf is this bullshit? AI benchmarcks are what direct progress in AI and allow to quantify it. And they are less and less artificial and more and more real world: E.g quora, reddit, Wikipedia and Facebook datasets.
  - inimino 7y ago
    >"If it was a high-return fruit somebody would be doing it." Not necessarily. With high likelihood given current funding of ML with $$$ applications but yes, not necessarily. > AI benchmarcks are what direct progress in AI Sadly this is largely true. The AI benchmarks + culture around it are the bullshit. What actually moves forward the field of AI is: - accessible - reproducible - comprehensible results done with some thought and reasoning which is explained well, published well, and justified by more than some #$!& "our F1 score went up by 2 therefore our approach makes sense" bullshit. AI benchmarks have done as much to retard progress in AI as they have to promote it. Current AI benchmark top scores are gamification for big companies to waste even more resources running algorithms they can't explain. They are not machine learning, they are machine pissing contests.

p1esk 7y ago

Many important NLP tasks have almost nobody publicly working on them Well, then perhaps you should go work on them, instead of ranting here.

The_rationalist 7y ago
Why the ad hominem? I am pointing a problem of allocation of ressources on the AI research field. It's not to me to fixe that, but yes I am actively working on a logical fallacies detector which is the first of human history and works for the 256 possible forms of syllogisms, I'm expanding it to other logical forms such as modus ponens/tollens.
- p1esk 7y ago
  It's not to me to fixe that There's nothing to fix. People work on what they want to work on. Things that seem important to you are not important to me, and the opposite. I'm OK with that.
  - The_rationalist 7y ago
    "People work on what they want to work" ideally yes, but ultimately they work on something that please them AND that give them a decent salary. Funding should not go to fun (but useless in the real world) Nlp tasks. "Things that seem important to you are not important to me, and the opposite." and here's go relativism or the abandon of thought... It's indeed difficult to quantify cardinally the utility of an NLP task against an other, but we can agree on an ordinality (order of magnitude) E.g do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. Thus making them the most important NLP tasks as they enable other Nlp tasks and are the most used in practice? You think that what exactly is more important? Are you talking about text generation? Why is that important? Something important enable to solve important problems in the real world. How text generation solve any real world problem is beyond my knowledge. But if you rationally think that it's more important that angular Nlp tasks, you can probably explain why and give an example or two? Yes, an AGI will need to emit text just as humans do, indeed. But before that she needs to understund the natural language before emitting it. GPT-2 maybe capture an aesthetic of the initial input pretty well but it does not generate meaningful sentences or only by accident, so no GPT-2 does not advance the quest to create an intelligent agent mastering natural language.
    - p1esk 7y ago
      do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. I'm not sure. I rarely have to do that explicitly in my head. Perhaps a model should learn to infer/guess them implicitly, from context, just like I do. what exactly is more important? In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or from prior conversational history, or using some externally supplied conditioning) would be far more important. GPT-2 does not generate meaningful sentences or only by accident You think adding POS tags would help it generate meaningful sentences?
Der_Einzige 7y ago
Peer review is so shit at major AI conferences that his paper was most likely rejected for nonsensical reasons
- The_rationalist 7y ago
  Why do you say that? Does it add anything to the conversation?

roseway4 7y ago

Inference, question-answering, NER detection/disambiguation are pretty important NLP tasks (at least from a practitioner's perspective). While GPT-2 has gained mindshare for its generative capabilities, BERT and other pre-trained Transformer Encoder models are used for production workloads (we use BERT for text classification and explored using it for clustering). It's useful to view the GitHub projects for these models as reference implementations. They're intended to provide a roadmap for reproducing the research and to aid in implementing production libraries. Regarding the latter, take a look at the work by HuggingFace, the Flair project, Spark-NLP and others.

The_rationalist 7y ago
"Inference, question-answering, NER detection/disambiguation are pretty important NLP tasks" Yes indeed. "While GPT-2 has gained mindshare for its generative capabilities, BERT and other pre-trained Transformer Encoder models are used for production workloads" You rephrased my point pretty well, while openAI search for "fun" tasks, deepmind and others allow progress on real world tasks. You use BERT which is nice but do you consider using it's successor: XLnet? "take a look at the work by HuggingFace, the Flair project, Spark-NLP and others." I was aware of Flair (from Zalando) but thank you for Huggingface and Spark-NLP, I will take a look!
- deleted 7y ago
  [deleted]

LiamMcCalloway 7y ago

> are there any real use case? Writing Bloomberg's "market wrap" articles.

the8472 7y ago

The article mentions TabNine, which has made it to the HN frontpage before.

dgellow 7y ago
For reference, the blog article for the release of "Deep TabNine", the auto-completion engine based on GPT-2: https://www.tabnine.com/blog/deep/ https://www.tabnine.com/blog/deep/. And earlier this month they released a local version: https://www.tabnine.com/blog/local/ https://www.tabnine.com/blog/local/.

inimino 7y ago

> Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before If it was a high-return fruit somebody would be doing it. Not necessarily publishing papers about it or trying to beat useless artificial benchmarks on it.

The_rationalist 7y ago
If it was a high-return fruit somebody would be doing it." Not necessarily. Not necessarily publishing papers about it Yes. trying to beat useless artificial benchmarks on it. Wtf is this bullshit? AI benchmarcks are what direct progress in AI and allow to quantify it. And they are less and less artificial and more and more real world: E.g quora, reddit, Wikipedia and Facebook datasets.
- inimino 7y ago
  >"If it was a high-return fruit somebody would be doing it." Not necessarily. With high likelihood given current funding of ML with $$$ applications but yes, not necessarily. > AI benchmarcks are what direct progress in AI Sadly this is largely true. The AI benchmarks + culture around it are the bullshit. What actually moves forward the field of AI is: - accessible - reproducible - comprehensible results done with some thought and reasoning which is explained well, published well, and justified by more than some #$!& "our F1 score went up by 2 therefore our approach makes sense" bullshit. AI benchmarks have done as much to retard progress in AI as they have to promote it. Current AI benchmark top scores are gamification for big companies to waste even more resources running algorithms they can't explain. They are not machine learning, they are machine pissing contests.