5 ms·

One of the team members from Project Turing. Happy to answer any questions.

• osipov 7y ago
  Why the lack of number on the more popular SQuAD and Glue benchmarks?
  • saurkt 7y ago
    SQUAD and GLUE are tasks for language representation models -- aka BERT-like. This is a language generation model -- GPT-like. Hence, SQUAD/GLUE test sets are not really applicable. We are reporting on the wikitext and lambada sets that openAI also uses for similar models (numbers are in the blogpost).
    • igravious 7y ago
      What's the difference between the two models?
      • octbash 7y ago
        One is a language generation model, the other is a fill-in-the-blank model. It sounds like they might be similar, but in practice they are different enough objectives (and in particular the "bi-directional" aspect of BERT-type models) that the models learn different things.
      • Voloskaya 7y ago
        * BERT & language representation models: They basically turn a sentence into a compact vector that represents it so you can then do some downstream task on it such as sentiment detection, or matching the similarity between two sentences etc. * GPT & language generation models: Given some context (say a sentence), they can generate text to complete it, or to summarize it, etc. The task here is to actually write something.
        • riku_iki 7y ago
          Both are language representation models, text generation is just a way of training model. BERT is also trained on text generation task: it asked to fill gaps in text (15% of text is blanked during training).
          • Voloskaya 7y ago
            Maybe I am not understanding your point. Out of the box, given a sequence of n tokens, BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary. You can then fine-tune a model on this representation to do various tasks, e.g. sentiment classification. Thus BERT is said to be a language representation model. Out of the box, given a sequence, GPT-2 returns a distribution over the vocabulary [2] from which you can draw to find the most likely next word. Thus GPT-2 is said to be a language generation model. You could of course play with the masking token of BERT call it recursively to force BERT to generate something, and you could chop off some layers of GPT-2 to get some representation of your input sequence, but I think that is a little past the original question. [1] https://github.com/google-research/bert/blob/master/modeling.py#L230 https://github.com/google-research/bert/blob/master/modeling... [2] https://github.com/openai/gpt-2/blob/master/src/model.py#L172 https://github.com/openai/gpt-2/blob/master/src/model.py#L17...
            • riku_iki 7y ago
              > BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary "BERT returns" is ambiguous here. During pretraining last layer is loggits for one hot vocab vector, the same as in GPT: https://github.com/google-research/bert/blob/master/run_pretraining.py#L265 https://github.com/google-research/bert/blob/master/run_pret...
• flowerlad 7y ago
  How does it compare to Google’s BERT and do you have an online demo? Here’s a demo of BERT https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/ https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/
  • saurkt 7y ago
    (Similar to the response for another question.) BERT is a language representation model while Turing-NLG is a language generation model (similar to GPT). They are not directly comparable (they can potentially be massaged to mimic the other, but, not something that we have done yet.)
    • SomewhatLikely 7y ago
      Google's T5 paper pretty convincingly combines the two doesn't it?
• alexwg 7y ago
  Have you evaluated against the AI2 Leaderboard benchmarks? https://leaderboard.allenai.org/ https://leaderboard.allenai.org/
  • saurkt 7y ago
    Not yet. We will try to run against those benchmarks soon.
• Tenoke 7y ago
  Any plans on training other (non nlp) huge models using ZeRO? Specifically for Transformers - any plans to train a big model with a bigger context window? Not that this one isn't very impressive, of course.
  • saurkt 7y ago
    Thanks for your kind words. Yes, we would like to next train a language representation model. And our hunch is that probably something which is a mixture of language representation and language generation would be able to get the best of both worlds.
• hatsuseno 7y ago
  How close do you think the technology is to answering -this- question?
  • throwawayhhakdl 7y ago
    1) How close do you think the technology is to answering -this- question? Four days! 2) How long in years? Three years!
  • deleted 7y ago
    [deleted]
• timgilbert 7y ago
  I'm honestly not trolling with this question, but can you explain what the practical applications of text generation are? From what I've seen of GPT-2, it's a cool toy, but I have never seen it create anything that seems like it would be useful to solve a problem (eg, a human-computer interaction problem). The only applications I can think of for text generation are malevolent ones: I'm sure it would be great at generating spam sites which can fool Google's PageRank algorithms, and it seems like you could easily use it in an information warfare / astroturf setting where you could generate the illusion of consensus by arming a lot of bots with short, somewhat convincing opinions about a certain topic. Is there something obvious I'm missing? It seems too imprecise to actually deliver meaningful information to an end-user, so I'm frankly baffled as to what its purpose is.
• choppaface 7y ago
  “We are releasing a private demo of T-NLG, including its freeform generation, question answering, and summarization capabilities, to a small set of users within the academic community for initial testing and feedback.” What’s the deal with these private demos? (GPT-2 was also essentially private). More importantly, why even announce the existence of a private demo to people who were not invited?