3 ms·

This is pretty cool! However, these two samples are very simple to solve. I'd love an "AI" to find root causes for problems that are not obvious. Just throw th

by bbu 6y ago

This is pretty cool! However, these two samples are very simple to solve. I'd love an "AI" to find root causes for problems that are not obvious. Just throw the whole log collection at it and let it solve all the issues. One can dream ;)

stochastimus 6y ago
Yeah, my experience so far has been that if I just pile a bunch of logs in there, if it's not given salient lines, the language model tends to either rat-hole on some particular detail that's irrelevant, or else construct a non-factual narrative. But, when Zebrium did the picking of these lines autonomously and GPT-3 summarized them, we see a meaningful summary. Having said that, I would also like to get GPT-3 more and more savvy with larger and larger log-based prompts, and I'm hoping it's possible with some tweaks to the prompt and some finetuning. I'll keep the blog posted as we do more experiments.
- a-dub 6y ago
  this is really cool!
deleted 6y ago
[deleted]
m463 6y ago
GPT-3-stackoverflow
- deeeeplearning 6y ago
  But seriously, was Stackoverflow part of the training data used to train GPT-3? Would definitely be an interesting fine tuning experiment
  - stochastimus 6y ago
    From what I've read, the answer is "yes", stackoverflow was crawled. EDIT: I looked and stackoverflow is included in the Common Crawl dataset, which is one of the datasets on which GPT-3 was trained. Having said that, it's not clear to me the degree of coverage of that domain guaranteed by that crawl... looks pretty comprehensive, though. http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.stackoverflow.com&output=json http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.sta...
    - sillysaurusx 6y ago
      The training data was so large that I don’t think a single epoch was used. I.e. it’s not a guarantee that any specific site is known by GPT-3 just because it’s in the data. But, with the nature of random samples, it’s probably likely that it saw at least a little bit of SO.

by bbu 6y ago

stochastimus 6y ago

Yeah, my experience so far has been that if I just pile a bunch of logs in there, if it's not given salient lines, the language model tends to either rat-hole on some particular detail that's irrelevant, or else construct a non-factual narrative. But, when Zebrium did the picking of these lines autonomously and GPT-3 summarized them, we see a meaningful summary. Having said that, I would also like to get GPT-3 more and more savvy with larger and larger log-based prompts, and I'm hoping it's possible with some tweaks to the prompt and some finetuning. I'll keep the blog posted as we do more experiments.

a-dub 6y ago
this is really cool!

deleted 6y ago

[deleted]

m463 6y ago

GPT-3-stackoverflow

deeeeplearning 6y ago
But seriously, was Stackoverflow part of the training data used to train GPT-3? Would definitely be an interesting fine tuning experiment
- stochastimus 6y ago
  From what I've read, the answer is "yes", stackoverflow was crawled. EDIT: I looked and stackoverflow is included in the Common Crawl dataset, which is one of the datasets on which GPT-3 was trained. Having said that, it's not clear to me the degree of coverage of that domain guaranteed by that crawl... looks pretty comprehensive, though. http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.stackoverflow.com&output=json http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.sta...
  - sillysaurusx 6y ago
    The training data was so large that I don’t think a single epoch was used. I.e. it’s not a guarantee that any specific site is known by GPT-3 just because it’s in the data. But, with the nature of random samples, it’s probably likely that it saw at least a little bit of SO.