5 ms·

Didn't Google have a project to scan and OCR all the books? I wonder whether these data were fed to Bard.

lobstersammich 3y ago
You can find GPT-2's training dataset list - at a high level - in the GPT-2 repository on Github: https://github.com/openai/gpt-2/blob/master/model_card.md#datasets https://github.com/openai/gpt-2/blob/master/model_card.md#da... However, OpenAI goes dark after that regarding the 'data soup' that was fed into their LLMs. In general, start around 2019 and definitely by 2020 you'll notice that research labs became much less forthcoming about the data that went into their models. As far as I'm aware, BookCorpus is one of the more commonly-used 'large books dataset' that's been utilized in recent years to train large language models (LLMs) like generative pretrained transformers: https://12ft.io/proxy?q=https%3A%2F%2Ftowardsdatascience.com%2Fdirty-secrets-of-bookcorpus-a-key-dataset-in-machine-learning-6ee2927e8650 https://12ft.io/proxy?q=https%3A%2F%2Ftowardsdatascience.com... At my alma mater I remember the large-scale Google book scanning devices and what a herculean effort that was to digitize the largest university library system's books - University of Michigan - although only 7M texts from the entire collection of ~16 million texts: https://en.wikipedia.org/wiki/University_of_Michigan_Library https://en.wikipedia.org/wiki/University_of_Michigan_Library) were digitized.I too was curious about the state of the Google Books project: https://www.edsurge.com/news/2017-08-10-what-happened-to-google-s-effort-to-scan-millions-of-university-library-books https://www.edsurge.com/news/2017-08-10-what-happened-to-goo... This is an interesting piece of ephemera from 2005, when Google started digitizing books at UMich: https://apps.lib.umich.edu/files/services/mdp/faq.pdf https://apps.lib.umich.edu/files/services/mdp/faq.pdf As far as I recall, the Books project allowed the early n-grams functionality to be built out: https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.html https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-... The Google Books Ngram Viewer tool is actually still in existence; you can play around with it here: https://books.google.com/ngrams/graph?corpus=0&content=Vorsprung%20durch%20Technik https://books.google.com/ngrams/graph?corpus=0&content=Vorsp...
codr7 3y ago
If that was the case, it threw more than half of it up again, because it's not making much sense atm.
samstave 3y ago
I'd be crazy if I didnt think that google is sitting on some stuff that nobody knows about and they are stroking their cat from the lair as we type.
- JimtheCoder 3y ago
  "...and they are stroking their cat from the lair..." On the first quick read though, I thought to myself, "Can he use that sort of language here?" Then I pictured Dr. Evil and it made more sense...
  - jhbadger 3y ago
    I think Blofeld was the reference. Dr Evil is a parody of Blofeld.
- Spooky23 3y ago
  It’s funny that the general internet pessimism about Google misses stuff like this. I mean ChatGPT 3 went viral and Google managed to ship Bard in a few weeks. I think the consensus is that ChatGPT is better, but it was literally sitting on the shelf ready to go.
- m4rtink 3y ago
  The cat has been deprecated half a year ago. ;-)
qingcharles 3y ago
Yes, and while there were copyright issues with them putting the books out there in public, they still retain all the scans to use for search projects. https://books.google.com/ https://books.google.com/
PeterisP 3y ago
It was claimed to use book data, but IMHO nowadays the available internet data is larger than all the books ever published; so while book data definitely should be used, it's not a pathway to significant increases in data size.

You can find GPT-2's training dataset list - at a high level - in the GPT-2 repository on Github: https://github.com/openai/gpt-2/blob/master/model_card.md#datasets https://github.com/openai/gpt-2/blob/master/model_card.md#da... However, OpenAI goes dark after that regarding the 'data soup' that was fed into their LLMs. In general, start around 2019 and definitely by 2020 you'll notice that research labs became much less forthcoming about the data that went into their models. As far as I'm aware, BookCorpus is one of the more commonly-used 'large books dataset' that's been utilized in recent years to train large language models (LLMs) like generative pretrained transformers: https://12ft.io/proxy?q=https%3A%2F%2Ftowardsdatascience.com%2Fdirty-secrets-of-bookcorpus-a-key-dataset-in-machine-learning-6ee2927e8650 https://12ft.io/proxy?q=https%3A%2F%2Ftowardsdatascience.com... At my alma mater I remember the large-scale Google book scanning devices and what a herculean effort that was to digitize the largest university library system's books - University of Michigan - although only 7M texts from the entire collection of ~16 million texts: https://en.wikipedia.org/wiki/University_of_Michigan_Library https://en.wikipedia.org/wiki/University_of_Michigan_Library) were digitized.I too was curious about the state of the Google Books project: https://www.edsurge.com/news/2017-08-10-what-happened-to-google-s-effort-to-scan-millions-of-university-library-books https://www.edsurge.com/news/2017-08-10-what-happened-to-goo... This is an interesting piece of ephemera from 2005, when Google started digitizing books at UMich: https://apps.lib.umich.edu/files/services/mdp/faq.pdf https://apps.lib.umich.edu/files/services/mdp/faq.pdf As far as I recall, the Books project allowed the early n-grams functionality to be built out: https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.html https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-... The Google Books Ngram Viewer tool is actually still in existence; you can play around with it here: https://books.google.com/ngrams/graph?corpus=0&content=Vorsprung%20durch%20Technik https://books.google.com/ngrams/graph?corpus=0&content=Vorsp...

codr7 3y ago

If that was the case, it threw more than half of it up again, because it's not making much sense atm.

samstave 3y ago

I'd be crazy if I didnt think that google is sitting on some stuff that nobody knows about and they are stroking their cat from the lair as we type.

JimtheCoder 3y ago
"...and they are stroking their cat from the lair..." On the first quick read though, I thought to myself, "Can he use that sort of language here?" Then I pictured Dr. Evil and it made more sense...
- jhbadger 3y ago
  I think Blofeld was the reference. Dr Evil is a parody of Blofeld.
Spooky23 3y ago
It’s funny that the general internet pessimism about Google misses stuff like this. I mean ChatGPT 3 went viral and Google managed to ship Bard in a few weeks. I think the consensus is that ChatGPT is better, but it was literally sitting on the shelf ready to go.
m4rtink 3y ago
The cat has been deprecated half a year ago. ;-)

qingcharles 3y ago

Yes, and while there were copyright issues with them putting the books out there in public, they still retain all the scans to use for search projects. https://books.google.com/ https://books.google.com/

PeterisP 3y ago

It was claimed to use book data, but IMHO nowadays the available internet data is larger than all the books ever published; so while book data definitely should be used, it's not a pathway to significant increases in data size.