3 ms·

Google Books, like so many Google projects, had a dual purpose. Making books accessible is noble and on-mission. But more importantly natural language models ca

Google Books, like so many Google projects, had a dual purpose. Making books accessible is noble and on-mission. But more importantly natural language models can be trained on the scanned corpus. The same was true of the original GOOG 411, which provided a free service, but was really put in place to train up their voice recognition projects. This is a long running strategy of Google, and it's a shrewd one. The main thing is not to mistake it for a public good. It is an act of privatization.

visarga 4y ago
That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. I think Google initially wanted to augment the web results with a large book collection to get "all the world information and make it searchable", same with Google News.
- philsnow 4y ago
  > That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. ... pull the other one. Okay, your statement could be true depending on what you mean by "large". But what makes you think that companies like Google haven't been working on language models without releasing them and/or without discussing them publicly? There's an advantage to be had by keeping corporate secrets.
  - visarga 4y ago
    The gap between research and publication is real but its length is less than one year. I don't think even Google has the resources to do it secretly. Who would work on it and how could they have kept the secret so tight? AI researchers want to publish, especially the best ones, it's essential for their careers. Someone else could plant the flag on their discovery and claim the fame.
    - philsnow 4y ago
      For a time while I was working at Google I nursed the idea of transferring to the natural language processing / language modeling areas, but I only have a bachelors in linguistics whereas it seems like they strongly prefer PhDs. Linguistics PhDs can pretty much teach linguistics or go into another field. It would not be hard for Google to find a couple hundred of them and entice them to join up.
- vintermann 4y ago
  Before there were neural language models, there were n-gram models, skip n-gram models, latent dirichlet allocation models, a whole zoo of non-neural machine learning models. Google used some of them to power old Google Translate, which was still a very impressive piece of technology.
- xavriley 4y ago
  At the time Google Books started, “big data” was in vogue and there were data-driven ML approaches to NLP e.g. word2vec, that would have benefited from a large corpus

visarga 4y ago

That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. I think Google initially wanted to augment the web results with a large book collection to get "all the world information and make it searchable", same with Google News.

philsnow 4y ago
> That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. ... pull the other one. Okay, your statement could be true depending on what you mean by "large". But what makes you think that companies like Google haven't been working on language models without releasing them and/or without discussing them publicly? There's an advantage to be had by keeping corporate secrets.
- visarga 4y ago
  The gap between research and publication is real but its length is less than one year. I don't think even Google has the resources to do it secretly. Who would work on it and how could they have kept the secret so tight? AI researchers want to publish, especially the best ones, it's essential for their careers. Someone else could plant the flag on their discovery and claim the fame.
  - philsnow 4y ago
    For a time while I was working at Google I nursed the idea of transferring to the natural language processing / language modeling areas, but I only have a bachelors in linguistics whereas it seems like they strongly prefer PhDs. Linguistics PhDs can pretty much teach linguistics or go into another field. It would not be hard for Google to find a couple hundred of them and entice them to join up.
vintermann 4y ago
Before there were neural language models, there were n-gram models, skip n-gram models, latent dirichlet allocation models, a whole zoo of non-neural machine learning models. Google used some of them to power old Google Translate, which was still a very impressive piece of technology.
xavriley 4y ago
At the time Google Books started, “big data” was in vogue and there were data-driven ML approaches to NLP e.g. word2vec, that would have benefited from a large corpus