3 ms·

Google published a lot. I still wonder this original transformer paper, and the attention, etc. Why would they allow it to let go in the open? Perhaps because w

by larodi 3mo ago

Google published a lot. I still wonder this original transformer paper, and the attention, etc. Why would they allow it to let go in the open? Perhaps because was intended for translation first before someone decided to loop it over itself? Or was so obscure to fellow researchers what do they actually publish?

willy_k 3mo ago
It took 6 years to get from Attention Is All You Need to a consumer product. Up until ChatGPT’s release, LLMs were obscure and really were just fancier autocomplete. Deepmind was largely a speculative research division until Google decided to play catch up, and even that took a bit of time as they figured out how to proceed in a way that wouldn’t cannabalize search. So in short they didn’t realize the potential. Thats my somewhat naive take.
- jeffbee 3mo ago
  Google itself was shipping transformer-based features to consumers within 2 years, notably BERT for websearch.
  - umeshunni 3mo ago
    and Google Translate: https://arxiv.org/abs/1804.09849 https://arxiv.org/abs/1804.09849
leoh 3mo ago
Kind of makes you wonder if there are other papers from Google that are potentially also profoundly groundbreaking but overlooked
- whiplash451 3mo ago
  Groundbreaking includes the “explain to the world why it matters” part