4 ms·

Hi Richard, Please feel free to respond to as many questions as you have time for. 1. I'm a huge fan of your ML work while you were at Stanford. If I may ste

Hi Richard, Please feel free to respond to as many questions as you have time for. 1. I'm a huge fan of your ML work while you were at Stanford. If I may steal a minute or two from you, I'd like to ask, why do you think recursive neural networks have lost out to other architectures (basically transformers) when it comes to NLP, where intuition would suggest the composionality of language would favor recursive NNs? Do you think recursive NNs will make a come-back sometime? 2. Also, you say on HN that you.com is "the world's first open search engine platform that summarizes the web for you". Do you mean this in the sense of summarization in the NLP sense, or some other interpretation? I did a couple of searches on you.com, and I didn't notice anything that "summarized" (in the NLP sense). 3. From the business side, what's to stop Google from straight up copying your features if you gain traction? They can certainly afford to pay for talent, and are not known to be sloths when it comes to new tech.

richardsocher 5y ago
Great deep questions :) 1. The driving force behind neural net architecture choice right now is hardware architecture. The model that can be trained with more data and more parameters, will win over any other model. Parallelization is amazing. I do think they'll make a comeback when people try to do more high level reasoning and try to learn that. 2. I do mean it in several senses. We looked into neural net algos for summarization but they often produce outputs that sound good but might not be factual. The other problem is that when you search for "best thai restaurants near me" you don't want a nnet lm - generated text. Most ppl would prefer a list of tiles with restaurants and important facts about them. So the models will need to be multimodal. Lastly, each app is essentially one dimension of looking at a query/topic. You can always "zoom" into each dimension more by swiping left/right or choose a different way of looking at this query by swiping up/down. 3. They do have all the talent but it's hard to copy privacy features for Google because they make too much money "selling" it to advertisers (ie making it accessible to hyper target people everywhere online/on the phone/etc). It'll also be hard to copy our entire design and apps and give people agency because they hyper optimized their design for maximizing ad revenue. They also won't open up their platform to let anybody contribute or people decide the order of their results. Etc.... So... I'm just a little worried, not too much :)
- davidatbu 5y ago
  Great, deep responses :) 1. Wonder if improving training speed / parallelization for recursive NNs is a good research direction ... 2. Thanks for the insights. 3. That all makes sense.

richardsocher 5y ago

Great deep questions :) 1. The driving force behind neural net architecture choice right now is hardware architecture. The model that can be trained with more data and more parameters, will win over any other model. Parallelization is amazing. I do think they'll make a comeback when people try to do more high level reasoning and try to learn that. 2. I do mean it in several senses. We looked into neural net algos for summarization but they often produce outputs that sound good but might not be factual. The other problem is that when you search for "best thai restaurants near me" you don't want a nnet lm - generated text. Most ppl would prefer a list of tiles with restaurants and important facts about them. So the models will need to be multimodal. Lastly, each app is essentially one dimension of looking at a query/topic. You can always "zoom" into each dimension more by swiping left/right or choose a different way of looking at this query by swiping up/down. 3. They do have all the talent but it's hard to copy privacy features for Google because they make too much money "selling" it to advertisers (ie making it accessible to hyper target people everywhere online/on the phone/etc). It'll also be hard to copy our entire design and apps and give people agency because they hyper optimized their design for maximizing ad revenue. They also won't open up their platform to let anybody contribute or people decide the order of their results. Etc.... So... I'm just a little worried, not too much :)

davidatbu 5y ago
Great, deep responses :) 1. Wonder if improving training speed / parallelization for recursive NNs is a good research direction ... 2. Thanks for the insights. 3. That all makes sense.