4 ms·
Google has a data moat on traditional search. Its really hard to train a traditional search system without google scale data. LLMs effectively side stepped that
by ldjkfkdsjnv 3y ago
Google has a data moat on traditional search. Its really hard to train a traditional search system without google scale data. LLMs effectively side stepped that whole issue
- ben_w 3y agoGoogle's special sauce, is the data of all user's responses to the things Google's algorithm shows them. This description also works for ChatGPT, what it has which others do not (at anything close to the same scale). Google's search engine is a well understood bit of matrix multiplication on a scale many others can easily afford to replicate. This description also works for GPT-3.
- ldjkfkdsjnv 3y agoNo becuause ChatGPT was trained on the whole internet using a new algorithm. You needed perfect tagged data for the old algorithm (page rank). LLMs are a fundamental leap forward
- ben_w 3y agoThe GPT algorithm isn't a secret. The RLFH data — users giving a thumbs up or down, or regenerating a response, or even getting detectably angry in their subsequent messages — is. PageRank isn't a secret. Google's version of RLFH — which link does a user click on, do they ignore the results without clicking any, do they write several related queries before clicking a link, do they return to the search results soon after clicking a link — are also secret. That the Transformer model is a breakthrough doesn't make it a moat; that the Transformer model is public doesn't mean people using it don't have a moat. Hence why I'm criticising the use of "moat" as mere parroting.