4 ms·
Attention models learn what to pay attention to. It's been found that data that begin with "Wikipedia:" are automatically weighted higher by language models du
by HeatrayEnjoyer 2y ago
Attention models learn what to pay attention to.
It's been found that data that begin with "Wikipedia:" are automatically weighted higher by language models during training, completely unsupervised.
- blueflow 2y agoBut this is the same problem - Wikipedia is a secondary source and should always get overruled by the primary source.
- dotancohen 2y agoAnd there are documented cases of concentrated efforts to manipulate the content of Wikipedia.
- brookst 2y agoYou’re probably being facetious but just in case: https://www.bbc.com/news/technology-28481876.amp https://www.bbc.com/news/technology-28481876.amp https://www.bbc.com/news/technology-58559412.amp https://www.bbc.com/news/technology-58559412.amp (With apologies for amp links)
- dotancohen 2y agoI was most certainly not being facetious.
- HeatrayEnjoyer 2y agoIt's still better than the information on most of the internet, which is what most of the dataset is.