4 ms·

GPT Is Rather Good at Feed Ranking

2 points by sebg 4y ago

PaulHoule 4y ago
I like it! But... I am doing something like this for my intelligent feed reader but using local models instead of the GPT API and using moderate-shot instead of zero-shot. Models like RoBERTa and miniLM frequently can learn pretty good models with a few hundred or thousand examples. I'd point to https://arxiv.org/pdf/2204.03954.pdf https://arxiv.org/pdf/2204.03954.pdf Now for years people thought it would be more scalable for some big co (say Facebook) to train a big model for everybody but that list in there of problems, particularly 2-7 and 10-12 are problems that only exist for the big company and that have to do with developing standards for the whole community. An individual might love pornography or hate pornography and have no problem at all expressing their preferences, but Google is forced to split the difference and split hairs that no individual needs to split. It's got a huge amount of resources to deal with it that somewhat compensate for that, but it's a big problem. When I wrote this paper https://www.pnas.org/doi/abs/10.1073/pnas.0308253100 https://www.pnas.org/doi/abs/10.1073/pnas.0308253100 I was looking at all the failing RSS readers with the same failing interface and thinking "Why doesn't somebody make one that uses proven text classification technology?" and believed mostly that it was because people were lazy, very few people would tolerate classifying a few thousand documents before it started to learn. I thought most people would classify at most 4 or 5, even 50 would be too much. In December I was sick and tired of Twitter and Tik Tok and decided to put into action what I'd been thinking about for 18 years. With a simple model and some ideas from active learning my system came on pretty strong with 200 judgements and even though the ROC-AUC hasn't budged much since it got 1500 judgements my satisfaction with the results has improved greatly since then for reasons I don't really understand. In theory I'm still interested in the "few shot" case and I've got a vast amount of data to run simulations on that now, but frankly I don't care much about it because I have 15,000+ judgements now and few-shot isn't a pressing problem. Big picture, ChatGPT is overkill for this, you don't need something HFRL'd to be agreeable and sound like it knows what it is talking about, you need something closer to a cross-encoder https://sbert.net/examples/applications/cross-encoder/README.html https://sbert.net/examples/applications/cross-encoder/README... but the left and right sides are asymmetric. The left is some description of what you are "searching for" (it's been obvious to me for 20 years that real improvements in search relevance and usability involve treating search as a classification problem where the query is a parameter to the classifier) and the right is the document. Then you need to be able to hold the left side constant and fine tune it varying the right.

2 points by sebg 4y ago

I like it! But... I am doing something like this for my intelligent feed reader but using local models instead of the GPT API and using moderate-shot instead of zero-shot. Models like RoBERTa and miniLM frequently can learn pretty good models with a few hundred or thousand examples. I'd point to https://arxiv.org/pdf/2204.03954.pdf https://arxiv.org/pdf/2204.03954.pdf Now for years people thought it would be more scalable for some big co (say Facebook) to train a big model for everybody but that list in there of problems, particularly 2-7 and 10-12 are problems that only exist for the big company and that have to do with developing standards for the whole community. An individual might love pornography or hate pornography and have no problem at all expressing their preferences, but Google is forced to split the difference and split hairs that no individual needs to split. It's got a huge amount of resources to deal with it that somewhat compensate for that, but it's a big problem. When I wrote this paper https://www.pnas.org/doi/abs/10.1073/pnas.0308253100 https://www.pnas.org/doi/abs/10.1073/pnas.0308253100 I was looking at all the failing RSS readers with the same failing interface and thinking "Why doesn't somebody make one that uses proven text classification technology?" and believed mostly that it was because people were lazy, very few people would tolerate classifying a few thousand documents before it started to learn. I thought most people would classify at most 4 or 5, even 50 would be too much. In December I was sick and tired of Twitter and Tik Tok and decided to put into action what I'd been thinking about for 18 years. With a simple model and some ideas from active learning my system came on pretty strong with 200 judgements and even though the ROC-AUC hasn't budged much since it got 1500 judgements my satisfaction with the results has improved greatly since then for reasons I don't really understand. In theory I'm still interested in the "few shot" case and I've got a vast amount of data to run simulations on that now, but frankly I don't care much about it because I have 15,000+ judgements now and few-shot isn't a pressing problem. Big picture, ChatGPT is overkill for this, you don't need something HFRL'd to be agreeable and sound like it knows what it is talking about, you need something closer to a cross-encoder https://sbert.net/examples/applications/cross-encoder/README.html https://sbert.net/examples/applications/cross-encoder/README... but the left and right sides are asymmetric. The left is some description of what you are "searching for" (it's been obvious to me for 20 years that real improvements in search relevance and usability involve treating search as a classification problem where the query is a parameter to the classifier) and the right is the document. Then you need to be able to hold the left side constant and fine tune it varying the right.