4 ms·
A new 7B model, Snorkel-Mistral-PairRM-DPO, using a similar self-rewarding pipeline was just released: * Announcement: https://twitter.com/billyuchenlin/status
by lhl 3y ago
A new 7B model, Snorkel-Mistral-PairRM-DPO, using a similar self-rewarding pipeline was just released:
* Announcement: https://twitter.com/billyuchenlin/status/1749975138307825933 https://twitter.com/billyuchenlin/status/1749975138307825933
* Model Card: https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO
* Response Re-Ranker: https://huggingface.co/llm-blender/PairRM https://huggingface.co/llm-blender/PairRM
"We would also like to acknowledge contemporary work published independently on arXiv on 2024-01-18 by Meta & NYU (Yuan, et al) in a paper called Self-Rewarding Language Models, which proposes a similar general approach for creating alignment pairs from a larger set of candidate responses, but using the LLM as the reward model. While this may work for general-purpose models, our experience has shown that task-specific reward models guided by SMEs are necessary for most enterprise applications of LLMs for specific use cases, which is why we focus on the use of external reward models."
- azinman2 3y agoI assume this doesn’t yet run on llama.cpp?
- tarruda 3y agoIt is based on Mistral which llama.cpp supports, so I assume it does run (you might need to convert to GGUF format and quantize it).
- lhl 3y agohere are some GGUFs https://huggingface.co/brittlewis12/Snorkel-Mistral-PairRM-DPO-GGUF https://huggingface.co/brittlewis12/Snorkel-Mistral-PairRM-D...
- mark_l_watson 3y agoThanks for this, and the link you provided below for GGUF files! I just cleared my schedule this afternoon to kick the tires.
- spidersenses 3y ago>Snorkel-Mistral-PairRM-DPO The naming of these models is getting ridiculous...
- Loic 3y agoA bit like the User-Agent string.
- column 3y agoI kind of disagree. It's not "user friendly" but it is very descriptive. They are codenames afterall. Take "dolphin-2.6-mistral-7b-dpo-laser" for instance : with a little LLM background knowledge, just from the name you know it is a 7 billion parameters model based on Mistral, with a filtered dataset to remove alignment and bias (dolphin), version 2.6 and using the techniques described in the Direct Preference Optimization (https://arxiv.org/pdf/2305.18290.pdf https://arxiv.org/pdf/2305.18290.pdf) and Laser (https://arxiv.org/pdf/2312.13558.pdf https://arxiv.org/pdf/2312.13558.pdf) papers to improve its output.
- spidersenses 3y agoThank you for a great and informative explanation despite my somewhat ignorant take. I'm an occasional visitor to huggingface, so I'm actually superficially familiar with the taxonomy. I just felt like, even if I tried to satirize it, I wouldn't be able to come up with a crazier name. And that's not even the end of the Cambrian explosion of LLMs.
- salad-tycoon 3y agoAnd here I was thinking they were somehow using the first three words from my Bitcoin wallet.