4 ms·
Evals: a framework for evaluating OpenAI models and a registry of benchmarks
- aliljet 4y agoI think this is a clever way for OpenAI to trade access to GPT-4 and simultaneously decrease their own intellectual burden for their for-profit models. Here's the offending parts of the README.md: "For a limited time, we will be granting GPT-4 access to those who contribute high quality evals. Please follow the instructions mentioned above and note that spam or low quality submissions will be ignored!"
- nabakin 4y agoI'm surprised they would give it out when millions, if not billions are potentially at stake and following the LLaMA leak
- Centigonal 4y agoAccess to the inference API, not model code or weights.
- not-chatgpt 4y agoto be fair, with alpaca, access to API is even dangerous
- malborodog 4y agohow so??
- nabakin 4y agoAre you sure? API access is already given to everyone I think Edit: We have access through ChatGPT+, not the API. Got confused there.
- monkeydust 4y agoNo, its on waitlist.
- nabakin 4y agoTy
- astrange 4y agoThere wasn't a leak. You don't need to make it sound like we're in a cyberpunk RPG.
- nabakin 4y agowhat are you talking about? The LLaMA weights leaked from someone on 4chan.
- astrange 4y agoFacebook released them with essentially zero effort to stop anyone from resharing them. I think they let anyone with a .edu email get them. It wasn't hacked or anything.
- smoldesu 4y agoFacebook is still directly hosting the model, if they wanted to stop people from downloading it then it wouldn't be on their public CDN.
- piqi 4y ago> Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks. Is the purpose to know which of the models OpenAI offers is most suitable for your workload/app? Could I use this to know if the cheaper model is sufficient for a particular use-case?
- nprateem 4y agoSee, they are an open source company after all. They want you to help make their stuff better.