5 ms·

Gpt2-Chatbot Removed from Lmsys

39 points by synthwave 2y ago

deleted 2y ago
[deleted]
kevingadd 2y ago
I don't understand the connection between the submission title and the actual page it links to. Am I missing something obvious?
- Cheer2171 2y ago
  There was a big rumor that this Gpt2 chatbot was ChatGPT 4.5 or 5. That model is no longer on lmsys. See big discussion: https://news.ycombinator.com/item?id=40199715 https://news.ycombinator.com/item?id=40199715
- synthwave 2y ago
  Sorry, should have clarified. The part of the site that states the model is currently unavailable links to their evaluation policies which is the link used for this submission :)
deleted 2y ago
[deleted]
jerbear4328 2y ago
The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section: > Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing. > Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps: > 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users. > 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). > 4. Remove the model from Arena.
- synthwave 2y ago
  It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.
  - novia 2y ago
    They rate limited it at some point later in the day. The rate limiting applied to all the other OpenAI models also, but none of the non-OpenAI models. Just another data point of validation.
    - synthwave 2y ago
      Yeah it's also possible they temporarily took it down because of the flood of users that were slowing down the whole site for everyone. Potential for it to be back but the rate limits are really low, something like 2 queries per user per day
  - unraveller 2y ago
    Maybe they detected too many people were trying to extract training data from it. The purpose was clearly hype to squash any llama3 400B hype, I'm quite sure they don't need anonymous votes from users when they can afford other ways. Like someone else said in the thread, it could also be hype management to stop delicate people from being so freaked by novel capabilities on release.
- Jackson__ 2y ago
  > 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). So... make the users do the work for free, but never publicly publish the data? That seems kinda scummy to me. I was really motivated with the whole public ranking thing, but knowing I'm just doing work for another company with the data never being published sucks the fun out of it.

39 points by synthwave 2y ago

deleted 2y ago

[deleted]

I don't understand the connection between the submission title and the actual page it links to. Am I missing something obvious?

Cheer2171 2y ago
There was a big rumor that this Gpt2 chatbot was ChatGPT 4.5 or 5. That model is no longer on lmsys. See big discussion: https://news.ycombinator.com/item?id=40199715 https://news.ycombinator.com/item?id=40199715
synthwave 2y ago
Sorry, should have clarified. The part of the site that states the model is currently unavailable links to their evaluation policies which is the link used for this submission :)

deleted 2y ago

[deleted]

jerbear4328 2y ago

The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section: > Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing. > Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps: > 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users. > 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). > 4. Remove the model from Arena.

synthwave 2y ago
It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.
- novia 2y ago
  They rate limited it at some point later in the day. The rate limiting applied to all the other OpenAI models also, but none of the non-OpenAI models. Just another data point of validation.
  - synthwave 2y ago
    Yeah it's also possible they temporarily took it down because of the flood of users that were slowing down the whole site for everyone. Potential for it to be back but the rate limits are really low, something like 2 queries per user per day
- unraveller 2y ago
  Maybe they detected too many people were trying to extract training data from it. The purpose was clearly hype to squash any llama3 400B hype, I'm quite sure they don't need anonymous votes from users when they can afford other ways. Like someone else said in the thread, it could also be hype management to stop delicate people from being so freaked by novel capabilities on release.
Jackson__ 2y ago
> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). So... make the users do the work for free, but never publicly publish the data? That seems kinda scummy to me. I was really motivated with the whole public ranking thing, but knowing I'm just doing work for another company with the data never being published sucks the fun out of it.