4 ms·

I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample out

I don't put a lot of stock on evals. many of the models claiming gpt-4 like benchmark scores feel a lot worse for any of my use-cases. Anyone got any sample output? Claude isn't available in EU yet, else i'd try it myself. :(

Alifatisk 3y ago
> Claude isn't available in EU yet, else i'd try it myself. I'm currently in EU and I have access to it?
- egeozcan 3y ago
  AFAIK there's no strict EU ban but no EU country is listed here: https://www.anthropic.com/claude-ai-locations https://www.anthropic.com/claude-ai-locations Perhaps you meant Europe the continent or using a VPN? edit: They seem to have updated that list after I posted my comment, the outdated list I based my comment on: https://web.archive.org/web/20240225034138/https://www.anthropic.com/claude-ai-locations https://web.archive.org/web/20240225034138/https://www.anthr... edit2: I was confused. There is another list for API regions, which has all EU countries. The frontend is still not updated.
  - addandsubtract 3y ago
    They updated the list of supported countries here: https://www.anthropic.com/supported-countries https://www.anthropic.com/supported-countries I was just able to sign up, while not being able to a few weeks ago.
    - Alifatisk 3y ago
      When I go to my account settings, it says my country is invalid haha
    - egeozcan 3y ago
      Oh well, it seems to have updated after my comment. Now it seems they support the whole EU and many more additional countries. But it still errors out when trying to sign up from Germany: https://i.imgur.com/rX0XA8d.jpeg https://i.imgur.com/rX0XA8d.jpeg https://i.imgur.com/Xlyqm8D.jpeg https://i.imgur.com/Xlyqm8D.jpeg
    - AlanYx 3y ago
      That's the list of countries supported by the API. For some reason, they support fewer countries through their frontend. I'm curious why that is.
  - Alifatisk 3y ago
    > AFAIK there's no strict EU ban but no EU country is listed here That's really weird, I just signed up with no issues and my country together with some other EU countries was listed. Now when I try to signup a new account, it says that my region is not supported. I still have the sms verification from them as proof.
lelag 3y ago
You can use Claude 2.1 on openrouter. Hopefully, they will be able to add the Claude 3 family too.
hackerlight 3y ago
One good sign is they're only a slight improvement on knowledge recall evals but a big improvement on code and reasoning evals. Hope this stands up to scrutiny and we get something better than GPT-4 for code generation. Although the best model is a lot more expensive.
- ethbr1 3y ago
  On the other hand, programmers are very expensive. At some level of accuracy and consistency (human order-of-magnitude?), the pricing of the service should start approaching the pricing of the human alternative. And first glance at numbers, LLMs are still way underpriced relative to humans.
  - hackerlight 3y ago
    The value/competency may approach that of a human but the price won't necessarily follow. Price will be determined by market forces. If compute is cheap and competition is fierce then the price can be near free even if it is at human-level intelligence. Then there will be a lot of surplus value created because buyers would be happy to pay $50/million tokens but only have to pay $0.1/million tokens thanks to competition. Frontier models will probably always be expensive though, because frontier by definition means you're sucking up all the available compute which will probably always be expensive.
  - SubiculumCode 3y ago
    NVidia's execs think so. It would be an ironic thing that it was open source that killed the programmer; as how would they train it otherwise? As a scientist, should I continue to support open access journals, just so I can be trained away? Slightly tongue in check, but not really.
    - ethbr1 3y ago
      I have a suspicion that greenfield science will be the last thing automated, at least the non-brute-force kind. AI assistants to do the drugery (smart search agents), but not pick the directions to proceed in. Too little relevant training data in niche, state of the art topics. But to the broader point, isn't this progress in a nutshell? (1) Figure out a thing can be done, (2) figure out how to manufacture with humans, (3) maximize productivity of human effort, (4) automate select portions of the optimized and standardized process, (5) find the last 5% isn't worth automating, because it's too branchy. From that perspective, software development isn't proceeding differently than any other field historically, with the added benefit that all its inputs and outputs are inherently digital.
avereveard 3y ago
I think aws has Claude in Frankfurt not the new one but instant and 2 should be there.
phillipcarter 3y ago
> I don't put a lot of stock on evals. Same, although they are helpful for setting expectations for me. I have some use cases (I'm hesitant to call them evals) related to how we use GPT for our product that are a good "real world" test case. I've found that Claude models are the only ones that are up to par with GPT in the past.
swalsh 3y ago
I've also seen the opposite, where tiny little 7B models get real close to GPT4 quality results on really specifically use cases. If you're trying to scale just that use case it's significantly cheaper, and also faster to just scale up inference with that specialty model. An example of this is using an LLM to extract medical details from a record.
deleted 3y ago
[deleted]
stolsvik 3y ago
There are two different "available in these regions" URLs. The one for chat: https://www.anthropic.com/claude-ai-locations https://www.anthropic.com/claude-ai-locations The one for API: https://www.anthropic.com/supported-countries https://www.anthropic.com/supported-countries The latter has Norway in it, while the former does not. One wonders why.

Alifatisk 3y ago

> Claude isn't available in EU yet, else i'd try it myself. I'm currently in EU and I have access to it?

egeozcan 3y ago
AFAIK there's no strict EU ban but no EU country is listed here: https://www.anthropic.com/claude-ai-locations https://www.anthropic.com/claude-ai-locations Perhaps you meant Europe the continent or using a VPN? edit: They seem to have updated that list after I posted my comment, the outdated list I based my comment on: https://web.archive.org/web/20240225034138/https://www.anthropic.com/claude-ai-locations https://web.archive.org/web/20240225034138/https://www.anthr... edit2: I was confused. There is another list for API regions, which has all EU countries. The frontend is still not updated.
- addandsubtract 3y ago
  They updated the list of supported countries here: https://www.anthropic.com/supported-countries https://www.anthropic.com/supported-countries I was just able to sign up, while not being able to a few weeks ago.
  - Alifatisk 3y ago
    When I go to my account settings, it says my country is invalid haha
  - egeozcan 3y ago
    Oh well, it seems to have updated after my comment. Now it seems they support the whole EU and many more additional countries. But it still errors out when trying to sign up from Germany: https://i.imgur.com/rX0XA8d.jpeg https://i.imgur.com/rX0XA8d.jpeg https://i.imgur.com/Xlyqm8D.jpeg https://i.imgur.com/Xlyqm8D.jpeg
  - AlanYx 3y ago
    That's the list of countries supported by the API. For some reason, they support fewer countries through their frontend. I'm curious why that is.
- Alifatisk 3y ago
  > AFAIK there's no strict EU ban but no EU country is listed here That's really weird, I just signed up with no issues and my country together with some other EU countries was listed. Now when I try to signup a new account, it says that my region is not supported. I still have the sms verification from them as proof.

lelag 3y ago

You can use Claude 2.1 on openrouter. Hopefully, they will be able to add the Claude 3 family too.

hackerlight 3y ago

One good sign is they're only a slight improvement on knowledge recall evals but a big improvement on code and reasoning evals. Hope this stands up to scrutiny and we get something better than GPT-4 for code generation. Although the best model is a lot more expensive.

ethbr1 3y ago
On the other hand, programmers are very expensive. At some level of accuracy and consistency (human order-of-magnitude?), the pricing of the service should start approaching the pricing of the human alternative. And first glance at numbers, LLMs are still way underpriced relative to humans.
- hackerlight 3y ago
  The value/competency may approach that of a human but the price won't necessarily follow. Price will be determined by market forces. If compute is cheap and competition is fierce then the price can be near free even if it is at human-level intelligence. Then there will be a lot of surplus value created because buyers would be happy to pay $50/million tokens but only have to pay $0.1/million tokens thanks to competition. Frontier models will probably always be expensive though, because frontier by definition means you're sucking up all the available compute which will probably always be expensive.
- SubiculumCode 3y ago
  NVidia's execs think so. It would be an ironic thing that it was open source that killed the programmer; as how would they train it otherwise? As a scientist, should I continue to support open access journals, just so I can be trained away? Slightly tongue in check, but not really.
  - ethbr1 3y ago
    I have a suspicion that greenfield science will be the last thing automated, at least the non-brute-force kind. AI assistants to do the drugery (smart search agents), but not pick the directions to proceed in. Too little relevant training data in niche, state of the art topics. But to the broader point, isn't this progress in a nutshell? (1) Figure out a thing can be done, (2) figure out how to manufacture with humans, (3) maximize productivity of human effort, (4) automate select portions of the optimized and standardized process, (5) find the last 5% isn't worth automating, because it's too branchy. From that perspective, software development isn't proceeding differently than any other field historically, with the added benefit that all its inputs and outputs are inherently digital.

avereveard 3y ago

I think aws has Claude in Frankfurt not the new one but instant and 2 should be there.

phillipcarter 3y ago

> I don't put a lot of stock on evals. Same, although they are helpful for setting expectations for me. I have some use cases (I'm hesitant to call them evals) related to how we use GPT for our product that are a good "real world" test case. I've found that Claude models are the only ones that are up to par with GPT in the past.

swalsh 3y ago

I've also seen the opposite, where tiny little 7B models get real close to GPT4 quality results on really specifically use cases. If you're trying to scale just that use case it's significantly cheaper, and also faster to just scale up inference with that specialty model. An example of this is using an LLM to extract medical details from a record.

deleted 3y ago

[deleted]

stolsvik 3y ago

There are two different "available in these regions" URLs. The one for chat: https://www.anthropic.com/claude-ai-locations https://www.anthropic.com/claude-ai-locations The one for API: https://www.anthropic.com/supported-countries https://www.anthropic.com/supported-countries The latter has Norway in it, while the former does not. One wonders why.