4 ms·

Claude 3.5 Sonnet by Anthropic is the best model out there, if you are trying to have an extremely talented programmer paired to you. Somehow, OpenAI is playin

by tzury 2y ago

Claude 3.5 Sonnet by Anthropic is the best model out there, if you are trying to have an extremely talented programmer paired to you. Somehow, OpenAI is playing catch with them rather than vice versa.

jiggawatts 2y ago
I just can't get past the "You must have a valid phone number to use Anthropic’s services." Umm... why? Nobody else in the AI space wants to track my number. I'm sure Anthropic has their "reasons". I just doubt it is one that I would like.
- 4gotunameagain 2y ago
  I definitely had to give up a number when registering for chatGPT.
  - raverbashing 2y ago
    Same here. I can understand, they don't want their usage to go over the roof with fake accts
- BoredPositron 2y ago
  For API access I didn’t need to provide a phone number. I use it with a selfhosted lobechat instance without problems.
- maccard 2y ago
  I'm not affiliated with Claude, but assuming you're serious: > Umm... why? https://support.anthropic.com/en/articles/8287232-why-do-i-need-to-verify-my-phone-number https://support.anthropic.com/en/articles/8287232-why-do-i-n... My guess is, these models are incredibly expensive to run, Claude has a fairly generous free tier, and phone numbers are one of the easiest ways to significantly reduce the number of duplicate accounts. > Nobody else in the AI space wants to track my number. Given they're likely hoovering up all of the data you're sending to them, and they have your email address to identify you, this seems like an odd hill to die on.
- strogonoff 2y ago
  Advanced ML products are forbidden[0] to export to many places, so those who skimp on KYC are playing with fire. Paid products do not have this issue since you provide a billing address, but there is no good, free, and legal LLM that does not use a reliable way of verifying at least user’s location. Whether they are serious about it or use it as an excuse to collect more PII (or both/neither), collecting verified phone numbers presumably allows them to demonstrate compliance. [0] https://cset.georgetown.edu/article/dont-forget-the-catch-all-basics-ai-export-controls/ https://cset.georgetown.edu/article/dont-forget-the-catch-al...
  - diggan 2y ago
    > but there is no good, free, and legal LLM that does not use a reliable way of verifying at least user’s location. In the US, other locations may/may not have the same export controls. Base your AI business in one of the non-US countries and it'll be legal to not keep strict controls on who is using your service.
- Kjahd 2y ago
  [flagged]
- methyl 2y ago
  For one, to avoid massive number of bots using the API for free.
bradgessler 2y ago
How do you get it to work so well? I’ve tried it a few times now and it seems just as capable as gpt-4o.
- Zealotux 2y ago
  When I gave the same prompt to both, Sonnet 3.5 immediately gave me functional code, while GPT-4o sometimes failed after 4-5 attempts, at which point I usually gave up. Sonnet 3.5 is spectacular at debugging its output, while GPT-4o will keep hallucinating and giving me the same buggy code. A concrete example: I was doing shader programming with Sonnet 3.5 and ran into a visual bug. Sonnet asked me to add four debugging modes, cycle through each one, and describe what I saw for each one. With one more prompt, it resolved the issue. In my experience, GPT-4o has never bothered proposing debug modes and just produced more buggy code. For non-trivial coding, Sonnet 3.5 was miles above anything else, and I didn't even have to try hard.
  - FiberBundle 2y ago
    Why can't you just debug this yourself? I don't think completely relying on LLMs for something like this will do you any good in the long run.
    - creesch 2y ago
      That's a different question though. The person you replied to was asked to explain why they think Sonnet 3.5 works well/better compared to GPT-4o. To which they gave a good answer of Sonnet actually taking context and new information better into account when following up. They might be able to debug it themselves, maybe they should be able to debug it themselves. But I feel like that is a completely different conversation.
    - danielbln 2y ago
      Why not? It depends on how you use these systems. Let the LLM debug this for me, give me a nice explanation for what's happening and what solution paths could be and then it's on me to evaluate and make the right decision there. Don't rely blindly on these systems, in the same vein as you shouldn't rely blindly on some solution found while using Google.
threeseed 2y ago
> if you are trying to have an extremely talented programmer paired to you I've found it to be on par with Stack Overflow / Google Search. More convenient than cut/paste but more prone to inaccuracies and out of context answers. But at no point did it remotely feel like a top tier programmer.
- nicce 2y ago
  When we go from junior stuff to senior stuff, there is way too much hallucination, at least in Rust. I went back to forums after mainly using AI models for one year. These models are good at generating template code and many straightforward things, but if you add anything complex, you start wasting your time.
stavros 2y ago
I'd replace "extremely talented programmer" with "knowledgeable junior", in my experience. It's much better than GPT-4o, but still not great.
- maxlamb 2y ago
  GPT-4 is way more powerful than GPT-4o for programming tasks.
  - stavros 2y ago
    That's true, but they both made more mistakes than Sonnet for me. I use them with Aider.
    - nicce 2y ago
      Sonnet sometimes repeat its’ previous response too often, when you ask for changes. It claims there were changes, but there aren’t, because the output was already the best that model can produce. This behaviour seems to be deeply added somewhere as it is hard to change.
- daghamm 2y ago
  I use both side by side. It really depends on the language and the prompt. Sometimes one shines and the other produces garbage and it's usually 50/50
croes 2y ago
Isn't that dependent on the programming language?
RicoElectrico 2y ago
Claude is better by the virtue of the ridiculously large context window. You can literally drop a whole directory of source code spaghetti and it will make sense of it.

by tzury 2y ago

jiggawatts 2y ago

I just can't get past the "You must have a valid phone number to use Anthropic’s services." Umm... why? Nobody else in the AI space wants to track my number. I'm sure Anthropic has their "reasons". I just doubt it is one that I would like.

4gotunameagain 2y ago
I definitely had to give up a number when registering for chatGPT.
- raverbashing 2y ago
  Same here. I can understand, they don't want their usage to go over the roof with fake accts
BoredPositron 2y ago
For API access I didn’t need to provide a phone number. I use it with a selfhosted lobechat instance without problems.
maccard 2y ago
I'm not affiliated with Claude, but assuming you're serious: > Umm... why? https://support.anthropic.com/en/articles/8287232-why-do-i-need-to-verify-my-phone-number https://support.anthropic.com/en/articles/8287232-why-do-i-n... My guess is, these models are incredibly expensive to run, Claude has a fairly generous free tier, and phone numbers are one of the easiest ways to significantly reduce the number of duplicate accounts. > Nobody else in the AI space wants to track my number. Given they're likely hoovering up all of the data you're sending to them, and they have your email address to identify you, this seems like an odd hill to die on.
strogonoff 2y ago
Advanced ML products are forbidden[0] to export to many places, so those who skimp on KYC are playing with fire. Paid products do not have this issue since you provide a billing address, but there is no good, free, and legal LLM that does not use a reliable way of verifying at least user’s location. Whether they are serious about it or use it as an excuse to collect more PII (or both/neither), collecting verified phone numbers presumably allows them to demonstrate compliance. [0] https://cset.georgetown.edu/article/dont-forget-the-catch-all-basics-ai-export-controls/ https://cset.georgetown.edu/article/dont-forget-the-catch-al...
- diggan 2y ago
  > but there is no good, free, and legal LLM that does not use a reliable way of verifying at least user’s location. In the US, other locations may/may not have the same export controls. Base your AI business in one of the non-US countries and it'll be legal to not keep strict controls on who is using your service.
Kjahd 2y ago
[flagged]
methyl 2y ago
For one, to avoid massive number of bots using the API for free.

bradgessler 2y ago

How do you get it to work so well? I’ve tried it a few times now and it seems just as capable as gpt-4o.

Zealotux 2y ago
When I gave the same prompt to both, Sonnet 3.5 immediately gave me functional code, while GPT-4o sometimes failed after 4-5 attempts, at which point I usually gave up. Sonnet 3.5 is spectacular at debugging its output, while GPT-4o will keep hallucinating and giving me the same buggy code. A concrete example: I was doing shader programming with Sonnet 3.5 and ran into a visual bug. Sonnet asked me to add four debugging modes, cycle through each one, and describe what I saw for each one. With one more prompt, it resolved the issue. In my experience, GPT-4o has never bothered proposing debug modes and just produced more buggy code. For non-trivial coding, Sonnet 3.5 was miles above anything else, and I didn't even have to try hard.
- FiberBundle 2y ago
  Why can't you just debug this yourself? I don't think completely relying on LLMs for something like this will do you any good in the long run.
  - creesch 2y ago
    That's a different question though. The person you replied to was asked to explain why they think Sonnet 3.5 works well/better compared to GPT-4o. To which they gave a good answer of Sonnet actually taking context and new information better into account when following up. They might be able to debug it themselves, maybe they should be able to debug it themselves. But I feel like that is a completely different conversation.
  - danielbln 2y ago
    Why not? It depends on how you use these systems. Let the LLM debug this for me, give me a nice explanation for what's happening and what solution paths could be and then it's on me to evaluate and make the right decision there. Don't rely blindly on these systems, in the same vein as you shouldn't rely blindly on some solution found while using Google.

threeseed 2y ago

> if you are trying to have an extremely talented programmer paired to you I've found it to be on par with Stack Overflow / Google Search. More convenient than cut/paste but more prone to inaccuracies and out of context answers. But at no point did it remotely feel like a top tier programmer.

nicce 2y ago
When we go from junior stuff to senior stuff, there is way too much hallucination, at least in Rust. I went back to forums after mainly using AI models for one year. These models are good at generating template code and many straightforward things, but if you add anything complex, you start wasting your time.

stavros 2y ago

I'd replace "extremely talented programmer" with "knowledgeable junior", in my experience. It's much better than GPT-4o, but still not great.

maxlamb 2y ago
GPT-4 is way more powerful than GPT-4o for programming tasks.
- stavros 2y ago
  That's true, but they both made more mistakes than Sonnet for me. I use them with Aider.
  - nicce 2y ago
    Sonnet sometimes repeat its’ previous response too often, when you ask for changes. It claims there were changes, but there aren’t, because the output was already the best that model can produce. This behaviour seems to be deeply added somewhere as it is hard to change.
daghamm 2y ago
I use both side by side. It really depends on the language and the prompt. Sometimes one shines and the other produces garbage and it's usually 50/50

croes 2y ago

Isn't that dependent on the programming language?

RicoElectrico 2y ago

Claude is better by the virtue of the ridiculously large context window. You can literally drop a whole directory of source code spaghetti and it will make sense of it.