4 ms·
But I JUST switched from GPT4o to Claude! :( Kidding, but it isn't clear how to use this thing, as others have pointed out.
by rjurney 2y ago
But I JUST switched from GPT4o to Claude! :( Kidding, but it isn't clear how to use this thing, as others have pointed out.
- ukuina 2y agoWhat made you switch?
- pelagicAustral 2y agoI'm using both, been doing that for months now. I can confidently assert that while Claude is getting better and better, GPT 4 and 4o seem the be getting dumbed down for some unexplained reason. Claude is now my go-to for anything code. (I do Ruby and C#, btw, other might have a different experience)
- marcyb5st 2y agoI guess they are distilling the models so that they can save $$$ on serving.
- ldjkfkdsjnv 2y agoGPT4o is way behind sonnet 3.5
- mountainriver 2y agoHuh I guess all the benchmarks are wrong then
- rjurney 2y agoI think it depends on what your use case is and how optimized your prompts are.
- rjurney 2y agoClaude is much better. Overwhelmingly better. It not only implements deep learning models for me, it has great suggestions on evolving them to actually work.
- mountainriver 2y agolol no it’s not, the benchmarks don’t show that at all. Both have issues in different ways
- causal 2y agoBenchmarks are pretty flawed IMO, in particular their weakness here seems to be that they are poor at evaluating long-tail multiturn conversations. 4o often gives a great first response, then spirals into a repetition. Sonnet 3.5 is much better at seeing the big picture in a longer conversation IMO.
- orbital-decay 2y agoRepetition in multiturn conversations is actually Sonnet's fatal flaw, both 3 and 3.5. 4o is also repetitive to an extent. Opus is way better than both at being non-repetitive.
- stavros 2y agoI made a mobile app the other day using LLMs (I had never used React or TypeScript before, and I built an app with React Native). I was pretty disappointed, both Sonnet 3.5 and gpt-4-turbo performed pretty poorly, making mistakes like missing a closing bracket somewhere and meaning I had to revert, because I had no idea where they meant to put it. Also they did the thing that junior developers tend to do, where you have a race condition of some sort, and they just work around it by adding some if checks. The app is at around 400 lines right now, it works but feels pretty brittle. Adding a tiny feature here or there breaks something else, and GPT does the wrong thing half the time. All in all, I'm not complaining, because I made an app in two days, but it won't replace a developer yet, no matter how much I want it to.
- 2y ago
- throwup238 2y agoClaude Projects which allow attaching a bunch of files to fill up the 200k context. I wrote up a script to dump a bunch of code and documentation files to markdown as context and I add them to a bunch of Claude projects on a per topic basis. For example, I'm currently working on a Rust/Qt desktop app so I have a project with the whole Qt6 book attached to ask questions about Qt, a project with my SQL schema and ORM/Sqlite docs to ask questions about the app's data and generate models without dealing with hallucinations, a project with all my QML files and Rust QML element code, a project with a bunch of Rust crate docs, and so on and on. GPTs allow attaching files too but Claude Projects dump the entire contents of the files into the context rather than trying to do some hacky RAG that never works like I want it to.
- funnygiraffe 2y agoI was under the impression that with LLMs, in order to get high-quality answers, it's always best to keep context short. Is that not the case anymore? Does Claude under this usage paradigm not struggle with very long contexts in ways as for example described in the "lost in the middle" paper (https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172)?
- throwup238 2y agoI don't have the time to evaluate the effects of context length on my use cases so I have no idea. There might be some degradation when I attach the Qt book which is probably already in Claude's training data but when using it against my private code base, it's not like I have any other choice. The UX of drag and dropping a few monolithic markdown files to include entire chunks of a large project outweighs the downsides of including irrelevant context in my experience.
- inciampati 2y agoNo, you need to provide as much information in context as possible. Otherwise you are sampling from the mode. "Write me an essay about cows" = garbage boring and probably 200 words. "here are twenty papers about cow evolution, write me an overview of findings" = yes
- 2y ago