9 ms·

I consistently get significantly better performance from Anthropic at a literal order of magnitude less cost. I am incredibly doubtful that this new GPT is 10x

I consistently get significantly better performance from Anthropic at a literal order of magnitude less cost. I am incredibly doubtful that this new GPT is 10x Claude unless it is embracing some breakthrough, secret, architecture nobody has heard of.

nurettin 2y ago
Or Anthropic will follow suit.
- MuffinFlavored 2y ago
  Am I wrong that Anthropic doesn't really have a match yet to ChatGPT's o1 model (a "reasoning" model?)
  - apsec112 2y ago
    They don't have a model that does o1-style "thought tokens" or is specialized for math, but Sonnet 3.6 is really strong in other ways. I'm guessing they will have an o1-style model within six months if there's demand
  - tokioyoyo 2y ago
    To my understanding, Anthropic realizes that they can’t compete in name recognition yet, so they have to overdeliver in terms of quality to win the war. It’s hard to beat the incumbent, especially when “chatgpt’ing” is basically a well understood verb.
  - jerjerjer 2y ago
    Is a "reasoning" model really different? Or is it just clever prompting (and feeding previous outputs) for an existing model? Possibly with some RLHF reasoning examples? OpenAI doesn't have a large enough database of reasoning texts to train a foundational LLM off it? I thought such a db simply does not exist as humans don't really write enough texts like this.
    - griomnib 2y ago
      It’s clever marketing.
    - logicchains 2y ago
      It's trained via reinforcement learning on essentially infinite synthetic reasoning data. You can generate infinite reasoning data because there are infinite math and coding problems that can be created with machine-checkable solutions, and machines can make infinite different attempts at reasoning their way to the answer. Similar to how models trained to learn chess by self-play have essentially unlimited training data.
    - int_19h 2y ago
      We don't know the specifics of GPT-o1 to judge, but we can look at open weights model for an example. Qwen-32B is a base model, QwQ-32B is a "reasoning" variant. You're broadly correct that the magic, such as it is, is in training the model into a long-winded CoT, but the improvements from it are massive. QwQ-32B beats larger 70B models in most tasks, and in some cases it beats Claude.
      - emporas 2y ago
        I just tried QwQ 32B, i didn't know about it. I used it to generate, some code GPT generated 2 days ago perfect code without even sweating. QwQ generated 10 pages of it's reasoning steps, and the code is probably not correct. [1] includes both answers from QwQ and GPT. Breaking down it's reasoning steps to such an excruciating detailed prose is certainly not user friendly, but it is intriguing. I wonder what an ideal use case for it would be. [1] https://gist.github.com/defmarco/9eb4b1d0c547936bafe39623ec6595ca https://gist.github.com/defmarco/9eb4b1d0c547936bafe39623ec6...
  - airstrike 2y ago
    Claude Sonnet 3.5 has outperformed o1 in most tasks based on my own anecdotal assessment. So much so that I'm debating canceling my ChatGPT subscription. I just literally do not use it anymore, despite being a heavy user for a long time in the past
aliasxneo 2y ago
I haven't used ChatGPT in a few weeks now. I still maintain subscriptions to both ChatGPT and Claude, but I'm very close to dropping ChatGPT entirely. The only useful thing it provides over Claude is a decent mobile voice mode and web search.
- HanClinto 2y ago
  I'm in the same boat — I maintain subscriptions to both. The main thing I like OpenAI for is that when I'm on a long drive, I like to have conversations with OpenAI's voice mode. If Claude had a voice mode, I could see dropping OpenAI entirely, but for now it feels like the subscriptions to both is a near-negligible cost relative to the benefits I get from staying near the front of the AI wave.
- bluedays 2y ago
  I’ve been considering dropping ChatGPT for the same reason. Now that the app is out the only thing I actually care about is search.
- asterix_pano 2y ago
  If you don't want to necessarily have to pick between one or the other, there are services like this one that let you basically access all the major LLMs and only pay per use: https://nano-gpt.com/ https://nano-gpt.com/
  - sumedh 2y ago
    Does it have Claude's artifact feature
  - pc86 2y ago
    I've used TypingMind and it's pretty great, I like the idea of just plugging in a couple API keys and paying a fraction, but I really wish there was some overlap. If a random query via the API costs a fifth of a cent why can't I can't 10 free API calls w/ my $20/mo premium subscription?
MP_1729 2y ago
That's not how pricing works. If o1-pro is 10% better than Claude, but you are a guy who makes $300,000 per year, but now can make $330,000 because o1-pro makes you more productive, then it makes sense to give Sam $2,400.
- 015a 2y ago
  The math is never this clean, and no one has ever experienced this (though I'm sure its a justification that was floated at OAI HQ at least once).
  - xur17 2y ago
    It's never this clean, but it is direction-ally correct. If I make $300k / year, and I can tell that chatgpt already saves me hours or even days per month, $200 is a laughable amount. If I feel like pro is even slightly better, it's worth $200 just to know that I always have the best option available. Heck, it's probably worth $200 even if I'm not confident it's better just in case it is. For the same reason I don't start with the cheapest AI model when asking questions and then switch to the more expensive if it doesn't work. The more expensive one is cheap enough that it doesn't even matter, and $200 is cheap enough (for a certain subsection of users) that they'll just pay it to be sure they're using the best option.
    - 015a 2y ago
      That's only true if your time is metered by the hour; and the vast majority of roles which find some benefit from AI, at this time, are not compensated hourly. This plan might be beneficial to e.g. CEO-types, but I question who at OpenAI thought it would be a good idea to lead their 12 days of hollowhype with this launch, then; unless this is the highest impact release they've got (one hopes it is not).
      - cmeacham98 2y ago
        Employers might be willing to get their employees a subscription if they believe it makes their employees they are paying $$$$$ more X% productive. (Where X% of their salary works out to more than $2400/year)
vessenes 2y ago
I think of them as different people -- I'll say that I use them in "ensemble mode" for coding, the workflow is Claude 3.5 by default -- when Claude is spinning, o1-preview to discuss, Claude to implement. Worst case o1-preview to implement, although I think its natural coding style is slightly better than Claude's. The speed difference isn't worth it. The intersection of problems I have where both have trouble is pretty small. If this closes the gap even more, that's great. That said, I'm curious to try this out -- the ways in which o1-preview fails are a bit different than prior gpt-line LLMs, and I'm curious how it will feel on the ground.
- vessenes 2y ago
  Okay, tried it out. Early indications - it feels a bit more concise, thank god, certainly more concise than 4o -- it's s l o w. Getting over 1m times to parse codebases. There's some sort of caching going on though, follow up queries are a bit faster (30-50s). I note that this is still superhuman speeds, but it's not writing at the speed Groqchat can output Llama 3.1 8b, that is for sure. Code looks really clean. I'm not instantly canceling my subscription.
  - pc86 2y ago
    When you say "parse codebases" is this uploading a couple thousand lines in a few different files? Or pasting in 75 lines into the chat box? Or something else?
    - vessenes 2y ago
      $ find web -type f $ -name '.go' -o -name '.tsx' $ | tar -cf code.tar -T; cat code.tar | pbcopy Then I paste it in and say "can you spot any bugs in the API usage? Write out a list of tasks for a senior engineer to get the codebase in basically perfect shape," or something along those lines. Alternately: "write a go module to support X feature, and implement the react typescript UI side as well. Use the existing styles in the tsx files you find; follow these coding guidelines, etc. etc."
xixixao 2y ago
Which ChatGPT model have you been using? In my experience nothing beats 4. (Not claude, not 4o)
superfrank 2y ago
I've heard this a lot and so I switched to Claude for a month and was super disappointed. What are you mainly using ChatGPT for? Personally, I found Claude marginally better for coding, but far, far worse for just general purpose questions (e.g. I'm a new home owner and I need to winterize my house before our weather drops below freezing. What are some steps I should take or things I should look into?)
- BoorishBears 2y ago
  It's ironic because I never want to ask an LLM for something like your example general purpose question, where I can't just cheaply and directly test the correctness of the answer But we're hurtling towards all the internet's answers to general purpose questions being SEO spam that was generated by an LLM anyways. Since OpenAI probably isn't hiring as many HVAC technicians to answer queries as they are programmers, it feels like we're headed towards a death spiral where either having the LLM do actual research from non-SEO affected primary sources, or finding a human who's done that research will be the only options for generic knowledge questions that are off the beaten path - Actually to test my hypothesis I just tried this with ChatGPT with internet access. The list of winterization tips cited an article that felt pretty "delvey". I search the author's name and their LinkedIn profile is about how they professionally write marketing content (nothing about HVAC), one of their accomplishments is Generative AI, and their like feed is full of AI mentions for writing content. So ChatGPT is already at a place where when it searches for "citations", it's just spitting back out its own uncited answers above answers by actual experts (since the expert sources aren't as SEO-driven)
  - superfrank 2y ago
    > I can't just cheaply and directly test the correctness of the answer I feel that, but I think for me the key is knowing that LLMs can be wrong and I should treat the answer as a starting point and not an actual expert. I find it really helpful for topics where I don't even know where to start because, like you said, most search engines are utter trash now. For things like that, I find ChatGPT to be a good diving off point. For example, this is what I got when I asked: ``` Preparing your townhouse for winter involves addressing common issues associated with the region's wet and cool climate. Here's a concise checklist to help you get started: 1. Exterior Maintenance Roof Inspection: Check for damaged or missing shingles to prevent leaks during heavy rains. Gutter Cleaning: Remove leaves and debris to ensure proper drainage and prevent water damage. Downspouts: Ensure they direct water away from the foundation to prevent pooling and potential leaks. Siding and Trim: Inspect for cracks or gaps and seal them to prevent moisture intrusion. 2. Windows and Doors Weatherstripping: Install or replace to seal gaps and prevent drafts, improving energy efficiency. Caulking: Apply around window and door frames to block moisture and cold air. 3. Heating System Furnace Inspection: Have a professional service your furnace to ensure it's operating efficiently. Filter Replacement: Change furnace filters to maintain good air quality and system performance. 4. Plumbing Outdoor Faucets: Disconnect hoses and insulate faucets to prevent freezing. Pipe Insulation: Insulate exposed pipes, especially in unheated areas, to prevent freezing and bursting. 5. Landscaping Tree Trimming: Prune branches that could break under snow or ice and damage your property. Drainage: Ensure the yard slopes away from the foundation to prevent water accumulation. 6. Safety Checks Smoke and Carbon Monoxide Detectors: Test and replace batteries to ensure functionality. Fireplace and Chimney: If applicable, have them inspected and cleaned to prevent fire hazards. By addressing these areas, you can help protect your home from common winter-related issues in Seattle's climate. ``` Once I dove into the links ChatGPT provided I found the detail I needed and things I needed to investigate more, but it saved 30 minutes of pulling together a starting list from the top 5-10 articles on Google.
VeejayRampay 2y ago
Claude is so much better
cryptoegorophy 2y ago
I've heard so much about Claude and decided to give it a try and it has been rather a major disappointment. I ended up using chatgpt as an assistant for claude's code writing because it just couldn't get things right. Had to cancel my subscription, no idea why people still promote it everywhere like it is 100x times better than chatgpt.
- sumedh 2y ago
  > Had to cancel my subscription, no idea why people still promote it everywhere like it is 100x times better than chatgpt. You need to learn how to ask it the right questions.
moralestapia 2y ago
I mean ... anecdata for anecdata. I use LLMs for many projects and 4o is the sweet spot for me. >literal order of magnitude less cost This is just not true. If your use case can be solved with 4o-mini (I know, not all do) OpenAI is the one which is an order of magnitude cheaper.
404mm 2y ago
I pay for both GPT and Claude and use them both extensively. Claude is my go-to for technical questions, GPT (4o) for simple questions, internet searches and validation of Claude answers. GPT o1-preview is great for more complex solutions and work on larger projects with multiple steps leading to finish. There’s really nothing like it that Anthropic provides. But $200/mo is way above what I’m willing to pay.
- griomnib 2y ago
  I have several local models I hit up first (Mixtral, Llama), if I don’t like the results then I’ll give same prompt to Claude and GPT. Overall though it’s really just for reference and/or telling me about some standard library function I didn’t know of. Somewhat counterintuitively I spend way more time reading language documentation than I used to, as the LLM is mainly useful in pointing me to language features. After a few very bad experiences I never let LLM write more than a couple lines of boilerplate for me, but as a well-read assistant they are useful. But none of them are sufficient alone, you do need a “team” of them - which is why I also don’t see the value is spending this much on one model. I’d spend that much on a system that polled 5 models concurrently and came up with a summary of sorts.
  - ifwinterco 2y ago
    People keep talking about using LLMs for writing code, and they might be useful for that, but I've found them much more useful for explaining human-written code than anything else, especially in languages/frameworks outside my core competency. E.g. "why does this (random code in a framework I haven't used much) code cause this error?" About 50% of the time I get a helpful response straight away that saves me trawling through Stack Overflow and random blog posts. About 25% of the time the response is at least partially wrong, but it still helps me get on the right track. 25% of the time the LLM has no idea and won't admit it so I end up wasting a small amount of time going round in circles, but overall it's a significant productivity boost when I'm working on unfamiliar code.
  - 404mm 2y ago
    What model sizes do you run locally? Anything that would work on a 16GB M1?
bhouston 2y ago
Yeah, I've switched to Anthropic fully as well for personal usage. It seems better to me and/or equivalent in all use cases.
- replwoacause 2y ago
  Same. Honestly if they released a $200 a month plan I’d probably bite, but OpenAI hasn’t earned that level of confidence from me yet. They have some catching up to do.
acchow 2y ago
I find o1 much better for having discussions or solving problems, then usually switch to Claude for code generation.
rmbyrro 2y ago
Sonnet isn't good at debugging, or even architecting. o1 shines, it feels like magic. The kinds of bugs it helped me nail were incredible to me.

nurettin 2y ago

Or Anthropic will follow suit.

MuffinFlavored 2y ago
Am I wrong that Anthropic doesn't really have a match yet to ChatGPT's o1 model (a "reasoning" model?)
- apsec112 2y ago
  They don't have a model that does o1-style "thought tokens" or is specialized for math, but Sonnet 3.6 is really strong in other ways. I'm guessing they will have an o1-style model within six months if there's demand
- tokioyoyo 2y ago
  To my understanding, Anthropic realizes that they can’t compete in name recognition yet, so they have to overdeliver in terms of quality to win the war. It’s hard to beat the incumbent, especially when “chatgpt’ing” is basically a well understood verb.
- jerjerjer 2y ago
  Is a "reasoning" model really different? Or is it just clever prompting (and feeding previous outputs) for an existing model? Possibly with some RLHF reasoning examples? OpenAI doesn't have a large enough database of reasoning texts to train a foundational LLM off it? I thought such a db simply does not exist as humans don't really write enough texts like this.
  - griomnib 2y ago
    It’s clever marketing.
  - logicchains 2y ago
    It's trained via reinforcement learning on essentially infinite synthetic reasoning data. You can generate infinite reasoning data because there are infinite math and coding problems that can be created with machine-checkable solutions, and machines can make infinite different attempts at reasoning their way to the answer. Similar to how models trained to learn chess by self-play have essentially unlimited training data.
  - int_19h 2y ago
    We don't know the specifics of GPT-o1 to judge, but we can look at open weights model for an example. Qwen-32B is a base model, QwQ-32B is a "reasoning" variant. You're broadly correct that the magic, such as it is, is in training the model into a long-winded CoT, but the improvements from it are massive. QwQ-32B beats larger 70B models in most tasks, and in some cases it beats Claude.
    - emporas 2y ago
      I just tried QwQ 32B, i didn't know about it. I used it to generate, some code GPT generated 2 days ago perfect code without even sweating. QwQ generated 10 pages of it's reasoning steps, and the code is probably not correct. [1] includes both answers from QwQ and GPT. Breaking down it's reasoning steps to such an excruciating detailed prose is certainly not user friendly, but it is intriguing. I wonder what an ideal use case for it would be. [1] https://gist.github.com/defmarco/9eb4b1d0c547936bafe39623ec6595ca https://gist.github.com/defmarco/9eb4b1d0c547936bafe39623ec6...
- airstrike 2y ago
  Claude Sonnet 3.5 has outperformed o1 in most tasks based on my own anecdotal assessment. So much so that I'm debating canceling my ChatGPT subscription. I just literally do not use it anymore, despite being a heavy user for a long time in the past

aliasxneo 2y ago

I haven't used ChatGPT in a few weeks now. I still maintain subscriptions to both ChatGPT and Claude, but I'm very close to dropping ChatGPT entirely. The only useful thing it provides over Claude is a decent mobile voice mode and web search.

HanClinto 2y ago
I'm in the same boat — I maintain subscriptions to both. The main thing I like OpenAI for is that when I'm on a long drive, I like to have conversations with OpenAI's voice mode. If Claude had a voice mode, I could see dropping OpenAI entirely, but for now it feels like the subscriptions to both is a near-negligible cost relative to the benefits I get from staying near the front of the AI wave.
bluedays 2y ago
I’ve been considering dropping ChatGPT for the same reason. Now that the app is out the only thing I actually care about is search.
asterix_pano 2y ago
If you don't want to necessarily have to pick between one or the other, there are services like this one that let you basically access all the major LLMs and only pay per use: https://nano-gpt.com/ https://nano-gpt.com/
- sumedh 2y ago
  Does it have Claude's artifact feature
- pc86 2y ago
  I've used TypingMind and it's pretty great, I like the idea of just plugging in a couple API keys and paying a fraction, but I really wish there was some overlap. If a random query via the API costs a fifth of a cent why can't I can't 10 free API calls w/ my $20/mo premium subscription?

MP_1729 2y ago

That's not how pricing works. If o1-pro is 10% better than Claude, but you are a guy who makes $300,000 per year, but now can make $330,000 because o1-pro makes you more productive, then it makes sense to give Sam $2,400.

015a 2y ago
The math is never this clean, and no one has ever experienced this (though I'm sure its a justification that was floated at OAI HQ at least once).
- xur17 2y ago
  It's never this clean, but it is direction-ally correct. If I make $300k / year, and I can tell that chatgpt already saves me hours or even days per month, $200 is a laughable amount. If I feel like pro is even slightly better, it's worth $200 just to know that I always have the best option available. Heck, it's probably worth $200 even if I'm not confident it's better just in case it is. For the same reason I don't start with the cheapest AI model when asking questions and then switch to the more expensive if it doesn't work. The more expensive one is cheap enough that it doesn't even matter, and $200 is cheap enough (for a certain subsection of users) that they'll just pay it to be sure they're using the best option.
  - 015a 2y ago
    That's only true if your time is metered by the hour; and the vast majority of roles which find some benefit from AI, at this time, are not compensated hourly. This plan might be beneficial to e.g. CEO-types, but I question who at OpenAI thought it would be a good idea to lead their 12 days of hollowhype with this launch, then; unless this is the highest impact release they've got (one hopes it is not).
    - cmeacham98 2y ago
      Employers might be willing to get their employees a subscription if they believe it makes their employees they are paying $$$$$ more X% productive. (Where X% of their salary works out to more than $2400/year)

vessenes 2y ago

I think of them as different people -- I'll say that I use them in "ensemble mode" for coding, the workflow is Claude 3.5 by default -- when Claude is spinning, o1-preview to discuss, Claude to implement. Worst case o1-preview to implement, although I think its natural coding style is slightly better than Claude's. The speed difference isn't worth it. The intersection of problems I have where both have trouble is pretty small. If this closes the gap even more, that's great. That said, I'm curious to try this out -- the ways in which o1-preview fails are a bit different than prior gpt-line LLMs, and I'm curious how it will feel on the ground.

vessenes 2y ago
Okay, tried it out. Early indications - it feels a bit more concise, thank god, certainly more concise than 4o -- it's s l o w. Getting over 1m times to parse codebases. There's some sort of caching going on though, follow up queries are a bit faster (30-50s). I note that this is still superhuman speeds, but it's not writing at the speed Groqchat can output Llama 3.1 8b, that is for sure. Code looks really clean. I'm not instantly canceling my subscription.
- pc86 2y ago
  When you say "parse codebases" is this uploading a couple thousand lines in a few different files? Or pasting in 75 lines into the chat box? Or something else?
  - vessenes 2y ago
    $ find web -type f $ -name '.go' -o -name '.tsx' $ | tar -cf code.tar -T; cat code.tar | pbcopy Then I paste it in and say "can you spot any bugs in the API usage? Write out a list of tasks for a senior engineer to get the codebase in basically perfect shape," or something along those lines. Alternately: "write a go module to support X feature, and implement the react typescript UI side as well. Use the existing styles in the tsx files you find; follow these coding guidelines, etc. etc."

xixixao 2y ago

Which ChatGPT model have you been using? In my experience nothing beats 4. (Not claude, not 4o)

superfrank 2y ago

I've heard this a lot and so I switched to Claude for a month and was super disappointed. What are you mainly using ChatGPT for? Personally, I found Claude marginally better for coding, but far, far worse for just general purpose questions (e.g. I'm a new home owner and I need to winterize my house before our weather drops below freezing. What are some steps I should take or things I should look into?)

BoorishBears 2y ago
It's ironic because I never want to ask an LLM for something like your example general purpose question, where I can't just cheaply and directly test the correctness of the answer But we're hurtling towards all the internet's answers to general purpose questions being SEO spam that was generated by an LLM anyways. Since OpenAI probably isn't hiring as many HVAC technicians to answer queries as they are programmers, it feels like we're headed towards a death spiral where either having the LLM do actual research from non-SEO affected primary sources, or finding a human who's done that research will be the only options for generic knowledge questions that are off the beaten path - Actually to test my hypothesis I just tried this with ChatGPT with internet access. The list of winterization tips cited an article that felt pretty "delvey". I search the author's name and their LinkedIn profile is about how they professionally write marketing content (nothing about HVAC), one of their accomplishments is Generative AI, and their like feed is full of AI mentions for writing content. So ChatGPT is already at a place where when it searches for "citations", it's just spitting back out its own uncited answers above answers by actual experts (since the expert sources aren't as SEO-driven)
- superfrank 2y ago
  > I can't just cheaply and directly test the correctness of the answer I feel that, but I think for me the key is knowing that LLMs can be wrong and I should treat the answer as a starting point and not an actual expert. I find it really helpful for topics where I don't even know where to start because, like you said, most search engines are utter trash now. For things like that, I find ChatGPT to be a good diving off point. For example, this is what I got when I asked: ``` Preparing your townhouse for winter involves addressing common issues associated with the region's wet and cool climate. Here's a concise checklist to help you get started: 1. Exterior Maintenance Roof Inspection: Check for damaged or missing shingles to prevent leaks during heavy rains. Gutter Cleaning: Remove leaves and debris to ensure proper drainage and prevent water damage. Downspouts: Ensure they direct water away from the foundation to prevent pooling and potential leaks. Siding and Trim: Inspect for cracks or gaps and seal them to prevent moisture intrusion. 2. Windows and Doors Weatherstripping: Install or replace to seal gaps and prevent drafts, improving energy efficiency. Caulking: Apply around window and door frames to block moisture and cold air. 3. Heating System Furnace Inspection: Have a professional service your furnace to ensure it's operating efficiently. Filter Replacement: Change furnace filters to maintain good air quality and system performance. 4. Plumbing Outdoor Faucets: Disconnect hoses and insulate faucets to prevent freezing. Pipe Insulation: Insulate exposed pipes, especially in unheated areas, to prevent freezing and bursting. 5. Landscaping Tree Trimming: Prune branches that could break under snow or ice and damage your property. Drainage: Ensure the yard slopes away from the foundation to prevent water accumulation. 6. Safety Checks Smoke and Carbon Monoxide Detectors: Test and replace batteries to ensure functionality. Fireplace and Chimney: If applicable, have them inspected and cleaned to prevent fire hazards. By addressing these areas, you can help protect your home from common winter-related issues in Seattle's climate. ``` Once I dove into the links ChatGPT provided I found the detail I needed and things I needed to investigate more, but it saved 30 minutes of pulling together a starting list from the top 5-10 articles on Google.

VeejayRampay 2y ago

Claude is so much better

cryptoegorophy 2y ago

I've heard so much about Claude and decided to give it a try and it has been rather a major disappointment. I ended up using chatgpt as an assistant for claude's code writing because it just couldn't get things right. Had to cancel my subscription, no idea why people still promote it everywhere like it is 100x times better than chatgpt.

sumedh 2y ago
> Had to cancel my subscription, no idea why people still promote it everywhere like it is 100x times better than chatgpt. You need to learn how to ask it the right questions.

moralestapia 2y ago

I mean ... anecdata for anecdata. I use LLMs for many projects and 4o is the sweet spot for me. >literal order of magnitude less cost This is just not true. If your use case can be solved with 4o-mini (I know, not all do) OpenAI is the one which is an order of magnitude cheaper.

404mm 2y ago

I pay for both GPT and Claude and use them both extensively. Claude is my go-to for technical questions, GPT (4o) for simple questions, internet searches and validation of Claude answers. GPT o1-preview is great for more complex solutions and work on larger projects with multiple steps leading to finish. There’s really nothing like it that Anthropic provides. But $200/mo is way above what I’m willing to pay.

griomnib 2y ago
I have several local models I hit up first (Mixtral, Llama), if I don’t like the results then I’ll give same prompt to Claude and GPT. Overall though it’s really just for reference and/or telling me about some standard library function I didn’t know of. Somewhat counterintuitively I spend way more time reading language documentation than I used to, as the LLM is mainly useful in pointing me to language features. After a few very bad experiences I never let LLM write more than a couple lines of boilerplate for me, but as a well-read assistant they are useful. But none of them are sufficient alone, you do need a “team” of them - which is why I also don’t see the value is spending this much on one model. I’d spend that much on a system that polled 5 models concurrently and came up with a summary of sorts.
- ifwinterco 2y ago
  People keep talking about using LLMs for writing code, and they might be useful for that, but I've found them much more useful for explaining human-written code than anything else, especially in languages/frameworks outside my core competency. E.g. "why does this (random code in a framework I haven't used much) code cause this error?" About 50% of the time I get a helpful response straight away that saves me trawling through Stack Overflow and random blog posts. About 25% of the time the response is at least partially wrong, but it still helps me get on the right track. 25% of the time the LLM has no idea and won't admit it so I end up wasting a small amount of time going round in circles, but overall it's a significant productivity boost when I'm working on unfamiliar code.
- 404mm 2y ago
  What model sizes do you run locally? Anything that would work on a 16GB M1?

bhouston 2y ago

Yeah, I've switched to Anthropic fully as well for personal usage. It seems better to me and/or equivalent in all use cases.

replwoacause 2y ago
Same. Honestly if they released a $200 a month plan I’d probably bite, but OpenAI hasn’t earned that level of confidence from me yet. They have some catching up to do.

acchow 2y ago

I find o1 much better for having discussions or solving problems, then usually switch to Claude for code generation.

rmbyrro 2y ago

Sonnet isn't good at debugging, or even architecting. o1 shines, it feels like magic. The kinds of bugs it helped me nail were incredible to me.