7 ms·

I don’t think you’re crazy, something is off in their models. As an example I’ve been using an MCP tool to provide table schemas to Claude for months. There w

I don’t think you’re crazy, something is off in their models. As an example I’ve been using an MCP tool to provide table schemas to Claude for months. There was a point where it stopped recognizing the tool unless mentioned in early August. Maybe that’s related to their degraded quality issue. This morning after pulling the correct schema info Sonnet started hallucinating columns (from Shopify’s API docs) and added them to my query. That’s a use case I’ve been doing daily for months and in the last few weeks has gone from consistent low supervision to flaky and low quality. I don’t know what’s going on, Sonnet has definitely felt worse, and the timeline matches their status page incident, but it’s definitely not resolved. Opus 4.1 also feels flaky, it feels like it’s less consistent about recalling earlier prompt details than 4.0. I personally am frustrated that there’s no refund or anything after a month of degraded performance, and they’ve had a lot of downtime.

dingnuts 1y ago
I have read so many anecdotes about so many models that "were great" and aren't now. I actually think this is psychological bias. It got a few things right early on, and that's what you remember. As time passes, the errors add up, until the memory doesn't match reality. The "new shiny" feeling goes away, and you perceive it for what it really is: a kind of shitty slot machine > personally am frustrated that there’s no refund or anything after a month of degraded performance lol, LMAO. A company operates a shitty slot machine at a loss and you're surprised they have "issues" that reduce your usage? I'm not paying for any of this shit until these companies figure out how to align incentives. If they make more by applying limits, or charge me when the machine makes errors, that's good for them and bad for me! Why should I continue to pay to pull on the slot machine lever? It's a waste of time and money. I'll be richer and more productive if I just write the code myself, and the result will be better too.
- reactordev 1y ago
  This is equivalent to people reminiscing about WoW or EverQuest saying gaming peaked back then… I think you’re right. I think it’s complete bias with a little bit of “it does more tasks now” so it might behave a bit differently to the same prompt. I also think you’re right that there’s an incentive to dumb it down so you pull the lever more. Just 2 more $1 spins and maybe you’ll hit jackpot. Really it’s the enshitification of the SOTA for profits and glory.
- adonese 1y ago
  Claude has been constantly terrible for the last couple of weeks. You must have seen this, but just in case: https://x.com/claudeai/status/1965208247302029728 https://x.com/claudeai/status/1965208247302029728
- lacy_tinpot 1y ago
  Except this is a verifiable thing that actually is acknowledged and even tracked by people.
  - throwaway314155 1y ago
    Go on then. Verify and track it. Or at least cite a source that does.
    - deleted 1y ago
      [deleted]
    - fragmede 1y ago
      https://x.com/claudeai/status/1965208247302029728 https://x.com/claudeai/status/1965208247302029728
    - lacy_tinpot 1y ago
      https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-...
- mordymoop 1y ago
  I think you’re onto something but it works the opposite way too. When you first start using a new model you are more forgiving because almost by definition you were using a worse model before. You give if the sorts of problems the old model couldn’t do, and the new model can do them; you see only success, and the places where it fails, well, you can’t have it all. Then after using the new model for a few months you get used to it, you feel like you know what it should be able to do, and when it can’t do that, you’re annoyed. You feel like it got worse. But what happened is your expectations crept up. You’re now constantly riding it at 95% of its capabilities and hitting more edge cases where it messes up. You think you’re doing everything consistently, but you’re not, you’ve dramatically dialed up your expectations and demands relative to what you were doing months ago. I don’t mean “you,” I mean the royal “you”, this is what we all do. If you think your expectations haven’t risen, go back and look at your commits from six months ago and tell me I’m wrong.
- holoduke 1y ago
  You are saying that you are writing mock data, boiler plate code all yourself? I seriously don't believe that. Llms are already much much faster in these tasks. There is no going back there.
reissbaker 1y ago
FWIW I strongly recommend using some of the recent, good Chinese OSS models. I love GLM-4.5, and Kimi K2 0905 is quite good as well.
- jimbo808 1y ago
  I'd like to give these a try - what's your way of using them? I mostly use Claude because of Claude Code. Not sure what agentic coding tools people are using these days with OSS models. I'm not a big fan of manually uploading files into a web UI.
  - billyjobob 1y ago
    Both of those models have Anthropic API compatible endpoints, so you just set an environmental variable pointing to them before you run Claude Code.
  - deleted 1y ago
    [deleted]
  - reissbaker 1y ago
    The most private way is to use them on your own machine; a Mac Studio maxed out to 512GB RAM can run GLM-4.5 at FP8 with fairly long context, for example. If you don't have the hardware to run it locally, let me shill my own company for a minute: Synthetic [1] has a $20/month subscription to most of the good open-weight coding LLMs, with higher rate limits than Claude's $20/month sub. And our $60/month sub has higher rate limits than the $200/month maxed-out version of the Claude Max plan. You can still use Claude Code by using LiteLLM or similar tools that convert Anthropic-style API requests to OpenAI-style API requests; once you have one of those running locally, you override the ANTHROPIC_BASE_URL env var to point to your locally-running proxy. We'll also be shipping an Anthropic-compatible API this week to work with Claude Code directly. Some other good agentic tools you could use instead include Cline, Roo Code, KiloCode, OpenCode, or Octofriend (the last of which we maintain). 1: https://synthetic.new https://synthetic.new
    - sheepscreek 1y ago
8note 1y ago
ive been thinking its that my company mcp has blown up in context size, but using claude without claude code, i get context window overflows constantly now. another option could be a system prompt change to make it too long?
- data-ottawa 1y ago
  I think that’s because of the Artifacts feature and how it works. For me after a few revisions it uses a ton of tokens. As a baseline from a real conversation, 270 lines of sql is ~2500 tokens. Every language will be different, this is what I have open. When Claude edits an artifact it seems to keep the revisions in the chat context, plus it’s doing multiple changes per revision. After 10 iterations on a 1k loc artifact (10k tokens) you’re at 100k tokens. claude.ai has a 200k token window according to their docs (not sure if that’s accurate though). Depending on how Claude is doing those in place edits that could be the whole budget right there.

dingnuts 1y ago

I have read so many anecdotes about so many models that "were great" and aren't now. I actually think this is psychological bias. It got a few things right early on, and that's what you remember. As time passes, the errors add up, until the memory doesn't match reality. The "new shiny" feeling goes away, and you perceive it for what it really is: a kind of shitty slot machine > personally am frustrated that there’s no refund or anything after a month of degraded performance lol, LMAO. A company operates a shitty slot machine at a loss and you're surprised they have "issues" that reduce your usage? I'm not paying for any of this shit until these companies figure out how to align incentives. If they make more by applying limits, or charge me when the machine makes errors, that's good for them and bad for me! Why should I continue to pay to pull on the slot machine lever? It's a waste of time and money. I'll be richer and more productive if I just write the code myself, and the result will be better too.

reactordev 1y ago
This is equivalent to people reminiscing about WoW or EverQuest saying gaming peaked back then… I think you’re right. I think it’s complete bias with a little bit of “it does more tasks now” so it might behave a bit differently to the same prompt. I also think you’re right that there’s an incentive to dumb it down so you pull the lever more. Just 2 more $1 spins and maybe you’ll hit jackpot. Really it’s the enshitification of the SOTA for profits and glory.
adonese 1y ago
Claude has been constantly terrible for the last couple of weeks. You must have seen this, but just in case: https://x.com/claudeai/status/1965208247302029728 https://x.com/claudeai/status/1965208247302029728
lacy_tinpot 1y ago
Except this is a verifiable thing that actually is acknowledged and even tracked by people.
- throwaway314155 1y ago
  Go on then. Verify and track it. Or at least cite a source that does.
  - deleted 1y ago
    [deleted]
  - fragmede 1y ago
    https://x.com/claudeai/status/1965208247302029728 https://x.com/claudeai/status/1965208247302029728
  - lacy_tinpot 1y ago
    https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues https://www.anthropic.com/engineering/a-postmortem-of-three-...
mordymoop 1y ago
I think you’re onto something but it works the opposite way too. When you first start using a new model you are more forgiving because almost by definition you were using a worse model before. You give if the sorts of problems the old model couldn’t do, and the new model can do them; you see only success, and the places where it fails, well, you can’t have it all. Then after using the new model for a few months you get used to it, you feel like you know what it should be able to do, and when it can’t do that, you’re annoyed. You feel like it got worse. But what happened is your expectations crept up. You’re now constantly riding it at 95% of its capabilities and hitting more edge cases where it messes up. You think you’re doing everything consistently, but you’re not, you’ve dramatically dialed up your expectations and demands relative to what you were doing months ago. I don’t mean “you,” I mean the royal “you”, this is what we all do. If you think your expectations haven’t risen, go back and look at your commits from six months ago and tell me I’m wrong.
holoduke 1y ago
You are saying that you are writing mock data, boiler plate code all yourself? I seriously don't believe that. Llms are already much much faster in these tasks. There is no going back there.

reissbaker 1y ago

FWIW I strongly recommend using some of the recent, good Chinese OSS models. I love GLM-4.5, and Kimi K2 0905 is quite good as well.

jimbo808 1y ago
I'd like to give these a try - what's your way of using them? I mostly use Claude because of Claude Code. Not sure what agentic coding tools people are using these days with OSS models. I'm not a big fan of manually uploading files into a web UI.
- billyjobob 1y ago
  Both of those models have Anthropic API compatible endpoints, so you just set an environmental variable pointing to them before you run Claude Code.
- deleted 1y ago
  [deleted]
- reissbaker 1y ago
  The most private way is to use them on your own machine; a Mac Studio maxed out to 512GB RAM can run GLM-4.5 at FP8 with fairly long context, for example. If you don't have the hardware to run it locally, let me shill my own company for a minute: Synthetic [1] has a $20/month subscription to most of the good open-weight coding LLMs, with higher rate limits than Claude's $20/month sub. And our $60/month sub has higher rate limits than the $200/month maxed-out version of the Claude Max plan. You can still use Claude Code by using LiteLLM or similar tools that convert Anthropic-style API requests to OpenAI-style API requests; once you have one of those running locally, you override the ANTHROPIC_BASE_URL env var to point to your locally-running proxy. We'll also be shipping an Anthropic-compatible API this week to work with Claude Code directly. Some other good agentic tools you could use instead include Cline, Roo Code, KiloCode, OpenCode, or Octofriend (the last of which we maintain). 1: https://synthetic.new https://synthetic.new
  - sheepscreek 1y ago

8note 1y ago

ive been thinking its that my company mcp has blown up in context size, but using claude without claude code, i get context window overflows constantly now. another option could be a system prompt change to make it too long?

data-ottawa 1y ago
I think that’s because of the Artifacts feature and how it works. For me after a few revisions it uses a ton of tokens. As a baseline from a real conversation, 270 lines of sql is ~2500 tokens. Every language will be different, this is what I have open. When Claude edits an artifact it seems to keep the revisions in the chat context, plus it’s doing multiple changes per revision. After 10 iterations on a 1k loc artifact (10k tokens) you’re at 100k tokens. claude.ai has a 200k token window according to their docs (not sure if that’s accurate though). Depending on how Claude is doing those in place edits that could be the whole budget right there.