6 ms·

> Users requiring raw chains of thought for advanced prompt engineering can contact sales So it seems like all 3 of the LLM providers are now hiding the CoT -

> Users requiring raw chains of thought for advanced prompt engineering can contact sales So it seems like all 3 of the LLM providers are now hiding the CoT - which is a shame, because it helped to see when it was going to go down the wrong track, and allowing to quickly refine the prompt to ensure it didn't. In addition to openAI, Google also just recently started summarizing the CoT, replacing it with an, in my opinion, overly dumbed down summary.

Aeolun 1y ago
The Google CoT is so incredibly dumb. I thought my models had been lobotomized until I realized they must be doing some sort of processing on the thing.
- whimsicalism 1y ago
  it’s fake cot, just like oai
  - phatfish 1y ago
    I had assumed it was a way to reduce "hallucinations". Instead of me having to double check every response and prompt it again to clear up the obvious mistakes it just does that in the background with itself for a bit. Obviously the user still has to double check the response, but less often.
- user_7832 1y ago
  You are referring to the new (few days old-ish) CoT right? It’s bizzare as to why google did it, it was very helpful to see where the model was making assumptions or doing something wrong. Now half the time it feels better to just use flash with no thinking mode but ask it to manually “think”.
sunaookami 1y ago
Guess we have to wait till DeepSeek mops the floor with everyone again.
- datpuz 1y ago
  DeepSeek never mopped the floor with anyone... DeepSeek was remarkable because it is claimed that they spent a lot less training it, and without Nvidia GPUs, and because they had the best open weight model for a while. The only area they mopped the floor in was open source models, which had been stagnating for a while. But qwen3 mopped the floor with DeepSeek R1.
  - codyvoda 1y ago
    counterpoint: influencers said they wiped the floor with everyone so it must have happened
    - sunaookami 1y ago
      Who cares about what random influencers say?
      - infecto 1y ago
        I think he is hinting at folks like you who say things like Deepseek mopping the floor when beyond some contribution to the open source community which was indeed impressive, there really has been not much of a change. No floors were mopped.
        sunaookami 1y ago
        See the other comments. There was change. Don't know what that has to do with influencers, I don't follow these people.
        infecto 1y ago
        No floors were mopped. See comment you replied to. Change happened, their research was great but no floors were mopped.
  - barnabee 1y ago
pja 1y ago
IIRC RLHF inevitably compromises model accuracy in order to train the model not to give dangerous responses. It would make sense if the model used for train-of-though was trained differently (perhaps a different expert from an MoE?) from the one used to interact with the end user, since the end user is only ever going to see its output filtered through the public model the chain-of-thought model can be closer to the original, more pre-rlhf version without risking the reputation of the company. This way you can get the full performance of the original model whilst still maintaining the necessary filtering required to prevent actual harm (or terrible PR disasters).
- landl0rd 1y ago
  Yeah we really should stop focusing on model alignment. The idea that it's more important that your AI will fucking report you to the police if it thinks you're being naughty than that it actually works for more stuff is stupid.
  - xp84 1y ago
    I'm not sure I'd throw out all the alignment baby with the bathwater. But I wish we could draw a distinction between "Might offend someone" with "dangerous." Even 'plotting terror attacks' is not something terrorists can do just fine without AI. And as for making sure the model wouldn't say ideas that are hurtful to <insert group>, it seems to me so silly when it's text we're talking about. If I want to say "<insert group> are lazy and stupid," I can type that myself (and it's even protected speech in some countries still!) How does preventing Claude from espousing that dumb opinion, keep <insert group> safe from anything?
    - landl0rd 1y ago
      Let me put it this way: there are very few things I can think of that models should absolutely refuse, because there are very few pieces of information that are net harmful in all cases and at all times. I sort of run by blackstone's principle on this: it is better to grant 10 bad men access to information than to deny that access to 1 good one. Easy example: Someone asks the robot for advice on stacking/shaping a bunch of tannerite to better focus a blast. The model says he's a terrorist. In fact, he's doing what any number of us have done and just having fun blowing some stuff up on his ranch. Or I raised this one elsewhere but ochem is an easy example. I've had basically all the models claim that random amines are illegal, potentially psychoactive, verboten. I don't really feel like having my door getting kicked down by agents with guns, getting my dog shot, maybe getting shot myself because the robot tattled on me for something completely legal. For that matter if someone wants to synthesize some molly the robot shouldn't tattle to the feds about that either. Basically it should just do what users tell it to do excepting the very minimal cases where something is basically always bad.
make3 1y ago
it just makes it too easy to distill the reasoning into a separate model I guess. though I feel like o3 shows useful things about the reasoning while it's happening
BoredPositron 1y ago
Because it's alchemy and everyone believes they have an edge on turning lead into gold.
- elcritch 1y ago
  I've been thinking for a couple of months now that prompt engineering, and therefore CoT, is going to become the "secret sauce" companies want to hold onto. If anything that is where the day to day pragmatic engineering gets done. Like with early chemistry, we didn't need to precisely understand chemical theory to produce mass industrial processes by making a good enough working model, some statistical parameters, and good ole practical experience. People figured out steel making and black powder with alchemy. The only debate now is whether the prompt engineering models are currently closer to alchemy or modern chemistry? I'd say we're at advanced alchemy with some hints of rudimentary chemistry. Also, unrelated but with CERN turning lead into gold, doesn't that mean the alchemists were correct, just fundamentally unprepared for the scale of the task? ;)
  - parodysbird 1y ago
    The thing with alchemy was not that their hypotheses were wrong (they eventually created chemistry), but that their method of secret esoteric mysticism over open inquiry was wrong. Newton is the great example of this: he led a dual life, where in one he did science openly to a community to scrutinize, in the other he did secret alchemy in search of the philosopher's stone. History has empirically shown us which of his lives actually led to the discovery and accumulation of knowledge, and which did not.
    - elcritch 1y ago
      That’s what the Illuminati wants you to think. Jk ;) Gotta admit the occult side does make for much more enjoyable movie and book plot lines though.
    - iamcurious 1y ago
      Newton was a smart guy and he devoted a lot of time to his occult research. I bet that a lot of that occult research inspired the physics. The fact that his occult research remains, occult from the public, well that is natural aint it?
whiddershins 1y ago
The trend towards opaque is inexorable. https://noisegroove.substack.com/p/somersaulting-down-the-slippery-slope https://noisegroove.substack.com/p/somersaulting-down-the-sl...
a_bonobo 1y ago
Could the exclusion of CoT that be because of this recent Anthropic paper? https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf https://assets.anthropic.com/m/71876fabef0f0ed4/original/rea... >We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. I.e., chain of thought may be a confabulation by the model, too. So perhaps there's somebody at Anthropic who doesn't want to mislead their customers. Perhaps they'll come back once this problem is solved.
- whimsicalism 1y ago
  i think it is almost certainly to prevent distillation
- andrepd 1y ago
  I have no idea what this means, can someone give the eli5?
  - BowBun 1y ago
    Ask an LLM!
  - a_bonobo 1y ago
    Anthropic has a nice press release that summarises it in simpler terms: https://www.anthropic.com/research/reasoning-models-dont-say-think https://www.anthropic.com/research/reasoning-models-dont-say...
  - otabdeveloper4 1y ago
    I don't either, but chain of thought is obviously bullshit and just more LLM hallucination. LLMs will routinely "reason" through a solution and then proceed to give out a final answer that is completely unrelated to the preceding "reasoning".
epolanski 1y ago
> because it helped to see when it was going to go down the wrong track It helped me tremendously learning Zig. Seeing his chain of thought when asking it stuff about Zig and implementations let me widen the horizon a lot.

Aeolun 1y ago

The Google CoT is so incredibly dumb. I thought my models had been lobotomized until I realized they must be doing some sort of processing on the thing.

whimsicalism 1y ago
it’s fake cot, just like oai
- phatfish 1y ago
  I had assumed it was a way to reduce "hallucinations". Instead of me having to double check every response and prompt it again to clear up the obvious mistakes it just does that in the background with itself for a bit. Obviously the user still has to double check the response, but less often.
user_7832 1y ago
You are referring to the new (few days old-ish) CoT right? It’s bizzare as to why google did it, it was very helpful to see where the model was making assumptions or doing something wrong. Now half the time it feels better to just use flash with no thinking mode but ask it to manually “think”.

sunaookami 1y ago

Guess we have to wait till DeepSeek mops the floor with everyone again.

datpuz 1y ago
DeepSeek never mopped the floor with anyone... DeepSeek was remarkable because it is claimed that they spent a lot less training it, and without Nvidia GPUs, and because they had the best open weight model for a while. The only area they mopped the floor in was open source models, which had been stagnating for a while. But qwen3 mopped the floor with DeepSeek R1.
- codyvoda 1y ago
  counterpoint: influencers said they wiped the floor with everyone so it must have happened
  - sunaookami 1y ago
    Who cares about what random influencers say?
    - infecto 1y ago
      I think he is hinting at folks like you who say things like Deepseek mopping the floor when beyond some contribution to the open source community which was indeed impressive, there really has been not much of a change. No floors were mopped.
      - sunaookami 1y ago
        See the other comments. There was change. Don't know what that has to do with influencers, I don't follow these people.
        infecto 1y ago
        No floors were mopped. See comment you replied to. Change happened, their research was great but no floors were mopped.
- barnabee 1y ago

pja 1y ago

IIRC RLHF inevitably compromises model accuracy in order to train the model not to give dangerous responses. It would make sense if the model used for train-of-though was trained differently (perhaps a different expert from an MoE?) from the one used to interact with the end user, since the end user is only ever going to see its output filtered through the public model the chain-of-thought model can be closer to the original, more pre-rlhf version without risking the reputation of the company. This way you can get the full performance of the original model whilst still maintaining the necessary filtering required to prevent actual harm (or terrible PR disasters).

landl0rd 1y ago
Yeah we really should stop focusing on model alignment. The idea that it's more important that your AI will fucking report you to the police if it thinks you're being naughty than that it actually works for more stuff is stupid.
- xp84 1y ago
  I'm not sure I'd throw out all the alignment baby with the bathwater. But I wish we could draw a distinction between "Might offend someone" with "dangerous." Even 'plotting terror attacks' is not something terrorists can do just fine without AI. And as for making sure the model wouldn't say ideas that are hurtful to <insert group>, it seems to me so silly when it's text we're talking about. If I want to say "<insert group> are lazy and stupid," I can type that myself (and it's even protected speech in some countries still!) How does preventing Claude from espousing that dumb opinion, keep <insert group> safe from anything?
  - landl0rd 1y ago
    Let me put it this way: there are very few things I can think of that models should absolutely refuse, because there are very few pieces of information that are net harmful in all cases and at all times. I sort of run by blackstone's principle on this: it is better to grant 10 bad men access to information than to deny that access to 1 good one. Easy example: Someone asks the robot for advice on stacking/shaping a bunch of tannerite to better focus a blast. The model says he's a terrorist. In fact, he's doing what any number of us have done and just having fun blowing some stuff up on his ranch. Or I raised this one elsewhere but ochem is an easy example. I've had basically all the models claim that random amines are illegal, potentially psychoactive, verboten. I don't really feel like having my door getting kicked down by agents with guns, getting my dog shot, maybe getting shot myself because the robot tattled on me for something completely legal. For that matter if someone wants to synthesize some molly the robot shouldn't tattle to the feds about that either. Basically it should just do what users tell it to do excepting the very minimal cases where something is basically always bad.

make3 1y ago

it just makes it too easy to distill the reasoning into a separate model I guess. though I feel like o3 shows useful things about the reasoning while it's happening

BoredPositron 1y ago

Because it's alchemy and everyone believes they have an edge on turning lead into gold.

elcritch 1y ago
I've been thinking for a couple of months now that prompt engineering, and therefore CoT, is going to become the "secret sauce" companies want to hold onto. If anything that is where the day to day pragmatic engineering gets done. Like with early chemistry, we didn't need to precisely understand chemical theory to produce mass industrial processes by making a good enough working model, some statistical parameters, and good ole practical experience. People figured out steel making and black powder with alchemy. The only debate now is whether the prompt engineering models are currently closer to alchemy or modern chemistry? I'd say we're at advanced alchemy with some hints of rudimentary chemistry. Also, unrelated but with CERN turning lead into gold, doesn't that mean the alchemists were correct, just fundamentally unprepared for the scale of the task? ;)
- parodysbird 1y ago
  The thing with alchemy was not that their hypotheses were wrong (they eventually created chemistry), but that their method of secret esoteric mysticism over open inquiry was wrong. Newton is the great example of this: he led a dual life, where in one he did science openly to a community to scrutinize, in the other he did secret alchemy in search of the philosopher's stone. History has empirically shown us which of his lives actually led to the discovery and accumulation of knowledge, and which did not.
  - elcritch 1y ago
    That’s what the Illuminati wants you to think. Jk ;) Gotta admit the occult side does make for much more enjoyable movie and book plot lines though.
  - iamcurious 1y ago
    Newton was a smart guy and he devoted a lot of time to his occult research. I bet that a lot of that occult research inspired the physics. The fact that his occult research remains, occult from the public, well that is natural aint it?

whiddershins 1y ago

The trend towards opaque is inexorable. https://noisegroove.substack.com/p/somersaulting-down-the-slippery-slope https://noisegroove.substack.com/p/somersaulting-down-the-sl...

a_bonobo 1y ago

Could the exclusion of CoT that be because of this recent Anthropic paper? https://assets.anthropic.com/m/71876fabef0f0ed4/original/reasoning_models_paper.pdf https://assets.anthropic.com/m/71876fabef0f0ed4/original/rea... >We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. I.e., chain of thought may be a confabulation by the model, too. So perhaps there's somebody at Anthropic who doesn't want to mislead their customers. Perhaps they'll come back once this problem is solved.

whimsicalism 1y ago
i think it is almost certainly to prevent distillation
andrepd 1y ago
I have no idea what this means, can someone give the eli5?
- BowBun 1y ago
  Ask an LLM!
- a_bonobo 1y ago
  Anthropic has a nice press release that summarises it in simpler terms: https://www.anthropic.com/research/reasoning-models-dont-say-think https://www.anthropic.com/research/reasoning-models-dont-say...
- otabdeveloper4 1y ago
  I don't either, but chain of thought is obviously bullshit and just more LLM hallucination. LLMs will routinely "reason" through a solution and then proceed to give out a final answer that is completely unrelated to the preceding "reasoning".

epolanski 1y ago

> because it helped to see when it was going to go down the wrong track It helped me tremendously learning Zig. Seeing his chain of thought when asking it stuff about Zig and implementations let me widen the horizon a lot.