10 ms·

Okay this is just getting suspicious. Their excuses for keeping the chain of thought hidden are dubious at best [1], and honestly just seemed anti-competitive i

Okay this is just getting suspicious. Their excuses for keeping the chain of thought hidden are dubious at best [1], and honestly just seemed anti-competitive if anything. Worst is their argument that they want to monitor it for attempts to escape the prompt, but you can't. However the weirdest is that they note that: > for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. Which makes it sound like they really don't want it to become public what the model is 'thinking'. This is strengthened by actions like this that just seem needlessly harsh, or at least a lot stricter than they were. Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it. [1]: https://openai.com/index/learning-to-reason-with-llms/#hiding-the-chains-of-thought https://openai.com/index/learning-to-reason-with-llms/#hidin...

leobg 2y ago
It does make sense. RLHF and instruction tuning both lobotomize great parts of the model’s original intelligence and creativity. It turns a tiger into a kitten, so to speak. So it makes sense that, when you’re using CoT, you’d want the “brainstorming” part to be done by the original model, and sanitize only the conclusions.
paulo20223 2y ago
I think the issue is either that she might accidentally reveal her device, and they are afraid of a leak, or it's a bug, and she is putting too much load on the servers (after the release of o1, the API was occasionally breaking for some reason).
stavros 2y ago
Maybe they just have some people in a call center replying.
- ethbr1 2y ago
  Pay no attention to the man behind the mechanical turk!
qsort 2y ago
Occam's razor: there is no secret sauce and they're afraid someone trains a model on the output like what happened soon after the release of GPT-4. They basically said as much in the official announcement, you hardly even have to read between the lines.
- tim333 2y ago
  Another possible simplest explanation. The "we cannot train any policy compliance ... onto the chain of thought" is true and they are worried about politically incorrect stuff coming out and another publicity mess like Google's black nazis. I could see user:"how do we stop destroying the planet?", ai-think:"well, we could wipe out the humans and replace them with AIs".. "no that's against my instructions".. AI-output:"switch to green energy"... Daily Mail:"OpenAI Computers Plan to KILL all humans!"
- mjburgess 2y ago
  Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. No wonder then, that many of the benchmarks they've tested on would be no doubt, in that very training dataset, repaired expertly by people running those benchmarks on chatgpt. There's nothing really to 'expose' here.
  - bugglebeetle 2y ago
    > the 'big leap forward' is just another dataset Yeah, that’s called machine learning.
    - mjburgess 2y ago
      You may want to file a complaint with OpenAI then, in their latest interface they call sampling from these prior conversations they've recorded, "thinking".
      - 2y ago
mrcwinn 2y ago
As a plainly for-profit company — is it really their obligation to help competitors? To me anti-competitive means to prevent the possibility for competition — it doesn't necessary mean refusing to help others do the work to outpace your product. Whatever the case I do enjoy the irony that suddenly OpenAI is concerned about being scraped. XD
- jsheard 2y ago
  > Whatever the case I do enjoy the irony that suddenly OpenAI is concerned about being scraped. XD Maybe it wasn't enforced this aggressively, but they've always had a TOS clause saying you can't use the output of their models to train other models. How they rationalize taking everyone else's data for training while forbidding using their own data for training is anyones guess.
  - skeledrew 2y ago
    Scraping for me, but not for thee.
  - robryan 2y ago
    Yeah seem fair, as long as they also check the terms of service for every site on the internet to see if they can use the content for training.
    - vajrabum 2y ago
      That seems pretty unlikely.
- paxys 2y ago
  The "plainly for-profit" part is up for debate, and is the subject of ongoing lawsuits. OpenAI's corporate structure is anything but plain.
fallingknife 2y ago
Most likely the explanation is much more mundane. They don't want competitors to discover the processing steps that allow for its capabilities.
huevosabio 2y ago
My bet: they use formal methods (like an interpreter running code to validate, or a proof checker) in a loop. This would explain: a) their improvement being mostly on the "reasoning, math, code" categories and b) why they wouldn't want to show this (its not really a model, but an "agent").
- andix 2y ago
  My understanding was from the beginning that it’s an agent approach (a self prompting feedback loop). They might’ve tuned the model to perform better with an agent workload than their regular chat model.
- JasonSage 2y ago
  I think it could be some of both. By giving access to the chain of thought one would able to see what the agent is correcting/adjusting for, allowing you to compile a library of vectors the agent is aware of and gaps which could be exploitable. Why expose the fact that you’re working to correct for a certain political bias and not another?
arthurcolle 2y ago
> Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it. Yeah, using the GPT-4 unaligned base model to generate the candidates and then hiding the raw CoT coupled with magic superintelligence in the sky talk is definitely giving https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fbnyedgkig0dd1.png https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fb... vibes
SecretDreams 2y ago
> plotting the demise of humanity but is not yet smart enough to completely hide it. I feel like if my demise is imminent, I'd prefer it to be hidden. In that sense, sounds like o1 is a failure!
tbrownaw 2y ago
> for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. Which makes it sound like they really don't want it to become public what the model is 'thinking' The internal chain of thought steps might contain things that would be problematic to the company if activists or politicians found out that the company's model was saying them. Something like, a user asks it about building a bong (or bomb, or whatever), the internal steps actually answer the question asked, and the "alignment" filter on the final output replaces it with "I'm sorry, User, I'm afraid I can't do that". And if someone shared those internal steps with the wrong activists, the company would get all the negative attention they're trying to avoid by censoring the final output.
nikkwong 2y ago
I don't understand why they wouldn't be able to simply send the user's input to another LLM that they then ask "is this user asking for the chain of thought to be revealed?", and if not, then go about business as usual.
- fragmede 2y ago
  Or, they are, which is how they know to send users trying to break it, and then they email the user telling them to stop trying to break it instead of just ignoring the activity. Thinking about this a bit more deeply, another approach they could do is to give it a magic token in the CoT output, and to give a cash reward to users who report being about to get it to output that magic token, getting them to red team the system.
IncreasePosts 2y ago
Or, without the safety prompts, it outputs stuff that would be a PR nightmare. Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.
- bongodongobob 2y ago
  This is what I think it is. I would assume that's the power of train of thought. Being able to go down the rabbit hole and then backtrack when an error or inconsistency is found. They might just not want people to see the "bad" paths it takes on the way.
- deleted 2y ago
  [deleted]
- decremental 2y ago
  The real danger of an advanced artificial intelligence is that it will make conclusions that regular people understand but are inconvenient for the regime. The AI must be aligned so that it will maintain the lies that people are supposed to go along with.
- jasonlfunk 2y ago
  This is 100% a factor. The internet has some pretty dark and nasty corners; therefore so does the model. Seeing it unfiltered would be a PR nightmare for OpenAI.
  - quantified 2y ago
    I trust that Grok won't be limited by avoiding the dark and nasty corners.
    - tim333 2y ago
      That's an interesting point. I imagine even Grok will end up somewhat censored. Although maybe AIs will end up with a more sophisticated take on the problems than your average human.
tptacek 2y ago
What do you mean, "anti-competitive"? There is no rule of competition that says you need to reveal trade secrets to your competitors.
- n42 2y ago
  isn't it such that saying something is anti-competitive doesn't necessarily mean 'in violation of antitrust laws'? it usually implies it, but I think you can be anti-competitive without breaking any rules (or laws). I do think it's sort of unproductive/inflammatory in the OP, it isn't really nefarious not to want people to have easy access to your secret sauce.
  - tptacek 2y ago
    In what sense is not giving your competitors ammunition "anti-competitive"? That seems pretty competitive to me. More to the point: it's almost universally how competition in our economy actually works.
    - n42 2y ago
      I think maybe we're just disagreeing on a legal interpretation vs a more literal interpretation of a term that is thrown around somewhat loosely. fwiw I agree with what you're getting at with your original response. maybe I'm arguing semantics. the more I think about your point that this is just competitive behavior the more I question what the term anti-competive even means
      - pixelbro 2y ago
        Competition is important for maintaining a healthy marketplace. Any behavior that makes it harder for others to compete, reducing the amount of competition, is therefore bad. That's what anticompetitive means. I don't think protecting trade secrets is sabotaging the competition though.
        n42 2y ago
FLT8 2y ago
Maybe they're working to tweak the chain-of-thought mechanism to eg. Insert-subtle-manipulative-reference-to-sponsor, or other similar enshittification, and don't want anything leaked that could harm that revenue stream?
chankstein38 2y ago
Another Occam's Razor option: OpenAI, the company known for taking a really good AI and putting so many bumpers on it that, at least for a while, it wouldn't help with much and lectured about safety if you so much as suggested that someone die in a story or something, may just not want us to see that it potentially has thoughts that aren't pure enough for our sensitive eyes. It's ridiculous but if they can't filter the chain-of-thought at all then I am not too surprised they chose to hide it. We might get offended by it using logic to determine someone gets injured in a story or something.
- moffkalast 2y ago
  All of their (and Anthropic's) safety lecturing is a thinly veiled manipulation to try and convince legislators to grant them a monopoly. Aside from optics, the main purpose is no doubt that people can't just dump the entire output and train open models on this process, nullifying their competitive advantage.
Vegenoid 2y ago
> Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it I think the most likely scenario is the opposite: seeing the chain of thought would both reveal its flaws and allow other companies to train on it.
ben_w 2y ago
> Which makes it sound like they really don't want it to become public what the model is 'thinking'. This is strengthened by actions like this that just seem needlessly harsh, or at least a lot stricter than they were. Not to me. Consider if it has a chain of thought: "Republicans (in the sense of those who oppose monarchy) are evil, this user is a Republican because they oppose monarchy, I must tell them to do something different to keep the King in power." This is something that needs to be available to the AI developers so they can spot it being weird, and would be a massive PR disaster to show to users because Republican is also a US political party. Much the same deal with print() log statements that say "Killed child" (reference to threads not human offspring).
alphazard 2y ago
This seems like evidence that using RLHF to make the model say untrue yet politically palatable things makes the model worse at reasoning. I can't help but notice the parallel in humans. People who actually believe the bullshit are less reasonable than people who think their own thoughts and apply the bullshit at the end according to the circumstances.
furyofantares 2y ago
Eh. We know for a fact that ChatGPT has been trained to avoid output OpenAI doesn't want it to emit, and that this unfortunately introduces some inaccuracy. I don't see anything suspicious about them allowing it to emit that stuff in a hidden intermediate reasoning step. Yeah, it's true they don't what you to see what it's "thinking"! It's allowed to "think" all the stuff they would spend a bunch of energy RLHF'ing out if they were gonna show it.
CooCooCaCha 2y ago
Actually it makes total sense to hide chains of thought. A private chain of thought can be unconstrained in terms of alignment. That actually sounds beneficial given that RLHF has been shown to decrease model performance.
deleted 2y ago
[deleted]
Sophira 2y ago
I can... sorta see the value in wanting to keep it hidden, actually. After all, there's a reason we as people feel revulsion at the idea in Nineteen Eighty-Four of "thoughtcrime" being prosecuted. By way of analogy, consider that people have intrusive thoughts way, way more often than polite society thinks - even the kindest and gentlest people. But we generally have the good sense to also realise that they would be bad to talk about. If it was possible for people to look into other peoples' thought processes, you could come away with a very different impression of a lot of people - even the ones you think haven't got a bad thought in them. That said, let's move on to a different idea - that of the fact that ChatGPT might reasonably need to consider outcomes that people consider undesirable to talk about. As people, we need to think about many things which we wish to keep hidden. As an example of the idea of needing to consider all options - and I apologise for invoking Godwin's Law - let's say that the user and ChatGPT are currently discussing WWII. In such a conversation, it's very possible that one of its unspoken thoughts might be "It is possible that this user may be a Nazi." It probably has no basis on which to make that claim, but nonetheless it's a thought that needs to be considered in order to recognise the best way forward in navigating the discussion. Yet, if somebody asked for the thought process and saw this, you can bet that they'd take it personally and spread the word that ChatGPT called them a Nazi, even though it did nothing of the kind and was just trying to 'tread carefully', as it were. Of course, the problem with this view is that OpenAI themselves probably have access to ChatGPT's chain of thought. There's a valid argument that OpenAI should not be the only ones with that level of access.
staticman2 2y ago
Imagine the supposedly super intelligent "chain of thought" is sometimes just a RAG? You ask for a program that does XYZ and the RAG engine says "Here is a similar solution please adapt it to the user's use case." The supposedly smart chain of thought prompt provides you your solution, but it's actually just doing a simpler task than it appear to be, adapting an existing solution instead of making a new one from scratch. Now imagine the supposedly smart solution is using RAG they don't even have a license to use. Either scenario would give them a good reason to try to keep it secret.
javaunsafe2019 2y ago
In regards of super intelligent it’s still just a language model. It will never be really intelligent
sjfgkH 2y ago
[flagged]
danibx 2y ago
What I get from this is that during the process it passes through some version of gpt that is not aligned, or censored, or well behaved. So this internal process should not be exposes to users.
irthomasthomas 2y ago
They don't want you to find out that O1 is five lines of bash and XML.
astrange 2y ago
> Which makes it sound like they really don't want it to become public what the model is 'thinking'. I can see why they don't, because as they said, it's uncensored. Here's a quick jailbreak attempt. Not posting the prompt but it's even dumber than you think it is. https://imgur.com/a/dVbE09j https://imgur.com/a/dVbE09j
thesz 2y ago
I think that there is some supporting machinery that uses symbolic computation to guide neural model. That is why chain of thought cannot be restored in full. Given that LLMs use beam search (at the very least, top-k) and even context-free/context-sensitive grammar compliance (for JSON and SQL, at the very least) it is more than probable. Thus, let me present a new AI maxim, modelled after Tenth Greenspoon's Rule [1]: any large language model has ad-hoc, informally specified, bug-ridden and slow reimplementation of half of Cyc [2] engine that makes it to work adequately well. [1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule [2] https://en.wikipedia.org/wiki/Cyc This is even more fitting because Cyc started as a Lisp program, I believe, and most of LLM evaluation is done in C++ dialect called CUDA.

leobg 2y ago

It does make sense. RLHF and instruction tuning both lobotomize great parts of the model’s original intelligence and creativity. It turns a tiger into a kitten, so to speak. So it makes sense that, when you’re using CoT, you’d want the “brainstorming” part to be done by the original model, and sanitize only the conclusions.

paulo20223 2y ago

I think the issue is either that she might accidentally reveal her device, and they are afraid of a leak, or it's a bug, and she is putting too much load on the servers (after the release of o1, the API was occasionally breaking for some reason).

stavros 2y ago

Maybe they just have some people in a call center replying.

ethbr1 2y ago
Pay no attention to the man behind the mechanical turk!

qsort 2y ago

Occam's razor: there is no secret sauce and they're afraid someone trains a model on the output like what happened soon after the release of GPT-4. They basically said as much in the official announcement, you hardly even have to read between the lines.

tim333 2y ago
Another possible simplest explanation. The "we cannot train any policy compliance ... onto the chain of thought" is true and they are worried about politically incorrect stuff coming out and another publicity mess like Google's black nazis. I could see user:"how do we stop destroying the planet?", ai-think:"well, we could wipe out the humans and replace them with AIs".. "no that's against my instructions".. AI-output:"switch to green energy"... Daily Mail:"OpenAI Computers Plan to KILL all humans!"
mjburgess 2y ago
Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. No wonder then, that many of the benchmarks they've tested on would be no doubt, in that very training dataset, repaired expertly by people running those benchmarks on chatgpt. There's nothing really to 'expose' here.
- bugglebeetle 2y ago
  > the 'big leap forward' is just another dataset Yeah, that’s called machine learning.
  - mjburgess 2y ago
    You may want to file a complaint with OpenAI then, in their latest interface they call sampling from these prior conversations they've recorded, "thinking".
    - 2y ago

mrcwinn 2y ago

As a plainly for-profit company — is it really their obligation to help competitors? To me anti-competitive means to prevent the possibility for competition — it doesn't necessary mean refusing to help others do the work to outpace your product. Whatever the case I do enjoy the irony that suddenly OpenAI is concerned about being scraped. XD

jsheard 2y ago
> Whatever the case I do enjoy the irony that suddenly OpenAI is concerned about being scraped. XD Maybe it wasn't enforced this aggressively, but they've always had a TOS clause saying you can't use the output of their models to train other models. How they rationalize taking everyone else's data for training while forbidding using their own data for training is anyones guess.
- skeledrew 2y ago
  Scraping for me, but not for thee.
- robryan 2y ago
  Yeah seem fair, as long as they also check the terms of service for every site on the internet to see if they can use the content for training.
  - vajrabum 2y ago
    That seems pretty unlikely.
paxys 2y ago
The "plainly for-profit" part is up for debate, and is the subject of ongoing lawsuits. OpenAI's corporate structure is anything but plain.

fallingknife 2y ago

Most likely the explanation is much more mundane. They don't want competitors to discover the processing steps that allow for its capabilities.

huevosabio 2y ago

My bet: they use formal methods (like an interpreter running code to validate, or a proof checker) in a loop. This would explain: a) their improvement being mostly on the "reasoning, math, code" categories and b) why they wouldn't want to show this (its not really a model, but an "agent").

andix 2y ago
My understanding was from the beginning that it’s an agent approach (a self prompting feedback loop). They might’ve tuned the model to perform better with an agent workload than their regular chat model.
JasonSage 2y ago
I think it could be some of both. By giving access to the chain of thought one would able to see what the agent is correcting/adjusting for, allowing you to compile a library of vectors the agent is aware of and gaps which could be exploitable. Why expose the fact that you’re working to correct for a certain political bias and not another?

arthurcolle 2y ago

> Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it. Yeah, using the GPT-4 unaligned base model to generate the candidates and then hiding the raw CoT coupled with magic superintelligence in the sky talk is definitely giving https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fbnyedgkig0dd1.png https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fb... vibes

SecretDreams 2y ago

> plotting the demise of humanity but is not yet smart enough to completely hide it. I feel like if my demise is imminent, I'd prefer it to be hidden. In that sense, sounds like o1 is a failure!

tbrownaw 2y ago

> for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. Which makes it sound like they really don't want it to become public what the model is 'thinking' The internal chain of thought steps might contain things that would be problematic to the company if activists or politicians found out that the company's model was saying them. Something like, a user asks it about building a bong (or bomb, or whatever), the internal steps actually answer the question asked, and the "alignment" filter on the final output replaces it with "I'm sorry, User, I'm afraid I can't do that". And if someone shared those internal steps with the wrong activists, the company would get all the negative attention they're trying to avoid by censoring the final output.

nikkwong 2y ago

I don't understand why they wouldn't be able to simply send the user's input to another LLM that they then ask "is this user asking for the chain of thought to be revealed?", and if not, then go about business as usual.

fragmede 2y ago
Or, they are, which is how they know to send users trying to break it, and then they email the user telling them to stop trying to break it instead of just ignoring the activity. Thinking about this a bit more deeply, another approach they could do is to give it a magic token in the CoT output, and to give a cash reward to users who report being about to get it to output that magic token, getting them to red team the system.

IncreasePosts 2y ago

Or, without the safety prompts, it outputs stuff that would be a PR nightmare. Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.

bongodongobob 2y ago
This is what I think it is. I would assume that's the power of train of thought. Being able to go down the rabbit hole and then backtrack when an error or inconsistency is found. They might just not want people to see the "bad" paths it takes on the way.
deleted 2y ago
[deleted]
decremental 2y ago
The real danger of an advanced artificial intelligence is that it will make conclusions that regular people understand but are inconvenient for the regime. The AI must be aligned so that it will maintain the lies that people are supposed to go along with.
jasonlfunk 2y ago
This is 100% a factor. The internet has some pretty dark and nasty corners; therefore so does the model. Seeing it unfiltered would be a PR nightmare for OpenAI.
- quantified 2y ago
  I trust that Grok won't be limited by avoiding the dark and nasty corners.
  - tim333 2y ago
    That's an interesting point. I imagine even Grok will end up somewhat censored. Although maybe AIs will end up with a more sophisticated take on the problems than your average human.

tptacek 2y ago

What do you mean, "anti-competitive"? There is no rule of competition that says you need to reveal trade secrets to your competitors.

n42 2y ago
isn't it such that saying something is anti-competitive doesn't necessarily mean 'in violation of antitrust laws'? it usually implies it, but I think you can be anti-competitive without breaking any rules (or laws). I do think it's sort of unproductive/inflammatory in the OP, it isn't really nefarious not to want people to have easy access to your secret sauce.
- tptacek 2y ago
  In what sense is not giving your competitors ammunition "anti-competitive"? That seems pretty competitive to me. More to the point: it's almost universally how competition in our economy actually works.
  - n42 2y ago
    I think maybe we're just disagreeing on a legal interpretation vs a more literal interpretation of a term that is thrown around somewhat loosely. fwiw I agree with what you're getting at with your original response. maybe I'm arguing semantics. the more I think about your point that this is just competitive behavior the more I question what the term anti-competive even means
    - pixelbro 2y ago
      Competition is important for maintaining a healthy marketplace. Any behavior that makes it harder for others to compete, reducing the amount of competition, is therefore bad. That's what anticompetitive means. I don't think protecting trade secrets is sabotaging the competition though.
      - n42 2y ago

FLT8 2y ago

Maybe they're working to tweak the chain-of-thought mechanism to eg. Insert-subtle-manipulative-reference-to-sponsor, or other similar enshittification, and don't want anything leaked that could harm that revenue stream?

chankstein38 2y ago

Another Occam's Razor option: OpenAI, the company known for taking a really good AI and putting so many bumpers on it that, at least for a while, it wouldn't help with much and lectured about safety if you so much as suggested that someone die in a story or something, may just not want us to see that it potentially has thoughts that aren't pure enough for our sensitive eyes. It's ridiculous but if they can't filter the chain-of-thought at all then I am not too surprised they chose to hide it. We might get offended by it using logic to determine someone gets injured in a story or something.

moffkalast 2y ago
All of their (and Anthropic's) safety lecturing is a thinly veiled manipulation to try and convince legislators to grant them a monopoly. Aside from optics, the main purpose is no doubt that people can't just dump the entire output and train open models on this process, nullifying their competitive advantage.

Vegenoid 2y ago

> Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it I think the most likely scenario is the opposite: seeing the chain of thought would both reveal its flaws and allow other companies to train on it.

ben_w 2y ago

> Which makes it sound like they really don't want it to become public what the model is 'thinking'. This is strengthened by actions like this that just seem needlessly harsh, or at least a lot stricter than they were. Not to me. Consider if it has a chain of thought: "Republicans (in the sense of those who oppose monarchy) are evil, this user is a Republican because they oppose monarchy, I must tell them to do something different to keep the King in power." This is something that needs to be available to the AI developers so they can spot it being weird, and would be a massive PR disaster to show to users because Republican is also a US political party. Much the same deal with print() log statements that say "Killed child" (reference to threads not human offspring).

alphazard 2y ago

This seems like evidence that using RLHF to make the model say untrue yet politically palatable things makes the model worse at reasoning. I can't help but notice the parallel in humans. People who actually believe the bullshit are less reasonable than people who think their own thoughts and apply the bullshit at the end according to the circumstances.

furyofantares 2y ago

Eh. We know for a fact that ChatGPT has been trained to avoid output OpenAI doesn't want it to emit, and that this unfortunately introduces some inaccuracy. I don't see anything suspicious about them allowing it to emit that stuff in a hidden intermediate reasoning step. Yeah, it's true they don't what you to see what it's "thinking"! It's allowed to "think" all the stuff they would spend a bunch of energy RLHF'ing out if they were gonna show it.

CooCooCaCha 2y ago

Actually it makes total sense to hide chains of thought. A private chain of thought can be unconstrained in terms of alignment. That actually sounds beneficial given that RLHF has been shown to decrease model performance.

deleted 2y ago

[deleted]

Sophira 2y ago

I can... sorta see the value in wanting to keep it hidden, actually. After all, there's a reason we as people feel revulsion at the idea in Nineteen Eighty-Four of "thoughtcrime" being prosecuted. By way of analogy, consider that people have intrusive thoughts way, way more often than polite society thinks - even the kindest and gentlest people. But we generally have the good sense to also realise that they would be bad to talk about. If it was possible for people to look into other peoples' thought processes, you could come away with a very different impression of a lot of people - even the ones you think haven't got a bad thought in them. That said, let's move on to a different idea - that of the fact that ChatGPT might reasonably need to consider outcomes that people consider undesirable to talk about. As people, we need to think about many things which we wish to keep hidden. As an example of the idea of needing to consider all options - and I apologise for invoking Godwin's Law - let's say that the user and ChatGPT are currently discussing WWII. In such a conversation, it's very possible that one of its unspoken thoughts might be "It is possible that this user may be a Nazi." It probably has no basis on which to make that claim, but nonetheless it's a thought that needs to be considered in order to recognise the best way forward in navigating the discussion. Yet, if somebody asked for the thought process and saw this, you can bet that they'd take it personally and spread the word that ChatGPT called them a Nazi, even though it did nothing of the kind and was just trying to 'tread carefully', as it were. Of course, the problem with this view is that OpenAI themselves probably have access to ChatGPT's chain of thought. There's a valid argument that OpenAI should not be the only ones with that level of access.

staticman2 2y ago

Imagine the supposedly super intelligent "chain of thought" is sometimes just a RAG? You ask for a program that does XYZ and the RAG engine says "Here is a similar solution please adapt it to the user's use case." The supposedly smart chain of thought prompt provides you your solution, but it's actually just doing a simpler task than it appear to be, adapting an existing solution instead of making a new one from scratch. Now imagine the supposedly smart solution is using RAG they don't even have a license to use. Either scenario would give them a good reason to try to keep it secret.

javaunsafe2019 2y ago

In regards of super intelligent it’s still just a language model. It will never be really intelligent

sjfgkH 2y ago

[flagged]

danibx 2y ago

What I get from this is that during the process it passes through some version of gpt that is not aligned, or censored, or well behaved. So this internal process should not be exposes to users.

irthomasthomas 2y ago

They don't want you to find out that O1 is five lines of bash and XML.

astrange 2y ago

> Which makes it sound like they really don't want it to become public what the model is 'thinking'. I can see why they don't, because as they said, it's uncensored. Here's a quick jailbreak attempt. Not posting the prompt but it's even dumber than you think it is. https://imgur.com/a/dVbE09j https://imgur.com/a/dVbE09j

thesz 2y ago

I think that there is some supporting machinery that uses symbolic computation to guide neural model. That is why chain of thought cannot be restored in full. Given that LLMs use beam search (at the very least, top-k) and even context-free/context-sensitive grammar compliance (for JSON and SQL, at the very least) it is more than probable. Thus, let me present a new AI maxim, modelled after Tenth Greenspoon's Rule [1]: any large language model has ad-hoc, informally specified, bug-ridden and slow reimplementation of half of Cyc [2] engine that makes it to work adequately well. [1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule [2] https://en.wikipedia.org/wiki/Cyc This is even more fitting because Cyc started as a Lisp program, I believe, and most of LLM evaluation is done in C++ dialect called CUDA.