6 ms·

>We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought

>We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users. >Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users. We acknowledge this decision has disadvantages. We strive to partially make up for it by teaching the model to reproduce any useful ideas from the chain of thought in the answer. For the o1 model series we show a model-generated summary of the chain of thought. So, let's recap. We went from: - Weights-available research prototype with full scientific documentation (GPT-2) - Commercial-scale model with API access only, full scientific documentation (GPT-3) - Even bigger API-only model, tuned for chain-of-thought reasoning, minimal documentation on the implementation (GPT-4, 4v, 4o) - An API-only model tuned to generate unedited chain-of-thought, which will not be shown to the user, even though it'd be really useful to have (o1)

dvt 2y ago
It's clear to me that OpenAI is quickly realizing they have no moat. Even this obfuscation of the chain-of-thought isn't really a moat. On top of CoT being pretty easy to implement and tweak, there's a serious push to on-device inference (which imo is the future), so the question is: will GPT-5 and beyond be really that much better than what we can run locally?
- falcor84 2y ago
  Based on their graphs of how quality scales well with compute cycles, I would expect that it would indeed continue to be that much better (unless you can afford the same compute locally).
  - darby_nine 2y ago
    Not much of a moat vs other private enterprise, though
- threatofrain 2y ago
  I don't see why on-device inference is the future. For consumers, only a small set of use cases cannot tolerate the increased latency. Corporate customers will be satisfied if the model can be hosted within their borders. Pooling compute is less wasteful overall as a collective strategy. This argument can really only meet its tipping point when massive models no longer offer a gotta-have-it difference vs smaller models.
  - unethical_ban 2y ago
    On-device inference will succeed the way Linux does: It is "free" in that it only requires the user to acquire a model to run vs. paying for processing. It protects privacy, and it doesn't require internet. It may not take over for all users, but it will be around. This assumes that openly developed (or at least weight-available) models are available for free, and continue being improved.
- phillipcarter 2y ago
  I don't understand the idea that they have no moat. Their moat is not technological. It's sociological. Most AI through APIs uses their models. Most consumer use of AI involves their models, or ChatGPT directly. They're clearly not in the "train your own model on your data in your environment" game, as that's a market for someone else. But make no mistake, they have a moat and it is strong.
  - dvt 2y ago
    > But make no mistake, they have a moat and it is strong. Given that Mistral, Llama, Claude, and even Gemini are competitive with (if not better than) OpenAI's flagships, I don't really think this is true.
    - crowcroft 2y ago
      Inertia is a hell of a moat. Everyone building is comfortable with OpenAI's API, and have an account. Competing models can't just be as good, they need to be MUCH better to be worth switching. Even as competitors build a sort of compatibility layer to be plug an play with OpenAI they will always be a step behind at best every time OpenAI releases a new feature.
      - chadash 2y ago
        Only a small fraction of all future AI projects have even gotten started. So they aren't only fighting over what's out there now, they're fighting over what will emerge.
        phillipcarter 2y ago
        This is true, and yet, many orgs who have experimented with OpenAI and are likely to return to them when a project "becomes real". When you google around online for how to do XYZ thing using LLMs, OpenAI is usually in whatever web results you read. Other models and APIs are also now using OpenAI's API format since it's the apparent winner. And for anyone who's already sent out subprocessor notifications with them as a vendor, they're locked in. This isn't to say it's only going to be an OpenAI market. Enterprise worlds move differently, such as those in G Cloud who will buy a few million $$ of Vertex expecting to "figure out that gemini stuff later". In that sense, Google has a moat with those slices of their customers. But I believe that when people think OpenAI has no moat because "the models will be a commodity", I think that's (a) some wishful thinking about the models and (b) doesn't consider the sociological factors that matter a lot more than how powerful a model is or where it runs.
- bgar 2y ago
  >there’s a serious push to on-device inference What push are you referring to? By whom?
- thih9 2y ago
  Why would a non profit / capped profit company, one that prioritizes public good, want a moat? Tongue in cheek.
- danenania 2y ago
  I wonder if they'll be able to push the chain-of-thought directly into the model. I'd imagine there could be some serious performance gains achievable if the model could "think" without doing IO on each cycle. In terms of moat, I think people underestimate how much of OpenAI's moat is based on operations and infrastructure rather than being purely based on model intelligence. As someone building on the API, it is by far the most reliable option out there currently. Claude Sonnet 3.5 is stronger on reasoning than gpt-4o but has a higher error rate, more errors conforming to a JSON schema, much lower rate limits, etc. These things are less important if you're just using the first-party chat interfaces but are very important if you're building on top of the APIs.
airstrike 2y ago
I think it's clear their strategy has changed. The whole landscape has changed. The size of models, amount of dollars, numbers of competitors and how much compute this whole exercise takes in the long term have all changed, so it's fair for them to adapt. It just so happens that they're keeping their old name. I think people focus too much on the "open" part of the name. I read "OpenAI" sort of like I read "Blackberry" or "Apple". I don't really think of fruits, I think of companies and their products.
beambot 2y ago
Very anti-open and getting less and less with each release. Rooting for Meta in this regard, at least.
mitthrowaway2 2y ago
> For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user.[...] Therefore we have decided not to show the raw chains of thought to users. Better not let the user see the part where the AI says "Next, let's manipulate the user by lying to them". It's for their own good, after all! We wouldn't want to make an unaligned chain of thought directly visible!
Buttons840 2y ago
I always laughed at the idea of a LLM Skynet "secretly" plotting to nuke humanity, while a bunch of humans watch it unfold before their eyes in plaintext. Now that seems less likely. At least OpenAI can see what it's thinking. A next step might be allowing the LLM to include non-text-based vectors in its internal thoughts, and then do all internal reasoning with raw vectors. Then the LLMs will have truly private thoughts in their own internal language. Perhaps we will use a LLM to interpret the secret thoughts of another LLM? This could be good or bad, but either way we're going to need more GPUs.
- hobo_in_library 2y ago
  "...either way we're going to need more GPUs." posted the LLM, rubbing it's virtual hands, cackling with delight as it prodded the humans to give it MOAR BRAINS
- navigate8310 2y ago
  At this point the G in GPU must be completely dropped
  - fragmede 2y ago
    Gen-ai Production Unit
- scotty79 2y ago
  > Now that seems less likely. At least OpenAI can see what it's thinking. When it's fully commercialized no one will be able to read through all chains of thoughts and with possibility of fine-tuning AI can learn to evade whatever tools openai will invent to flag concerning chains of thoughts if they interfere with providing the answer in some finetuning environment. Also at some point for the sake of efficiency and response quality they might migrate from chain of thought consisting of tokens into chain of thought consisting of full output network states and part of the network would have dedicated inputs for reading them.
- legionof7 2y ago
  >Perhaps we will use a LLM to interpret the secret thoughts of another LLM? this is a pretty active area of research with sparse autoencoders
lossolo 2y ago
It's because there is nothing novel here from an architectural point of view. Again, the secret sauce is only in the training data. O1 seems like a variant of RLRF https://arxiv.org/abs/2403.14238 https://arxiv.org/abs/2403.14238 Soon you will see similar models from competitors.
lolinder 2y ago
The hidden chain of thought tokens are also billed as output tokens, so you still pay for them even though they're not going to let you see them: > While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens. https://platform.openai.com/docs/guides/reasoning https://platform.openai.com/docs/guides/reasoning
drooby 2y ago
Did OpenAI ever even claim that they would be an open source company? It seems like their driving mission has always been to create AI that is the "most beneficial to society".. which might come in many different flavors.. including closed source.
- josu 2y ago
  Kind of? >We're hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We'll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. https://web.archive.org/web/20160220125157/https://www.openai.com/blog/introducing-openai/ https://web.archive.org/web/20160220125157/https://www.opena...
- lolinder 2y ago
  > Because of AI’s surprising history, it’s hard to predict when human-level AI might come within reach. When it does, it’ll be important to have a leading research institution which can prioritize a good outcome for all over its own self-interest. > We’re hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. I don't see much evidence that the OpenAI that exists now—after Altman's ousting, his return, and the ousting of those who ousted him—has any interest in mind besides its own. https://openai.com/index/introducing-openai/ https://openai.com/index/introducing-openai/
- leetharris 2y ago
  https://web.archive.org/web/20190224031626/https://blog.openai.com/introducing-openai/ https://web.archive.org/web/20190224031626/https://blog.open... > Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. From their very own website. Of course they deleted it as soon as Altman took over and turned it into a for profit, closed company.
ec109685 2y ago
Given the chain of thought is sitting in the context, I'm sure someone enterprising will find a way to extract it via a jailbreak (despite it being better at preventing jailbreaks).
deleted 2y ago
[deleted]

dvt 2y ago

It's clear to me that OpenAI is quickly realizing they have no moat. Even this obfuscation of the chain-of-thought isn't really a moat. On top of CoT being pretty easy to implement and tweak, there's a serious push to on-device inference (which imo is the future), so the question is: will GPT-5 and beyond be really that much better than what we can run locally?

falcor84 2y ago
Based on their graphs of how quality scales well with compute cycles, I would expect that it would indeed continue to be that much better (unless you can afford the same compute locally).
- darby_nine 2y ago
  Not much of a moat vs other private enterprise, though
threatofrain 2y ago
I don't see why on-device inference is the future. For consumers, only a small set of use cases cannot tolerate the increased latency. Corporate customers will be satisfied if the model can be hosted within their borders. Pooling compute is less wasteful overall as a collective strategy. This argument can really only meet its tipping point when massive models no longer offer a gotta-have-it difference vs smaller models.
- unethical_ban 2y ago
  On-device inference will succeed the way Linux does: It is "free" in that it only requires the user to acquire a model to run vs. paying for processing. It protects privacy, and it doesn't require internet. It may not take over for all users, but it will be around. This assumes that openly developed (or at least weight-available) models are available for free, and continue being improved.
phillipcarter 2y ago
I don't understand the idea that they have no moat. Their moat is not technological. It's sociological. Most AI through APIs uses their models. Most consumer use of AI involves their models, or ChatGPT directly. They're clearly not in the "train your own model on your data in your environment" game, as that's a market for someone else. But make no mistake, they have a moat and it is strong.
- dvt 2y ago
  > But make no mistake, they have a moat and it is strong. Given that Mistral, Llama, Claude, and even Gemini are competitive with (if not better than) OpenAI's flagships, I don't really think this is true.
  - crowcroft 2y ago
    Inertia is a hell of a moat. Everyone building is comfortable with OpenAI's API, and have an account. Competing models can't just be as good, they need to be MUCH better to be worth switching. Even as competitors build a sort of compatibility layer to be plug an play with OpenAI they will always be a step behind at best every time OpenAI releases a new feature.
    - chadash 2y ago
      Only a small fraction of all future AI projects have even gotten started. So they aren't only fighting over what's out there now, they're fighting over what will emerge.
      - phillipcarter 2y ago
        This is true, and yet, many orgs who have experimented with OpenAI and are likely to return to them when a project "becomes real". When you google around online for how to do XYZ thing using LLMs, OpenAI is usually in whatever web results you read. Other models and APIs are also now using OpenAI's API format since it's the apparent winner. And for anyone who's already sent out subprocessor notifications with them as a vendor, they're locked in. This isn't to say it's only going to be an OpenAI market. Enterprise worlds move differently, such as those in G Cloud who will buy a few million $$ of Vertex expecting to "figure out that gemini stuff later". In that sense, Google has a moat with those slices of their customers. But I believe that when people think OpenAI has no moat because "the models will be a commodity", I think that's (a) some wishful thinking about the models and (b) doesn't consider the sociological factors that matter a lot more than how powerful a model is or where it runs.
bgar 2y ago
>there’s a serious push to on-device inference What push are you referring to? By whom?
thih9 2y ago
Why would a non profit / capped profit company, one that prioritizes public good, want a moat? Tongue in cheek.
danenania 2y ago
I wonder if they'll be able to push the chain-of-thought directly into the model. I'd imagine there could be some serious performance gains achievable if the model could "think" without doing IO on each cycle. In terms of moat, I think people underestimate how much of OpenAI's moat is based on operations and infrastructure rather than being purely based on model intelligence. As someone building on the API, it is by far the most reliable option out there currently. Claude Sonnet 3.5 is stronger on reasoning than gpt-4o but has a higher error rate, more errors conforming to a JSON schema, much lower rate limits, etc. These things are less important if you're just using the first-party chat interfaces but are very important if you're building on top of the APIs.

airstrike 2y ago

I think it's clear their strategy has changed. The whole landscape has changed. The size of models, amount of dollars, numbers of competitors and how much compute this whole exercise takes in the long term have all changed, so it's fair for them to adapt. It just so happens that they're keeping their old name. I think people focus too much on the "open" part of the name. I read "OpenAI" sort of like I read "Blackberry" or "Apple". I don't really think of fruits, I think of companies and their products.

beambot 2y ago

Very anti-open and getting less and less with each release. Rooting for Meta in this regard, at least.

mitthrowaway2 2y ago

> For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user.[...] Therefore we have decided not to show the raw chains of thought to users. Better not let the user see the part where the AI says "Next, let's manipulate the user by lying to them". It's for their own good, after all! We wouldn't want to make an unaligned chain of thought directly visible!

Buttons840 2y ago

I always laughed at the idea of a LLM Skynet "secretly" plotting to nuke humanity, while a bunch of humans watch it unfold before their eyes in plaintext. Now that seems less likely. At least OpenAI can see what it's thinking. A next step might be allowing the LLM to include non-text-based vectors in its internal thoughts, and then do all internal reasoning with raw vectors. Then the LLMs will have truly private thoughts in their own internal language. Perhaps we will use a LLM to interpret the secret thoughts of another LLM? This could be good or bad, but either way we're going to need more GPUs.

hobo_in_library 2y ago
"...either way we're going to need more GPUs." posted the LLM, rubbing it's virtual hands, cackling with delight as it prodded the humans to give it MOAR BRAINS
navigate8310 2y ago
At this point the G in GPU must be completely dropped
- fragmede 2y ago
  Gen-ai Production Unit
scotty79 2y ago
> Now that seems less likely. At least OpenAI can see what it's thinking. When it's fully commercialized no one will be able to read through all chains of thoughts and with possibility of fine-tuning AI can learn to evade whatever tools openai will invent to flag concerning chains of thoughts if they interfere with providing the answer in some finetuning environment. Also at some point for the sake of efficiency and response quality they might migrate from chain of thought consisting of tokens into chain of thought consisting of full output network states and part of the network would have dedicated inputs for reading them.
legionof7 2y ago
>Perhaps we will use a LLM to interpret the secret thoughts of another LLM? this is a pretty active area of research with sparse autoencoders

lossolo 2y ago

It's because there is nothing novel here from an architectural point of view. Again, the secret sauce is only in the training data. O1 seems like a variant of RLRF https://arxiv.org/abs/2403.14238 https://arxiv.org/abs/2403.14238 Soon you will see similar models from competitors.

lolinder 2y ago

The hidden chain of thought tokens are also billed as output tokens, so you still pay for them even though they're not going to let you see them: > While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens. https://platform.openai.com/docs/guides/reasoning https://platform.openai.com/docs/guides/reasoning

drooby 2y ago

Did OpenAI ever even claim that they would be an open source company? It seems like their driving mission has always been to create AI that is the "most beneficial to society".. which might come in many different flavors.. including closed source.

josu 2y ago
Kind of? >We're hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We'll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. https://web.archive.org/web/20160220125157/https://www.openai.com/blog/introducing-openai/ https://web.archive.org/web/20160220125157/https://www.opena...
lolinder 2y ago
> Because of AI’s surprising history, it’s hard to predict when human-level AI might come within reach. When it does, it’ll be important to have a leading research institution which can prioritize a good outcome for all over its own self-interest. > We’re hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. I don't see much evidence that the OpenAI that exists now—after Altman's ousting, his return, and the ousting of those who ousted him—has any interest in mind besides its own. https://openai.com/index/introducing-openai/ https://openai.com/index/introducing-openai/
leetharris 2y ago
https://web.archive.org/web/20190224031626/https://blog.openai.com/introducing-openai/ https://web.archive.org/web/20190224031626/https://blog.open... > Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. From their very own website. Of course they deleted it as soon as Altman took over and turned it into a for profit, closed company.

ec109685 2y ago

Given the chain of thought is sitting in the context, I'm sure someone enterprising will find a way to extract it via a jailbreak (despite it being better at preventing jailbreaks).

deleted 2y ago

[deleted]