4 ms·

So, leaving aside the idea that OA might've used data from the researchers Codex sessions: Do I understand correctly that the internal OpenAI work on the proble

by Semkas 24d ago

So, leaving aside the idea that OA might've used data from the researchers Codex sessions: Do I understand correctly that the internal OpenAI work on the problems was probably started after they heard Alpoge and Buckmaster had made process by using their models? And they used the publicly available info about the researchers past work to prompt their models? If compute is cheap, and the difficult thing with scientific discovery is now mostly in steering agents into promising areas, there's an obvious incentive for OA mathematicians to simply monitor closely which researchers are close to releasing exciting results, make some assumptions about their prompts based on their past work, and quickly prompt their own (stronger) model to look into the same areas.

AJRF 24d ago
> leaving aside the idea that OA might've used data from the researchers Codex sessions Why leave that aside? That is _the_ story. If a Chinese research lab did this we'd call it espionage.
- modeless 24d ago
  Because they didn't do that. Tristan doesn't specifically claim that they did, and Anthropic employees don't think they did either. https://x.com/_sholtodouglas/status/2097218240397410733 https://x.com/_sholtodouglas/status/2097218240397410733
  - Semkas 24d ago
    By default OA trains their models on codex-sessions. If I understand him correctly this is something Tristan explicitly mentions in his post as a possible reason for the fast results obtained by the internal OA team. Anthropic obviously doesn't want to challenge the idea that training is transformative, even if it means agreeing with their competitor.
  - FiberBundle 24d ago
    Well, of course Anthropic employees would say that, since they likely do the same. Claiming that your primary competitor doesn't engage in a certain malicious practice is supposed to make it look as if there's no way you would too. If somebody even says that about their competitor, then surely there must be truth to that, otherwise you would never give credit to someone you're opposed to.
  - hodgehog11 24d ago
    It's literally a toggle in the options for ChatGPT, one which is on by default and most researchers probably have on without realising it. So to say that it is unlikely is extremely suspicious. No, they did not literally pull user data. But user data is automatically added to their training set by default, so their latest in-house model would be trained on it if it is from several months ago. It isn't intentional on their part, and they probably realised they could not refute that they trained on Tristan's logs unintentionally, hence why they acted the way they did.
  - hellohello2 23d ago
    Its very easy for OpenAI to answer, yes or no, if the model they used trained on their chats.
- Semkas 24d ago
  But there's a bunch of people already in this thread calling that stuff unfounded speculation (which I disagree with), and my point is that even if that specific thing isn't true, OA's behavior here is obviously awful. If they're going to try to beat researchers to discoveries like this it disincentives researchers to talk about their progress publicly, and basically breaks the ecosystem of scientific cooperation / discovery. It's also immoral.
  - deleted 24d ago
    [deleted]
  - reasonableklout 24d ago
    Yep. The most uncharitable view of this might be: they stole the work of researchers to build their models, and now they're using said models to steal the proceeds of future work, too.
- paxys 24d ago
  Why is that the story? Is there anything to back it up beyond a single accusation?
  - defmacr0 24d ago
    As a prior I would say that a math professor has about infinite times more integrity than OpenAI.
- light_hue_1 24d ago
  If you think that OpenAI won't look at your data to gain a massive advantage, you're naive.
ummonk 24d ago
This is an excellent point...
stabbles 24d ago
Similar to https://news.ycombinator.com/item?id=47566442 https://news.ycombinator.com/item?id=47566442
ehhthing 24d ago
I’ve thought a lot about publishing research and wanting to do more of it, but right as I finally had the time and energy to start writing articles LLMs start to take off. Now all of a sudden, I’m acutely aware that everything I publish will be used for AI training. For math, a field that is built on incremental research it feels like AI labs will do nothing but discourage publishing research at all for fear that they will be able to spend the money for compute that publicly funded academia simply cannot afford. It feels like publishing anything at this point just means that your work will be fed to a machine that will make sure your work will never been seen by anyone else because it will always be the ones making the “true advancements”. Perhaps I’d feel better about this if AI labs really existed for humanity’s benefit, but for some reason I don’t think that comes up in their investor slide decks.
Davidzheng 24d ago
It's honestly unsurprising and not a problem that they do this in my view. The problem really starts when you start taking credit for work that they would've achieved. Like if i go to a talk on unfinished work, it's not really unethical for me to think about the problem--it's a problem if i scoop the authors but these problems can often be solved by collaboration or proper crediting and timing--IN MY VIEW
- softwaredoug 23d ago
  The difference is how credit and attribution works. And whether we feel it’s being laundered through models. And also whether the AI moon laser pointed at your problem is just going to be the thing that writes the final conclusion on ten years of your work.
octoberfranklin 23d ago
Exactly the same thing that's been happening with vulnerabilities and bugfixes the past few months. I fear that AI is going to cause ossifying secrecy in many fields, much like what happened semiconductor design the past 10-15 years.
heaney-555 23d ago
>If compute is cheap This cost millions of dollars of tokens.

by Semkas 24d ago

AJRF 24d ago

> leaving aside the idea that OA might've used data from the researchers Codex sessions Why leave that aside? That is _the_ story. If a Chinese research lab did this we'd call it espionage.

modeless 24d ago
Because they didn't do that. Tristan doesn't specifically claim that they did, and Anthropic employees don't think they did either. https://x.com/_sholtodouglas/status/2097218240397410733 https://x.com/_sholtodouglas/status/2097218240397410733
- Semkas 24d ago
  By default OA trains their models on codex-sessions. If I understand him correctly this is something Tristan explicitly mentions in his post as a possible reason for the fast results obtained by the internal OA team. Anthropic obviously doesn't want to challenge the idea that training is transformative, even if it means agreeing with their competitor.
- FiberBundle 24d ago
  Well, of course Anthropic employees would say that, since they likely do the same. Claiming that your primary competitor doesn't engage in a certain malicious practice is supposed to make it look as if there's no way you would too. If somebody even says that about their competitor, then surely there must be truth to that, otherwise you would never give credit to someone you're opposed to.
- hodgehog11 24d ago
  It's literally a toggle in the options for ChatGPT, one which is on by default and most researchers probably have on without realising it. So to say that it is unlikely is extremely suspicious. No, they did not literally pull user data. But user data is automatically added to their training set by default, so their latest in-house model would be trained on it if it is from several months ago. It isn't intentional on their part, and they probably realised they could not refute that they trained on Tristan's logs unintentionally, hence why they acted the way they did.
- hellohello2 23d ago
  Its very easy for OpenAI to answer, yes or no, if the model they used trained on their chats.
Semkas 24d ago
But there's a bunch of people already in this thread calling that stuff unfounded speculation (which I disagree with), and my point is that even if that specific thing isn't true, OA's behavior here is obviously awful. If they're going to try to beat researchers to discoveries like this it disincentives researchers to talk about their progress publicly, and basically breaks the ecosystem of scientific cooperation / discovery. It's also immoral.
- deleted 24d ago
  [deleted]
- reasonableklout 24d ago
  Yep. The most uncharitable view of this might be: they stole the work of researchers to build their models, and now they're using said models to steal the proceeds of future work, too.
paxys 24d ago
Why is that the story? Is there anything to back it up beyond a single accusation?
- defmacr0 24d ago
  As a prior I would say that a math professor has about infinite times more integrity than OpenAI.
light_hue_1 24d ago
If you think that OpenAI won't look at your data to gain a massive advantage, you're naive.

ummonk 24d ago

This is an excellent point...

stabbles 24d ago

Similar to https://news.ycombinator.com/item?id=47566442 https://news.ycombinator.com/item?id=47566442

ehhthing 24d ago

I’ve thought a lot about publishing research and wanting to do more of it, but right as I finally had the time and energy to start writing articles LLMs start to take off. Now all of a sudden, I’m acutely aware that everything I publish will be used for AI training. For math, a field that is built on incremental research it feels like AI labs will do nothing but discourage publishing research at all for fear that they will be able to spend the money for compute that publicly funded academia simply cannot afford. It feels like publishing anything at this point just means that your work will be fed to a machine that will make sure your work will never been seen by anyone else because it will always be the ones making the “true advancements”. Perhaps I’d feel better about this if AI labs really existed for humanity’s benefit, but for some reason I don’t think that comes up in their investor slide decks.

Davidzheng 24d ago

It's honestly unsurprising and not a problem that they do this in my view. The problem really starts when you start taking credit for work that they would've achieved. Like if i go to a talk on unfinished work, it's not really unethical for me to think about the problem--it's a problem if i scoop the authors but these problems can often be solved by collaboration or proper crediting and timing--IN MY VIEW

softwaredoug 23d ago
The difference is how credit and attribution works. And whether we feel it’s being laundered through models. And also whether the AI moon laser pointed at your problem is just going to be the thing that writes the final conclusion on ten years of your work.

octoberfranklin 23d ago

Exactly the same thing that's been happening with vulnerabilities and bugfixes the past few months. I fear that AI is going to cause ossifying secrecy in many fields, much like what happened semiconductor design the past 10-15 years.

heaney-555 23d ago

>If compute is cheap This cost millions of dollars of tokens.