3 ms·

Apples and oranges comparison. You couldn’t get the same participants, but you could get the same drugs. If you could get identical participants, that wouldn’t

by dartos 3y ago

Apples and oranges comparison. You couldn’t get the same participants, but you could get the same drugs. If you could get identical participants, that wouldn’t be very helpful since humans are so varied. But for GPT based papers, what you’re actually testing could change without you knowing. There’s no way to know if a paper is reproducible at all. If you can’t reproduce results, is it really research, or just show and tell?

mewpmewp2 3y ago
With API you can choose versions fixed to a date. Are you suggesting that OpenAI is lying about these being fixed to a date? Why would they lie about it? The whole point of these versions is so that when you build on top of that it would keep working as you expect.
- hnfong 3y ago
  I'm not saying they lie about it, but as a hypothetical there could be many reasons to lie. - realizing their model leaks confidential information against malicious prompts - copyright claims against them forcing them to remove bits of data from the training set - serious "alignment" bugs that need to be fixed - vastly improved optimization techniques that slightly affect results in 0.1% of the cases If updating the model would save the company a couple hundred million dollars, they might want to do it. And in some of the cases, I can imagine they have an incentive to keep the update low key.
IanCal 3y ago
> If you can’t reproduce results, is it really research, or just show and tell? You can't start with a statement about clinical trials not being perfectly reproducible and that's fine, then say this. > what you’re actually testing could change without you knowing If people are lying about an extremely important part of their product, which they have little reason to. But then this applies to pretty much everything. Starting with the assumption that people are lying about everything and nothing is as it seems may technically make things more reproducible but it's going to require unbelievable effort for very little return. > There’s no way to know if a paper is reproducible at all. This is a little silly because these models are available extremely easily and at a pay-as-you-go pricing. And again, it requires an assumption that openai is lying about a specific feature of a product.
- starbugs 3y ago
  > You can't start with a statement about clinical trials not being perfectly reproducible and that's fine, then say this. Nobody said that to begin with. Re-read their comment. > If people are lying about an extremely important part of their product [...] Nobody is alleging that anyone is lying. It's just that we cannot be sure what the research actually refers to, because of the nature of a proprietary/closed model. > This is a little silly because these models are available extremely easily and at a pay-as-you-go pricing. What does this have to do with the parent comment? I don't think it's appropriate to call anyone here silly, just because you don't like their comment and don't have good counter arguments.
  - IanCal 3y ago
    > Nobody is alleging that anyone is lying. Let's be clear, you have made an explicit claim that openai are lying. > What does this have to do with the parent comment? Because many other fields would kill for this level of reproducibility, grab an API key, spend a few quid running a script and you can get the results yourself.
    - deleted 3y ago
      [deleted]

by dartos 3y ago

mewpmewp2 3y ago

With API you can choose versions fixed to a date. Are you suggesting that OpenAI is lying about these being fixed to a date? Why would they lie about it? The whole point of these versions is so that when you build on top of that it would keep working as you expect.

hnfong 3y ago
I'm not saying they lie about it, but as a hypothetical there could be many reasons to lie. - realizing their model leaks confidential information against malicious prompts - copyright claims against them forcing them to remove bits of data from the training set - serious "alignment" bugs that need to be fixed - vastly improved optimization techniques that slightly affect results in 0.1% of the cases If updating the model would save the company a couple hundred million dollars, they might want to do it. And in some of the cases, I can imagine they have an incentive to keep the update low key.

IanCal 3y ago

> If you can’t reproduce results, is it really research, or just show and tell? You can't start with a statement about clinical trials not being perfectly reproducible and that's fine, then say this. > what you’re actually testing could change without you knowing If people are lying about an extremely important part of their product, which they have little reason to. But then this applies to pretty much everything. Starting with the assumption that people are lying about everything and nothing is as it seems may technically make things more reproducible but it's going to require unbelievable effort for very little return. > There’s no way to know if a paper is reproducible at all. This is a little silly because these models are available extremely easily and at a pay-as-you-go pricing. And again, it requires an assumption that openai is lying about a specific feature of a product.

starbugs 3y ago
> You can't start with a statement about clinical trials not being perfectly reproducible and that's fine, then say this. Nobody said that to begin with. Re-read their comment. > If people are lying about an extremely important part of their product [...] Nobody is alleging that anyone is lying. It's just that we cannot be sure what the research actually refers to, because of the nature of a proprietary/closed model. > This is a little silly because these models are available extremely easily and at a pay-as-you-go pricing. What does this have to do with the parent comment? I don't think it's appropriate to call anyone here silly, just because you don't like their comment and don't have good counter arguments.
- IanCal 3y ago
  > Nobody is alleging that anyone is lying. Let's be clear, you have made an explicit claim that openai are lying. > What does this have to do with the parent comment? Because many other fields would kill for this level of reproducibility, grab an API key, spend a few quid running a script and you can get the results yourself.
  - deleted 3y ago
    [deleted]