4 ms·

>> An obvious thing to do would be to either open-source older models (including the weights) when retiring them; or possibly transfer them to an institution wh

>> An obvious thing to do would be to either open-source older models (including the weights) when retiring them; or possibly transfer them to an institution who see their role specifically as serving as an archive Another obvious thing to do is do your research on non-commercial or open source things that can not be taken away from you. Sorry, I don't mean for the snark present in that statement. The frustration lies with the company and others that tend to pull rugs.

UncleEntity 4y ago
Exactly, expecting a company to maintain a project forever (or give it away) just because they were used in research projects is ludicrous. Maybe they did the research when “open” in their name meant something but it has been obvious for quite a while that ship has sailed.
- red1reaper 4y ago
  Expecting a company to maintain a project forever or give it away just because they were used in research projects is actually ver reasonable.
  - UncleEntity 4y ago
    Very reasonable if you’re the one doing the research but not if you’re the one subsidizing the maintenance of an obsolete project. Open sourcing it is its own can of worms because they may be using third-party code they can’t release or, you know, &etc.
z3c0 4y ago
I thought I must be going crazy until I saw your comment. This sounds like a bad research practice that probably shouldn't be reproduced to begin with.
- itsaquicknote 4y ago
  Research into systemically important infrastructure cannot be damned because that infrastructure isn't public. It's a cheap moralizing argument to say "pfff, this was predictable". Maybe so, but there isn't an alternative. Much like research on Twitter. Once these companies start to drift into providing what become broadscale social utilities and public services it doesn't matter that they're private. There are(/should be) obligations that come with that. You can't handwave and say go do your research on some micro-niche open source project that's way behind the SOTA and has nowhere near the same reach. That's not what "best practice" means here.
  - dahart 4y ago
    Sure but OpenAI isn’t preventing research. It’s not their responsibility to provide reproducibility, at their expense, for any researchers looking at GPT, that job is the responsibility of the researchers, and the researchers still can work. It might be unfortunate from their perspective that there used to be a nice tool that makes their job easier, but the flip side here is that OpenAI didn’t say why they’re removing access to codex, and they probably have good reasons, not least of which is it costs them money that researchers aren’t subsidizing.
  - z3c0 4y ago
    I'm going to be frank here, because I know my argument isn't "cheap". When one utilizes OSINT techniques (which using an ML service hosted by a third-party certainly qualifies as), there are baked-in assumptions that 1) this source could go away at any time, and 2) the source is only a reflection of the interests of the third-party, not something to be taken at face value. No 2 can certainly be the subject of research, but to do so without accounting for No 1 would indicate bad research practices from the jump. For example, they could have (and should have) been snapshotting the outputs, tagged with versions & dates. By the sound of it, the outputs weren't even the subject of research, but were instead propping up the research. That flies in the face of No 2 as well. Let them start over, with better methodology this time.
  - itsaquicknote 4y ago
    Replying to both responses because they're all good points. My argument boils down to the fact that some private companies end up becoming social utilities and once that happens, the rules (should) change as part of the social contract which means, yeah, they can't simply "pull the rug". The research is important precisely because its into systemically significant systems. I get that it's difficult to define the line where that gets crossed. But the idea to provide a publicly funded trust that manages legacy versions of things like this is not a bad idea.

UncleEntity 4y ago

Exactly, expecting a company to maintain a project forever (or give it away) just because they were used in research projects is ludicrous. Maybe they did the research when “open” in their name meant something but it has been obvious for quite a while that ship has sailed.

red1reaper 4y ago
Expecting a company to maintain a project forever or give it away just because they were used in research projects is actually ver reasonable.
- UncleEntity 4y ago
  Very reasonable if you’re the one doing the research but not if you’re the one subsidizing the maintenance of an obsolete project. Open sourcing it is its own can of worms because they may be using third-party code they can’t release or, you know, &etc.

z3c0 4y ago

I thought I must be going crazy until I saw your comment. This sounds like a bad research practice that probably shouldn't be reproduced to begin with.

itsaquicknote 4y ago
Research into systemically important infrastructure cannot be damned because that infrastructure isn't public. It's a cheap moralizing argument to say "pfff, this was predictable". Maybe so, but there isn't an alternative. Much like research on Twitter. Once these companies start to drift into providing what become broadscale social utilities and public services it doesn't matter that they're private. There are(/should be) obligations that come with that. You can't handwave and say go do your research on some micro-niche open source project that's way behind the SOTA and has nowhere near the same reach. That's not what "best practice" means here.
- dahart 4y ago
  Sure but OpenAI isn’t preventing research. It’s not their responsibility to provide reproducibility, at their expense, for any researchers looking at GPT, that job is the responsibility of the researchers, and the researchers still can work. It might be unfortunate from their perspective that there used to be a nice tool that makes their job easier, but the flip side here is that OpenAI didn’t say why they’re removing access to codex, and they probably have good reasons, not least of which is it costs them money that researchers aren’t subsidizing.
- z3c0 4y ago
  I'm going to be frank here, because I know my argument isn't "cheap". When one utilizes OSINT techniques (which using an ML service hosted by a third-party certainly qualifies as), there are baked-in assumptions that 1) this source could go away at any time, and 2) the source is only a reflection of the interests of the third-party, not something to be taken at face value. No 2 can certainly be the subject of research, but to do so without accounting for No 1 would indicate bad research practices from the jump. For example, they could have (and should have) been snapshotting the outputs, tagged with versions & dates. By the sound of it, the outputs weren't even the subject of research, but were instead propping up the research. That flies in the face of No 2 as well. Let them start over, with better methodology this time.
- itsaquicknote 4y ago
  Replying to both responses because they're all good points. My argument boils down to the fact that some private companies end up becoming social utilities and once that happens, the rules (should) change as part of the social contract which means, yeah, they can't simply "pull the rug". The research is important precisely because its into systemically significant systems. I get that it's difficult to define the line where that gets crossed. But the idea to provide a publicly funded trust that manages legacy versions of things like this is not a bad idea.