5 ms·

I'm not surprised-- I recently suggested that someone might try to pull an Aaron Swartz with the LLAMA weights (i.e., release them in an uncontrolled way simila

I'm not surprised-- I recently suggested that someone might try to pull an Aaron Swartz with the LLAMA weights (i.e., release them in an uncontrolled way similar to how Aaron attempted to release the JSTOR database). It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. If the paper is to be believed, this is a major development, allowing you to get close to GPT3 performance on a single GPU (at least for inference on the smallest model). Clearly some renegade academic feels the same way.

sebzim4500 4y ago
>Clearly some renegade academic feels the same way. Or someone pretending to be a renegade academic. It's not like there is a KYC process.
catchnear4321 4y ago
> It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. I mean at least they didn’t pick a name that heavily implied they were, are, and always will be open. Then do the opposite. You know, like OpenAI? So now we got some weights I guess.
vintermann 4y ago
It was already the most open language model in its class, given that the code for training and inference was available and it only used public data for training. For Google and OpenAIs offerings, have fun reimplementing it from descriptions in the paper (including small crucial details that they may have left out), training it for a month, and then wondering if the implementation or the training data is the reason your model isn't as good as theirs.
- ImprobableTruth 4y ago
  The training code is not available.
  - renonce 4y ago
    Weights are more valuable than training code in one regard. Even with the training code you may not have the dataset and reproduction requires a massive GPU cluster that few can afford.
    - eigenvalue 4y ago
      Weights are more valuable to random individuals who want to mess around with the model. Training code is more valuable to other companies that have the resources to use them, because then they can tweak/modify however they want. But even then, you still need the training data, which in the case of OpenAI and DeepMind is a big part of the secret sauce (not just the raw data but also the process for cleansing and de-duplicating it).
  - glomgril 4y ago
    Training code won't get you much if you don't have the infra/money to gather a suitable dataset or actually execute training. Plus if your goal is to "steal" or riff on the base model, it's already there in the weights. Also probably not difficult to figure out how to fine tune it once you have the weights and tokenizer.
Nimitz14 4y ago
Probably done with meta's blessing. Pretty obvious the scientists wanted to release it and legal stopped them.
kragen 4y ago
if aaron had attempted to release the jstor database he would have succeeded
- eigenvalue 4y ago
  Didn't he get caught before he fully downloaded everything? My understanding was that's what he was doing with that laptop stuffed in a supply closet on the MIT campus.
  - afunctionof 4y ago
    He could have been trying to evaluate how many public domain papers it wasn't sharing access to at the time, which would have required a collection of the database. Would also have explained why he wasn't trying to hide his behavior. He would have succeeded if he was just wanting to deaddrop all the data in a bin, probably without getting caught.
  - kragen 4y ago
    probably nobody knows but certainly he was capable of ensuring that everything he downloaded would be released if that was what he wanted
    - afunctionof 4y ago
      You should add your Matrix to your contact page. Edit: Also, is there going to be another der* release this year? They and therefore you have taught me more than I'd learned in the last five years otherwise.
yunwal 4y ago
> It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. Facebook almost certainly knew leaks would happen. My guess is keeping the model "contained" was a legal shield more than anything else, to protect themselves from liability in the case someone misuses the model.
deleted 4y ago
[deleted]
cubefox 4y ago
"It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics." I don't think it's misleading. Even saying that Facebook "hoards" the weights when they are more open than any other major AI company is baffling.
make3 4y ago
it's really easy to get access to the weights, I got access to them as a master's student with just a single very minor publication in the domain, and basically all my colleagues who applied did too. there is no Aaron Schwartz thing here, a huge amount of of people will have them, someone was going to leak for sure

sebzim4500 4y ago

>Clearly some renegade academic feels the same way. Or someone pretending to be a renegade academic. It's not like there is a KYC process.

catchnear4321 4y ago

> It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. I mean at least they didn’t pick a name that heavily implied they were, are, and always will be open. Then do the opposite. You know, like OpenAI? So now we got some weights I guess.

vintermann 4y ago

It was already the most open language model in its class, given that the code for training and inference was available and it only used public data for training. For Google and OpenAIs offerings, have fun reimplementing it from descriptions in the paper (including small crucial details that they may have left out), training it for a month, and then wondering if the implementation or the training data is the reason your model isn't as good as theirs.

ImprobableTruth 4y ago
The training code is not available.
- renonce 4y ago
  Weights are more valuable than training code in one regard. Even with the training code you may not have the dataset and reproduction requires a massive GPU cluster that few can afford.
  - eigenvalue 4y ago
    Weights are more valuable to random individuals who want to mess around with the model. Training code is more valuable to other companies that have the resources to use them, because then they can tweak/modify however they want. But even then, you still need the training data, which in the case of OpenAI and DeepMind is a big part of the secret sauce (not just the raw data but also the process for cleansing and de-duplicating it).
- glomgril 4y ago
  Training code won't get you much if you don't have the infra/money to gather a suitable dataset or actually execute training. Plus if your goal is to "steal" or riff on the base model, it's already there in the weights. Also probably not difficult to figure out how to fine tune it once you have the weights and tokenizer.

Nimitz14 4y ago

Probably done with meta's blessing. Pretty obvious the scientists wanted to release it and legal stopped them.

kragen 4y ago

if aaron had attempted to release the jstor database he would have succeeded

eigenvalue 4y ago
Didn't he get caught before he fully downloaded everything? My understanding was that's what he was doing with that laptop stuffed in a supply closet on the MIT campus.
- afunctionof 4y ago
  He could have been trying to evaluate how many public domain papers it wasn't sharing access to at the time, which would have required a collection of the database. Would also have explained why he wasn't trying to hide his behavior. He would have succeeded if he was just wanting to deaddrop all the data in a bin, probably without getting caught.
- kragen 4y ago
  probably nobody knows but certainly he was capable of ensuring that everything he downloaded would be released if that was what he wanted
  - afunctionof 4y ago
    You should add your Matrix to your contact page. Edit: Also, is there going to be another der* release this year? They and therefore you have taught me more than I'd learned in the last five years otherwise.

yunwal 4y ago

> It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. Facebook almost certainly knew leaks would happen. My guess is keeping the model "contained" was a legal shield more than anything else, to protect themselves from liability in the case someone misuses the model.

deleted 4y ago

[deleted]

cubefox 4y ago

"It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics." I don't think it's misleading. Even saying that Facebook "hoards" the weights when they are more open than any other major AI company is baffling.

make3 4y ago

it's really easy to get access to the weights, I got access to them as a master's student with just a single very minor publication in the domain, and basically all my colleagues who applied did too. there is no Aaron Schwartz thing here, a huge amount of of people will have them, someone was going to leak for sure