4 ms·

I've looked a few times, but have not seen any research on assigning provenance to the weights used in a particular inference run. It's a super interesting spac

I've looked a few times, but have not seen any research on assigning provenance to the weights used in a particular inference run. It's a super interesting space for a bunch of reasons. But the naive approach of having a table of how much each individual training item influenced every weight in the model seems impossibly big. For DALL-E 2's 6.5B parameters and 650m training items, that's 4.2 quadrillion associations. And then you have to figure out which weights contributed the most to an output. I would love to see any research or even just thinking that anyone's done on this topic. It seems like it will be important in the future, but it also seems like a crazy difficult scale problem as models get bigger.

bash-j 4y ago
Could you not use tags used to label the image? If your image contains more niche tags that match the user input, your revenue share will be higher. Depending how much extra people earn for certain tags, it might incentivise people to upload more images of what is missing from the training data.
- brookst 4y ago
  That's interesting, but I'm not sure it works. I think that works out to "for any given prompt, distribute credit to every source image that has a keyword that appears in the prompt, proportional to how many other source images had that same keyword". If I include the tag "floor", do I get some (tiny) percentage of every image that uses "floor" in the prompt, even if the bits from my image did not end up affecting model weights much at all in training? Worse, for tags like "dramatic lighting", it's likely that the important source images will depend on the other words in the prompt; "sunset, dramatic lighting" will probably not use the rely on the same weights or source images as "theater interior, dramatic lighting". And then you get the perverse incentives to tag every image with every possible tag :) I'd love to be convinced otherwise, but I'm not seeing prompt-to-tag association working.
  - bash-j 4y ago
    The tags could be added by a model rather than the user submitting the image. Maybe do both and verify the tags with a model? Users could get a rating based on how reliably they tag their pictures and are trusted to add more niche tags at higher ratings. You could even help tag other pictures to improve your rating.
gradys 4y ago
https://arxiv.org/abs/1703.04730 https://arxiv.org/abs/1703.04730 > How can we explain the predictions of a black-box model? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction.
- brookst 4y ago
  Oh thank you! Will go read and digest.
- batch12 4y ago
  This would be an interesting way to distill/finetune a model by finding the optimal minimum corpus needed to match a given use case.
natch 4y ago
> And then you have to figure out which weights contributed the most to an output. Why do you have to, though? What do you hope to trace back to exactly?
- brookst 4y ago
  The business problem is “which items from the training corpus contributes the most to this inference”

bash-j 4y ago

Could you not use tags used to label the image? If your image contains more niche tags that match the user input, your revenue share will be higher. Depending how much extra people earn for certain tags, it might incentivise people to upload more images of what is missing from the training data.

brookst 4y ago
That's interesting, but I'm not sure it works. I think that works out to "for any given prompt, distribute credit to every source image that has a keyword that appears in the prompt, proportional to how many other source images had that same keyword". If I include the tag "floor", do I get some (tiny) percentage of every image that uses "floor" in the prompt, even if the bits from my image did not end up affecting model weights much at all in training? Worse, for tags like "dramatic lighting", it's likely that the important source images will depend on the other words in the prompt; "sunset, dramatic lighting" will probably not use the rely on the same weights or source images as "theater interior, dramatic lighting". And then you get the perverse incentives to tag every image with every possible tag :) I'd love to be convinced otherwise, but I'm not seeing prompt-to-tag association working.
- bash-j 4y ago
  The tags could be added by a model rather than the user submitting the image. Maybe do both and verify the tags with a model? Users could get a rating based on how reliably they tag their pictures and are trusted to add more niche tags at higher ratings. You could even help tag other pictures to improve your rating.

gradys 4y ago

https://arxiv.org/abs/1703.04730 https://arxiv.org/abs/1703.04730 > How can we explain the predictions of a black-box model? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction.

brookst 4y ago
Oh thank you! Will go read and digest.
batch12 4y ago
This would be an interesting way to distill/finetune a model by finding the optimal minimum corpus needed to match a given use case.

natch 4y ago

> And then you have to figure out which weights contributed the most to an output. Why do you have to, though? What do you hope to trace back to exactly?

brookst 4y ago
The business problem is “which items from the training corpus contributes the most to this inference”