5 ms·

I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand t

I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand to accelerate training? For example, in a vision model, they can increase the "orangeness" collection of weights when detecting traffic cones. That way, instead of requiring thousands/millions more examples to calibrate "orangeness" right, it's accelerated by a human expert. The difficulty is obviously having this interface map to the collections of weights that mean different things, but is there a technical reason this can't be done?

amilios 3y ago
The technical reason it can't be done (or would be very difficult to do) is that weights are typically very uninterpretable. There aren't specific clusters of neurons that map to one concept or another, everything kind of does everything.
- Supply5411 3y ago
  I wonder if an expert can "impose" weights onto a model and the model will opt to continue with them when it resumes training. For example, in the vision example, the expert may not know where "orangeness" currently exists, but if they impose their own collection of weight adjustments that represent orangeness, will the model continue to use these weights as the path of least resistance when continuing to optimize? Just spitballing, but if we can't pick out which neurons do what, the alternative would seem to be to encourage the model to adopt a control interface of neurons.
  - astrange 3y ago
    That would make it less efficient - since learning is compression, a less compressed model will also learn less at the same size.
    - Jensson 3y ago
      Humans have much more compressed models, trying to transfer human learning to machine learning could potentially be a way to get more efficient models. The way we do that currently is by labeling data for training, but maybe there are better ways to do it. Like some semi code to write with hints for the model. Like, instead of labeled data, could have a series of "lectures" of labeled data that would lead to a good end state, instead of training on all the data in parallel. You don't teach a child calculus by showing them a million calculus problems after all, you ramp up starting with simple numbers and then slowly ramping up with more concepts. But to do that we would need to change how we train models. Edit: By doing it that way you could see the skill of the model after each lecture, and update the lecture to try to make the model learn better. Not sure how to do that, but such ways to work with parts of models is a potential way forward.
      - astrange 3y ago
        There are techniques for this called "curriculum learning" and "textbooks". I'm not sure exactly what is in the textbooks since I admit to not reading the papers yet. I'm personally wondering if you could increase the reliability of training on web data by labeling it with where each document came from, so it knows different authors disagree on things. But this brings back the issue where people don't like it if a model can write "in the style of Author Name"…
    - Supply5411 3y ago
      Might be a worthwhile tradeoff. Less efficient model but a clear control interface, versus more efficient model but it's a blackbox.
- gbear0 3y ago
  On the idea of interpreting the weights, I've been very interested if it's possible to compute basis vectors of the weights matrix to define the core concepts within the model and then do a change of basis to allow reorganizing the model to more human understood concepts? I think the inherent compression of a specific training set into a matrix makes this more difficult cause the basis vectors likely won't contain clean representations of human ideas, but I also wonder if starting a new training set with an initialized (or fixed) matrix of human defined concepts would help align the model's weights to something that can be interpretable
klysm 3y ago
The attention mechanisms present in transformers don’t seem easy to map to semantics that humans can understand. There are too many parameters involved
jarde 3y ago
The number of layers and weights is really not at a scale we can handle updating manually, and even if we could the downstream effects of modifying weights are way too hard to manage. Say you are updating the picture to be better at orange, but unless you can monitor all the other colours for correctness at the same time you probably are creating issues for other colours without realizing it
elesiuta 3y ago
> weights of a model, that a domain expert can tweak by hand This sounds similar to how image recognition was done before deep learning [1] [1] https://www.youtube.com/watch?v=8SF_h3xF3cE&t=1358s https://www.youtube.com/watch?v=8SF_h3xF3cE&t=1358s
- Supply5411 3y ago
  Great example. Right, the deep learning approach uncovers all kinds of hidden features and relationships automatically that a team of humans might miss. I guess I'm thinking about this problem from the perspective of these GPT models requiring more training data than a normal person can acquire. Currently, it seems you need the entire internet worth of training data (and a lot of money) to get something that can communicate reasonably well. But most people can communicate reasonably well, so it would be cool if that basic communication knowledge could be somehow used to accelerate training and minimize the reliance on training data.
  - EricMausler 3y ago
    I am still learning transformers, but I believe part of the issue may be that the weights do not necessarily correlate to things like "orangeness" Instead of a transformer for each color, you have like 5 to 100 weights that represent some arbitrary combination of colors. Literally the arbitrariness is defined by the dataset and the number of weights allocated. They may even represent more than just color. So I am not sure if a weight is actually a "dial" like you are describing it, where you can turn up or down different qualities. I think the relationship between weights and features is relatively chaotic. Like you may increase orangeness but decrease "cone shapedness" or accidentally make it identify deer as trees or something, all by just changing 1 value on 1 weight
    - space_boy 3y ago
      It is possible that the parameters, like weights in a machine learning model, interact to yield outcomes in a manner analogous to the interactions between genes in biological systems, which produce traits. These interactions involve complex interdependencies, so there really aren't 1 to 1 dials.
PoignardAzur 3y ago
The reason you're looking for is called "The Bitter Lesson". The short version is, trying to give human assistance to AIs is almost always less cost-effective than making them run on more computing power. By the time your human expect has calibrated your weight layers to detect orange traffic cones, your GPU cluster has trained its AI to detect traffic cones, traffic ligts, trees, other cars, traffic cones with a slightly different shade of orange, etc.
- civilized 3y ago
  The AI community has over-learned this bitter lesson and is now unprepared to make useful AI in data-constrained environments. This is why almost all of the countless enterprise AI demos taking place as we speak are doomed to failure.
- mistrial9 3y ago
  "The bitter lesson of machine learning is that building knowledge into agents does not work in the long run, and breakthrough progress comes from scaling computation by search and learning.02 This applies to domains where domain knowledge is weak or hard to express mathematically. The rapid progress of ML applied to LQCD, mol.2 dyn., protein folding, and computer graphics is the result of combining domain knowledge with ML." This passage says that the advantages of scaling a certain kind of learning are especially good when .. two conditions.. but a side-effect of that statement is, when knowledge is well known, and maybe straightforward to express, these kinds of learning systems, not as great.. this is true. without taking on big other topics, I think this "bitter lesson" is unspecific enough to include some self-serving utility.. just tell the other camps to give up, you lost. that sort of thing.

amilios 3y ago

The technical reason it can't be done (or would be very difficult to do) is that weights are typically very uninterpretable. There aren't specific clusters of neurons that map to one concept or another, everything kind of does everything.

Supply5411 3y ago
I wonder if an expert can "impose" weights onto a model and the model will opt to continue with them when it resumes training. For example, in the vision example, the expert may not know where "orangeness" currently exists, but if they impose their own collection of weight adjustments that represent orangeness, will the model continue to use these weights as the path of least resistance when continuing to optimize? Just spitballing, but if we can't pick out which neurons do what, the alternative would seem to be to encourage the model to adopt a control interface of neurons.
- astrange 3y ago
  That would make it less efficient - since learning is compression, a less compressed model will also learn less at the same size.
  - Jensson 3y ago
    Humans have much more compressed models, trying to transfer human learning to machine learning could potentially be a way to get more efficient models. The way we do that currently is by labeling data for training, but maybe there are better ways to do it. Like some semi code to write with hints for the model. Like, instead of labeled data, could have a series of "lectures" of labeled data that would lead to a good end state, instead of training on all the data in parallel. You don't teach a child calculus by showing them a million calculus problems after all, you ramp up starting with simple numbers and then slowly ramping up with more concepts. But to do that we would need to change how we train models. Edit: By doing it that way you could see the skill of the model after each lecture, and update the lecture to try to make the model learn better. Not sure how to do that, but such ways to work with parts of models is a potential way forward.
    - astrange 3y ago
      There are techniques for this called "curriculum learning" and "textbooks". I'm not sure exactly what is in the textbooks since I admit to not reading the papers yet. I'm personally wondering if you could increase the reliability of training on web data by labeling it with where each document came from, so it knows different authors disagree on things. But this brings back the issue where people don't like it if a model can write "in the style of Author Name"…
  - Supply5411 3y ago
    Might be a worthwhile tradeoff. Less efficient model but a clear control interface, versus more efficient model but it's a blackbox.
gbear0 3y ago
On the idea of interpreting the weights, I've been very interested if it's possible to compute basis vectors of the weights matrix to define the core concepts within the model and then do a change of basis to allow reorganizing the model to more human understood concepts? I think the inherent compression of a specific training set into a matrix makes this more difficult cause the basis vectors likely won't contain clean representations of human ideas, but I also wonder if starting a new training set with an initialized (or fixed) matrix of human defined concepts would help align the model's weights to something that can be interpretable

klysm 3y ago

The attention mechanisms present in transformers don’t seem easy to map to semantics that humans can understand. There are too many parameters involved

jarde 3y ago

The number of layers and weights is really not at a scale we can handle updating manually, and even if we could the downstream effects of modifying weights are way too hard to manage. Say you are updating the picture to be better at orange, but unless you can monitor all the other colours for correctness at the same time you probably are creating issues for other colours without realizing it

elesiuta 3y ago

> weights of a model, that a domain expert can tweak by hand This sounds similar to how image recognition was done before deep learning [1] [1] https://www.youtube.com/watch?v=8SF_h3xF3cE&t=1358s https://www.youtube.com/watch?v=8SF_h3xF3cE&t=1358s

Supply5411 3y ago
Great example. Right, the deep learning approach uncovers all kinds of hidden features and relationships automatically that a team of humans might miss. I guess I'm thinking about this problem from the perspective of these GPT models requiring more training data than a normal person can acquire. Currently, it seems you need the entire internet worth of training data (and a lot of money) to get something that can communicate reasonably well. But most people can communicate reasonably well, so it would be cool if that basic communication knowledge could be somehow used to accelerate training and minimize the reliance on training data.
- EricMausler 3y ago
  I am still learning transformers, but I believe part of the issue may be that the weights do not necessarily correlate to things like "orangeness" Instead of a transformer for each color, you have like 5 to 100 weights that represent some arbitrary combination of colors. Literally the arbitrariness is defined by the dataset and the number of weights allocated. They may even represent more than just color. So I am not sure if a weight is actually a "dial" like you are describing it, where you can turn up or down different qualities. I think the relationship between weights and features is relatively chaotic. Like you may increase orangeness but decrease "cone shapedness" or accidentally make it identify deer as trees or something, all by just changing 1 value on 1 weight
  - space_boy 3y ago
    It is possible that the parameters, like weights in a machine learning model, interact to yield outcomes in a manner analogous to the interactions between genes in biological systems, which produce traits. These interactions involve complex interdependencies, so there really aren't 1 to 1 dials.

PoignardAzur 3y ago

The reason you're looking for is called "The Bitter Lesson". The short version is, trying to give human assistance to AIs is almost always less cost-effective than making them run on more computing power. By the time your human expect has calibrated your weight layers to detect orange traffic cones, your GPU cluster has trained its AI to detect traffic cones, traffic ligts, trees, other cars, traffic cones with a slightly different shade of orange, etc.

civilized 3y ago
The AI community has over-learned this bitter lesson and is now unprepared to make useful AI in data-constrained environments. This is why almost all of the countless enterprise AI demos taking place as we speak are doomed to failure.
mistrial9 3y ago
"The bitter lesson of machine learning is that building knowledge into agents does not work in the long run, and breakthrough progress comes from scaling computation by search and learning.02 This applies to domains where domain knowledge is weak or hard to express mathematically. The rapid progress of ML applied to LQCD, mol.2 dyn., protein folding, and computer graphics is the result of combining domain knowledge with ML." This passage says that the advantages of scaling a certain kind of learning are especially good when .. two conditions.. but a side-effect of that statement is, when knowledge is well known, and maybe straightforward to express, these kinds of learning systems, not as great.. this is true. without taking on big other topics, I think this "bitter lesson" is unspecific enough to include some self-serving utility.. just tell the other camps to give up, you lost. that sort of thing.