4 ms·

> One thing that surprised me was when we asked [ChatGPT] to generate the same task – the same type of program in different languages – sometimes, for one langu

> One thing that surprised me was when we asked [ChatGPT] to generate the same task – the same type of program in different languages – sometimes, for one language, it would be secure and for a different one, it would be vulnerable. Because this type of language model is a bit of a black box, I really don't have a good explanation or a theory about this. Would be nice if they interviewed people who knew the absolute basics of how this thing works before commenting on its properties

cyanydeez 3y ago
No one who trained these models can diagram input to output
- fauxpause_ 3y ago
  Seems like a pretty outrageous claim to me
  - endofreach 3y ago
    How would one do that exactly?
    - fauxpause_ 3y ago
      If you mean the architecture it’s very well documented. If you mean the individual neuron weights then no, but that’s hardly required to understand how the model works. Tools exist to help unpack a black box neural net if you really cared.
      - salawat 3y ago
        Links, if you don't mind? Inquisitive minds wish to know.
        Our_Benefactors 3y ago
        Tensorboard
        fauxpause_ 3y ago
        Link to what?
        salawat 3y ago
        >Tools exist to help unpack a black box neural net if you really cared. If you're going to claim it exists, do "the class" a solid by sharing. I, for one, am highly interested in anything that can reverse from blob -> potential source characterization.
        fauxpause_ 3y ago
        This response feels… oddly passive aggressive to me. Idk if that was intentional? Like, I listed a few things and I just wanted to know what you wanted a link to. The answer is still “it depends” on what you’re trying to unpack. SHAP is a great tool for unpacking deep nets on image recognition. Show which pixels mattered and which did not. Very cool. Various dimension reduction or attention tracking or whatever exists for unpacking generate text and what not. If you want to understand the nuance of a particular output you’re best off looking at the distribution of next tokens and their pre-temperature weighted sampling. Different questions require different tools. Why did the model choose that phrasing? Why did the model choose this answer instead of that answer? What made the model make a certain claim? A lot of it will be driven by the context assuming that the model works. So if you ask “Is X good or bad and why” the majority of text will be on the “why”, but the generation of that text will be contingent on the initial tokens that determined if it should say “good or bad” first. The model does not come up with a reasoning for its assertions, it comes up with an assertion and then creates a reasoning to defend it (in this question format at least). So perhaps that is all you care about? Anyway, it depends on the question.
        salawat 3y ago

cyanydeez 3y ago

No one who trained these models can diagram input to output

fauxpause_ 3y ago
Seems like a pretty outrageous claim to me
- endofreach 3y ago
  How would one do that exactly?
  - fauxpause_ 3y ago
    If you mean the architecture it’s very well documented. If you mean the individual neuron weights then no, but that’s hardly required to understand how the model works. Tools exist to help unpack a black box neural net if you really cared.
    - salawat 3y ago
      Links, if you don't mind? Inquisitive minds wish to know.
      - Our_Benefactors 3y ago
        Tensorboard
      - fauxpause_ 3y ago
        Link to what?
        salawat 3y ago
        >Tools exist to help unpack a black box neural net if you really cared. If you're going to claim it exists, do "the class" a solid by sharing. I, for one, am highly interested in anything that can reverse from blob -> potential source characterization.
        fauxpause_ 3y ago
        This response feels… oddly passive aggressive to me. Idk if that was intentional? Like, I listed a few things and I just wanted to know what you wanted a link to. The answer is still “it depends” on what you’re trying to unpack. SHAP is a great tool for unpacking deep nets on image recognition. Show which pixels mattered and which did not. Very cool. Various dimension reduction or attention tracking or whatever exists for unpacking generate text and what not. If you want to understand the nuance of a particular output you’re best off looking at the distribution of next tokens and their pre-temperature weighted sampling. Different questions require different tools. Why did the model choose that phrasing? Why did the model choose this answer instead of that answer? What made the model make a certain claim? A lot of it will be driven by the context assuming that the model works. So if you ask “Is X good or bad and why” the majority of text will be on the “why”, but the generation of that text will be contingent on the initial tokens that determined if it should say “good or bad” first. The model does not come up with a reasoning for its assertions, it comes up with an assertion and then creates a reasoning to defend it (in this question format at least). So perhaps that is all you care about? Anyway, it depends on the question.
        salawat 3y ago