13 ms·
Of note: "... our technique works poorly for larger models, possibly because later layers are harder to explain." And even for GPT-2, which is what they used
by int_19h 4y ago
Of note:
"... our technique works poorly for larger models, possibly because later layers are harder to explain."
And even for GPT-2, which is what they used for the paper:
"... the vast majority of our explanations score poorly ..."
Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. This may be the first step towards that, but as they also note, the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; I would expect the former in particular to be much more common in larger models, which is perhaps why they're harder to analyze in this manner.
- rvz 4y ago> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. Exactly. Especially: > ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks. It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.
- jahewson 4y ago> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Like say, in court to detect if someone is lying? Or at an airport to detect drugs?
- int_19h 4y agoYou don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.
- carlmr 4y ago>Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. Evolution is still doing it's thing.
- deleted 4y ago[deleted]
- canadianfella 4y agoWhat’s the risk? Someone allergic to peanuts will eat peanuts because ChatGPT put it in their diet plan? That’s silly.
- int_19h 4y agoYes, that's the risk, and people are literally doing that because "if it put them in the recipe, it knows that quantity is safe for me", or "I asked it if it's okay and it cited a study saying that it is".
- coldtea 4y agoThose cases sound like Darwin Awards mediated by high technology
- canadianfella 4y ago[dead]
- pmarreck 4y ago> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Doesn't this criticism also apply to people to some extent? We don't know what the purpose of individual brain neurons is.
- TaylorAlexander 4y agoPeople are better understood intuitively. We understand how people fail and why. We can build trust with people with some degree of success. But machine models are new and can fail in unpredictable ways. They also get deployed to billions of users in a way that humans do not, and deployed in applications that humans do not. So its certainly useful to try to explain neural networks in as great of detail as we can.
- istjohn 4y agoOr we can build trust using black box methods like we do with humans, e.g., extrapolating from past behavior, administering tests, and the like.
- TaylorAlexander 4y agoWe can, but the nice thing about neural networks is the ability to do all kinds of computational and mathematical manipulations to them to basically pick them apart and really find out what’s going on. This is important not just for safe deployment but also for research on new methods that could be used to make them better. Plus we need this ability to help avoid neural networks with intentionally hidden features that appear to behave linearly in certain regimes but are designed with a strong nonlinear response when special inputs are applied. You could have all the tests you want for a self driving car based on real world conditions but some bad actor with access to the training system could create a special input that results in dangerous behavior.
- int_19h 4y ago
- ketzo 4y agoIs it really fair to say this brings us “no closer” to explainability? This seems like a novel approach to try to tackle the scale of the problem. Just because the earliest results aren’t great doesn’t mean it’s not a fruitful path to travel.
- TaylorAlexander 4y ago> the explainability situation that has been missing for decades in neural networks. Is this true? I thought explainability for things like DNNs for vision made pretty good progress in the last decade.
- deleted 4y ago[deleted]
- gitfan86 4y agoWe know that complex arrangements of neurons are triggered based on input and generating output that appears to have some intelligence to many humans. The more interesting question is why are intelligence/beauty/consciousness emergent properties that exist in our minds.
- dennisy 4y agoNature created humans to understand nature. We created GPT4 to understand ourselves.
- NobleLie 4y agoA mirror of ourselves*
- keyle 4y agoThat's beautiful until you think about it. Humans so far have done a great job at destroying nature faster than any other kind could. And GPT4 was created for profit.
- mensetmanusman 4y agoHumans are the universe asking who made it.
- slacka 4y agoYes, or put a bit more elegantly, 'The cosmos is within us. We are made of star-stuff. We are a way for the universe to know itself.' — Carl Sagan
- greenhearth 4y agoYes, "all serve "gpt4!" Get a grip, pretty please.
- otabdeveloper4 4y agoThere is no evidence that intelligence runs on neurons. Yes, there are neurons in brains, but there's also lots of other stuff in there too. And there are creatures that exhibit intelligent properties even though they have hardly any neurons at all. (An individual ant has only something like 250000 neurons, and yet they're the only creatures beside humans that managed to create a civilization.)
- imranq 4y agoI suspect that there's a sweet spot that combines a collection of several "neurons" and a human-readable explanation given a certain kind of prompt. However, this "three-body problem" will probably need some serious analytical capability to understand at scale
- PaulHoule 4y agoI like the idea. Note that LLMs have some skill at decoding sequential dense vectors in the human brain https://pub.towardsai.net/ais-mind-reading-revolution-how-gpt-llms-decode-thoughts-and-what-it-means-for-our-future-a24001a25d8 https://pub.towardsai.net/ais-mind-reading-revolution-how-gp... so why not have them decode sequential dense vectors of their own activations? As for the majority scoring poorly, they suggest that most neurons won't have clear activation semantics so that is intrinsic to the task and you'd have to move to "decoding the semantics of neurons that fire as a group"
- Imnimo 4y agoI don't think this is showing LLMs performing decoding. They're just using the LLM to propose possible words. The decoding is done by using another model to score how well a proposed word matches brain activity, and using that score to select a most likely sequence given the proposals from the LLM.
- ryandvm 4y agoFunny that we never quite understood how intelligence worked and yet it appears that we're pretty damn close to recreating it - still without knowing how it works. I wonder how often this happens in the universe...
- samiskin 4y agoEvolution created intelligence without even being intelligent itself
- dontknowwhyihn 4y agoHow do you know that’s true?
- chrischen 4y agoEvolution is not a single orchestrator. It is merely the natural result of a simple mechanical process over a timescale that exceeds the lifetime of the common human.
- candiddevmike 4y agoBecause we eat and breath through the same tube
- _justinfunk 4y agoThere are two tubes.
- p1esk 4y agoEvolution came up with the shared eating/breathing tube design because it made sense for aquatic animals (from which we evolved).
- KineticLensman 4y agoThe trachea and oesophagus both start in the common space of the pharynx. The epiglottis stops food going down the wrong one
- kfrzcode 4y agohere's a clue, start your research Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).
- nr2x 4y agoMy favorite line: “Our overall procedure is quite compute intensive.” Should that be measured in number of nuclear power plants needed to run the computation? Or like, fractions of a small star’s output?
- mnky9800n 4y agoYes but they are from openai so they can just write papers that say whatever they want to say without minding the metrics and then pretend like it is some kind of science.