3 ms·
> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. Exactly. Espec
by rvz 4y ago
> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to.
Exactly. Especially:
> ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles;
This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks.
It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.
- jahewson 4y ago> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Like say, in court to detect if someone is lying? Or at an airport to detect drugs?
- int_19h 4y agoYou don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.
- carlmr 4y ago>Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. Evolution is still doing it's thing.
- deleted 4y ago[deleted]
- canadianfella 4y agoWhat’s the risk? Someone allergic to peanuts will eat peanuts because ChatGPT put it in their diet plan? That’s silly.
- int_19h 4y agoYes, that's the risk, and people are literally doing that because "if it put them in the recipe, it knows that quantity is safe for me", or "I asked it if it's okay and it cited a study saying that it is".
- coldtea 4y agoThose cases sound like Darwin Awards mediated by high technology
- canadianfella 4y ago[dead]
- chaxor 4y agoI hear crushed glass has very great health benefits
- pmarreck 4y ago> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Doesn't this criticism also apply to people to some extent? We don't know what the purpose of individual brain neurons is.
- TaylorAlexander 4y agoPeople are better understood intuitively. We understand how people fail and why. We can build trust with people with some degree of success. But machine models are new and can fail in unpredictable ways. They also get deployed to billions of users in a way that humans do not, and deployed in applications that humans do not. So its certainly useful to try to explain neural networks in as great of detail as we can.
- istjohn 4y agoOr we can build trust using black box methods like we do with humans, e.g., extrapolating from past behavior, administering tests, and the like.
- TaylorAlexander 4y agoWe can, but the nice thing about neural networks is the ability to do all kinds of computational and mathematical manipulations to them to basically pick them apart and really find out what’s going on. This is important not just for safe deployment but also for research on new methods that could be used to make them better. Plus we need this ability to help avoid neural networks with intentionally hidden features that appear to behave linearly in certain regimes but are designed with a strong nonlinear response when special inputs are applied. You could have all the tests you want for a self driving car based on real world conditions but some bad actor with access to the training system could create a special input that results in dangerous behavior.
- int_19h 4y ago
- ketzo 4y agoIs it really fair to say this brings us “no closer” to explainability? This seems like a novel approach to try to tackle the scale of the problem. Just because the earliest results aren’t great doesn’t mean it’s not a fruitful path to travel.
- TaylorAlexander 4y ago> the explainability situation that has been missing for decades in neural networks. Is this true? I thought explainability for things like DNNs for vision made pretty good progress in the last decade.
- deleted 4y ago[deleted]