7 ms·
Huh. They've been able to demonstrate that the trained Othello player has a model. They did this by feeding the entire state of the trained system into a classi
by Animats 4y ago
Huh. They've been able to demonstrate that the trained Othello player has a model. They did this by feeding the entire state of the trained system into a classifier. But they have no idea what the form of that model is, or how it is used to predict moves. Even for this simple, well-defined problem, it's a total black box.
This level of incomprehensibility is worrisome.
- ramraj07 4y agoIsn’t our brain a black box too? Why exactly is it worrisome?
- tobr 4y agoBecause human brains have been known to intentionally lie and manipulate, and secretly plan horrible deeds.
- eternalban 4y agoWe'll need psych tests for our AI. Malicious AI will note this and adapt. An arms races ensues, where one adversary finances and invests in upgrading the other. Guess who wins?
- shadowofneptune 4y agoThe mind itself, perhaps. The person in possession of it though usually does have a conscious strategy to play a game once they are proficient, and can happily explain it to you.
- pyinstallwoes 4y agoHow many people can actually explain their state of mind and rational or seemingly rational thought process?
- space_fountain 4y agoAnd we have all sorts of social science these days implying the explanations when people do try to explain themselves they’re justifying decisions that came out of black box rather than actual describing how the decision happened
- nequo 4y agoAnd also neuroscience examining whether what we think we did out of choice was really done out of our choice: https://en.wikipedia.org/wiki/Neuroscience_of_free_will https://en.wikipedia.org/wiki/Neuroscience_of_free_will
- taneq 4y agoWhat do you (or they) think “our choice” is if not the result of these processes? It’s not like “you” are just the part of your brain that is consciously aware of making decisions.
- Tepix 4y agoI play Othello. In most situations i can give you detailed reasons why i choose to make a particular move.
- taneq 4y agoAre they actually the reasons for your move, or are they post-hoc justifications, and how would you know?
- Isinlor 4y agoYou are not explaining your internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused you to arrive at given moves. You will only provide proofs that your moves are good.
- bobbruno 4y agoYou can justify post facto the move you made. Can you write down the logical process (probably a combination of strategy, move generation, analysis and selection) that would replicate the moves you picked from the beginning of the game? I don’t even need the brain chemistry (the implementation), but I don’t think even writing down the logic that generates those moves (in contrast with a logic to justify each move after it was generated) is something humans can do in most cases.
- gpm 4y agoI strongly disagree with this statement. A good player can usually walk you through the portion of the game tree that they expanded in their head, but the moves they had to think about (expand their mental game tree to include the resulting position) versus the moves they instinctively ruled out is basically inexplicable, and where a lot of the skill lies. (I'm thinking more chess than othello, since I'm much better at the former, but I assume the thought process is similar)
- sour-taste 4y agoStrongly agree. I'm NOT good at chess, but I still like watching it on YouTube. The explanations that top players give for their moves sound like magic a lot of the time. If they sat down with me for an hour could they explain every step of how they got to their conclusion? I think they could, but I don't think that they'd explain the exact same process they actually went through, which is based on inexplicable heuristic as parent mentioned. Often times at complex decision points in chess players will just say that a move feels better. That feeling can't be explained.
- shadowofneptune 4y agoI have to admit you're right there, now thinking about how I would explain games I play, approaches to tasks, etc.
- jacquesm 4y agoThat's not correct. It doesn't matter whether they can explain their moves to you. What matters is if they can explain their moves to another grand master and they always can unless they moved a random piece to break up a position to make a boring game more interesting (this rarely happens, but it is an interesting way to play, but not to win contests).
- Isinlor 4y agoThey are not explaining their internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused them to arrive at given moves. They just provide proofs that their moves were good.
- squokko 4y agoA person with a million loyal adherents is worrisome even if those adherents are only sitting on computers. For similar reasons, a person with a trillion unflinchingly loyal AI adherents is worrisome.
- xwolfi 4y agoWhat, why? Unless they vote or are convincing to humans, it's not that worrisome. I mean end of the day, I care about eating well, raising my kids in peace, being useful to others: if an AI tries to convince me to act, it'd have to factor these constraints, in which case it's a useful fellow human able to understand my need and proposing an action for debate. Wars and disasters are usually pursued by minority groups in charge of means of violence, and manipulation is a way for them to trap you into letting a crack open long enough for them to invade it, but if we cant resist an AI, they deserve the Earth more than us :)
- stackbutterflow 4y agoBecause the world has had thousand years to evolve, adapt and settle around the capabilities and limitations of the human brain. It is somewhat stable and predictable. Now if a new species of humans suddenly emerged with a brain not only has capable as our but faster, more adaptable and extensible, it would disrupt the world.
- belugacat 4y agoEcosystems (what I infer you mean by "the world", since things that are part of the world such as gravity, plate tectonics, etc don't follow the laws of evolution) haven't had thousands of years to evolve, adapt, and settle around the capabilities that we've had since the industrial revolution started. AI is merely an aspect and consequence of the logical progression of that continuity. (In terms of tangible effect on the world today and what seems on track for the few decades to come, AI is still far behind coal - to pick one - when it comes to concrete negative externalities tho)
- gjm11 4y agoI think something is wrong with the logic of your first paragraph. Suppose (by way of analogy; this is not meant to be directly addressing AI) that in the next year some ingenious physicists finally get quantum gravity figured out, and that soon after the papers are published someone notices that the theory implies that one can use common household materials to make a device capable of destroying a planet. (Seems unlikely, but I don't think anyone would have predicted that the last revolutionary theory of gravity we worked out would imply that one can use not-so-common materials to make a reasonably small device capable of destroying a city, but it turns out it does.) It seems fairly likely that (1) the knowledge of how to do this could not be suppressed for ever, and that (2) if making the device turns out not to need all the fancy apparatus and hard-to-get materials that e.g. hydrogen bombs require, it wouldn't be that long before some idiot actually does it and destroys the earth. And yet, these new discoveries would be "merely an aspect and consequence of the logical progression of that continuity", as you put it. So, whatever the actual situation is with AI, I think it is demonstrably not the case that we can be confident AI won't somehow kill us all or destroy the world merely because AI is a thing we made and the world has had thousands of years to adapt to us. Maybe everything will be fine, maybe not, but figuring out which will require looking in more detail rather than handwaving about how ecosystems always adapt.
- mihaic 4y agoWe are forced to work with the imperfections of the human mind, and a great deal of effort in our world is put into working with them. Why would we want more of this?
- taylorius 4y agoYou make it sound like the human brain is a shoddy piece of work. In any case, even if it is imperfect, it isn't imperfect because it is a black box.
- mihaic 4y agoWe sometimes make life-changing decisions based on where Jupiter was when we were born. I mean, we've been struggling for milenia against our biases. I'd let some time pass until we can decide on what these ML models can get subtly wrong.
- bigbluedots 4y agoSo that if it starts killing people at some point, you can debug it or maybe figure out why it has done that.
- meindnoch 4y agoWe're all driving one, so at least we can extrapolate from our own experience.
- dweinus 4y agoBecause, as a human, I can introspect my world model. I can explain it to myself and others. I can explain how it guides my decisions. I can directly update it without needing "training data". I don't have a map of my neurons, but I have self awareness. For LLMs, there is the opposite: transparent structure but opaque understanding.
- d1sxeyes 4y agoComputers have historically been 'correct' about things. Obviously, humans who program computers make mistakes, and occasionally, things like cosmic bit flips can occur, leading a computer to produce an incorrect answer. An LLM upends that paradigm, and worse, the closer it comes to appearing to reason and discuss, the more it implies that the human brain is nothing more than a biological computer, which is worrying for a lot of people. If humans are nothing more than biological computers, do (or can) silicon computers have souls? If a collection of diodes can have a soul, what makes us special?
- yorwba 4y ago> they have no idea what the form of that model is, or how it is used to predict moves I think this assertion is incorrect. Since the classifiers they used are very simple, they constrain the form of the model quite strongly; it must represent different tiles by different directions in the vector space of the internal state, otherwise the classifiers wouldn't be able to work. The representation for the whole board is then a sum of representations for each individual tile. The experiments with modifying individual tiles in the model also show at a high level how they influence the predicted move. The researchers could've also looked at how that is implemented at the level of individual weights. It's not in this write-up and maybe they didn't even try, but that doesn't mean they have literally no idea. The worrisome part is that it's performance isn't perfect, so there are bugs, and you might actually need to know all the low-level details to identify and fix the bug.
- anuragvohraec 4y agoisn't it curve fitting at the end of the day. A multi parameter curve fitting ? why do people say they don't now how it works. Yeah i get it that the cocktail is fairly complex, after training it on very huge dataset (all most all possible logical scenarios). But telling it we do not know how it works, seems like just adding mysticism to it, which attracts "clicks", but is not an honest description.
- simonh 4y agoEven assuming that curve fitting is actually in any way a meaningful description, let’s say I give you that. It tells us absolutely nothing about the mechanisms evolved in the neural pathways, how it encodes memory if the current game distinctly from memory of past games, the reasons for the strength or weaknesses of one trained instance against another, or ways we could optimise the architecture to better complement the way it function. It doesn’t help us engineer the system, or reason about its possible limitations or failure modes. In other words it doesn’t tell us anything actually useful about it.
- cmorez 4y ago> Why do people say they don't now how it works "Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this. As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles needed to play at AlphaGo's level), and are not just exhaustive lists of statistical trends but more like strategies[3]. Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution. The curve fitting perspective would totally miss all this for a comfortable but misleading story: "the objective is curve fitting so it's just interpolating data points". [1] https://transformer-circuits.pub https://transformer-circuits.pub [2] https://arxiv.org/abs/2212.07677 https://arxiv.org/abs/2212.07677 [3] https://www.pnas.org/doi/10.1073/pnas.2206625119 https://www.pnas.org/doi/10.1073/pnas.2206625119 [4] https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2301.05217 [5] https://arxiv.org/abs/2210.01117 https://arxiv.org/abs/2210.01117
- quotemstr 4y ago> This level of incomprehensibility is worrisome The brains of other people are just as incomprehensible.
- loa_in_ 4y agoAnd we've created a society of elitarism and exclusion. Which was failing us at every step so we created an agonist of this - the internet. Which enabled us to do AI...
- pyinstallwoes 4y agoYet you exist thanks to the group effort
- r_hoods_ghost 4y agoBut we now know that there is a smaller box within the bigger black box and have empirical data to back up that assertion, which is a step up from where we were previously. That LLMs have internal models and use them to make predictions means that we now know what level of abstraction we should be talking at and narrows down the exploration space within which to make investigations. Now we know that we can learn more about how LLMs work by investigating how such internal models arise. This sort of result makes LLMs slightly less black boxy than they were previously.
- miohtama 4y agoBecause the black box is shrinking, does it mean we might ge able to get similar quality results with smaller models?
- wongarsu 4y agoEventually better understanding of the mechanisms should allow us to make better models. Right now OpenAI is still exploring the boundaries of "how much does it get better if we just make it bigger", but that is just the first phase of making LLMs. Remember that transformer models are basically brand new, having been invented just six years ago. There are bound to be lots of improvements left that will later be described as low-hanging fruit.
- jvanderbot 4y agoSounds a bit like our own minds, honestly. It seems to be a giant box of connections with only a vague understanding of subsystems.
- jacquesm 4y agoIt really is, on many levels. The thing that I'm mostly worried about is that bit by bit we will be moving critical decision making to systems that are poorly understood, be it self driving cars, insurance companies or banks deciding to let you borrow money - or not. For now we have some degree of accountability, models need to be explainable for some small subset of the possible applications. But the overall mechanism seems to be that if there is a utilitarian perceived benefit that the solution is greenlit, without knowing whether this is just a statistical fluke which will steer us to longer term unintended consequences or whether it is a structural improvement. And once the decision is made to accept the model the ability to measure a baseline is destroyed so any 'creep' will likely go undetected. Such longer term consequences could be really damaging.
- jackmott 4y agopoorly understood systems like humans?
- chrismatic 4y agoI guess they make up for it with accountability.
- mindslight 4y agoYes... But humans generally have to explain ourselves. At the societal level, it's generally not good enough for another human to make an arbitrary decision. Rather they have to justify it with the reasoning leading to it. The better the reasoning, the more other humans believe them. Whereas with regards to tech, society is currently at the level of "computer says foo" (so it must be true). Requiring explanatory output to justify decisions is a likely path forward for making the use of LLM's accountable. But given how little we've been able to make traditional human-guided tech companies accountable for things like sharing scoring formulas and abusing personal information, I'm not hopeful.
- PeterisP 4y agoIt is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision; even ignoring lying, we do not have privileged insight into all the factors our brain used to make a decision, and when asked for a justification for any decision which wasn't 100% the result of explicit rational planning (which is almost every decision, as even high-planning decisions are demonstrably influenced by non-rational factors, intuition, etc), we use our social skills to effectively invent a post-factum reasoning that both sounds plausible and is socially acceptable - just as language models can do. We often deceive ourselves, we have biases that discourage us from admitting and acknowledging certain reasons for why we did or do something; in some cases extensive therapy can help us recognize why some justification or reason we told others and ourselves was a false one, etc. In essence, dealing with plausible but potentially misleading justifications is something that we have had to do forever, and will still have to do for future artificial agents as well.
- YeGoblynQueenne 4y ago>> They've been able to demonstrate that the trained Othello player has a model. They haven't demonstrated that, it remains a conjecture. They show some evidence that they claim supports their conjecture. This evidence comes from "probing" (the classifier-based method you describe) which is a heuristic method that lacks the power to demonstrate anything conclusively. This is as the authors point out: It’s believed that the higher accuracy these classifiers can get, the better the activations have learned about these real-world properties, i.e., the existence of these concepts in the model. "It's believed" as in, we don't know, therefore we can't use to demonstrate.
- amelius 4y agoDo we even have methods to rate the performance of these systems, and how good are they?