5 ms·
It really is, on many levels. The thing that I'm mostly worried about is that bit by bit we will be moving critical decision making to systems that are poorly u
by jacquesm 4y ago
It really is, on many levels. The thing that I'm mostly worried about is that bit by bit we will be moving critical decision making to systems that are poorly understood, be it self driving cars, insurance companies or banks deciding to let you borrow money - or not.
For now we have some degree of accountability, models need to be explainable for some small subset of the possible applications. But the overall mechanism seems to be that if there is a utilitarian perceived benefit that the solution is greenlit, without knowing whether this is just a statistical fluke which will steer us to longer term unintended consequences or whether it is a structural improvement. And once the decision is made to accept the model the ability to measure a baseline is destroyed so any 'creep' will likely go undetected. Such longer term consequences could be really damaging.
- jackmott 4y agopoorly understood systems like humans?
- chrismatic 4y agoI guess they make up for it with accountability.
- mindslight 4y agoYes... But humans generally have to explain ourselves. At the societal level, it's generally not good enough for another human to make an arbitrary decision. Rather they have to justify it with the reasoning leading to it. The better the reasoning, the more other humans believe them. Whereas with regards to tech, society is currently at the level of "computer says foo" (so it must be true). Requiring explanatory output to justify decisions is a likely path forward for making the use of LLM's accountable. But given how little we've been able to make traditional human-guided tech companies accountable for things like sharing scoring formulas and abusing personal information, I'm not hopeful.
- PeterisP 4y agoIt is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision; even ignoring lying, we do not have privileged insight into all the factors our brain used to make a decision, and when asked for a justification for any decision which wasn't 100% the result of explicit rational planning (which is almost every decision, as even high-planning decisions are demonstrably influenced by non-rational factors, intuition, etc), we use our social skills to effectively invent a post-factum reasoning that both sounds plausible and is socially acceptable - just as language models can do. We often deceive ourselves, we have biases that discourage us from admitting and acknowledging certain reasons for why we did or do something; in some cases extensive therapy can help us recognize why some justification or reason we told others and ourselves was a false one, etc. In essence, dealing with plausible but potentially misleading justifications is something that we have had to do forever, and will still have to do for future artificial agents as well.
- mindslight 4y agoI fully agree [0]. And I can see how my original comment was worded such that you'd think I wasn't taking that into account ("the reasoning"). When using ChatGPT, I can't help but thinking "this sounds like a high school essay". But really, it's not. Rather, it's someone who has college+ level of reading and studying, but has never had to have their ideas tested or scrutinized. A user of ChatGPT kind of mitigates this by asking for clarification, which is a form of learning in the specific session. But in the real world, ChatGPT-as-student would be adjusted with that feedback and then incorporate it into its overall model for serving the next user. [0] For example, the SSC post about "predictive processing" resonated strongly with me
- jacquesm 4y agoIf they only add a feedback loop it will already improve hundredfold in a very short time with respect to the output because then suddenly people rather than endless unstructured input will guide the answers towards the expected outcome. That is definitely still a very far cry from actual intelligence but it may well become a lot more useful that way.
- hoseja 4y agoHumans are: - fundamentally limited - rough edges sanded off in the ball tumble mill of evolution
- klodolph 4y agoYes. I am worried that over the long term, models which are explainable or justifiable will be pushed aside by models which are nonexplainable and nonjustifiable, because the nonexplainable / nonjustifiable models are cheaper to get running and in the end we are all trapped inside a system that ruthlessly optimizes short-term costs. I've only done a small amount of statistical work professionally, but I remember that it was basically a nightmare to try and collect the right data and use the right variables to get a useful inference. Other engineers on my team would try to improve the models by throwing more variables at them or increasing the number of parameters in the model and I was always pushing back on this. I felt a bit crazy pushing back on this--surely, using more variables and more parameters will make the model fit historical data more closely, and a model which fits historical data more closely will fit future data more closely, and a model which fits future data more closely will be more useful? Explaining why this is wrong is really hard. I have deep respect for people who are able to explain the principles of statistics and statistical modeling.
- thaumasiotes 4y ago> I felt a bit crazy pushing back on this--surely, using more variables and more parameters will make the model fit historical data more closely, and a model which fits historical data more closely will fit future data more closely, and a model which fits future data more closely will be more useful? > Explaining why this is wrong is really hard. Seems straightforward. Points 1 and 3 are correct; point 2 is false.
- klodolph 4y agoPoints 1 and 2 are both wrong also. Statistics is not intuitive.
- thaumasiotes 4y agoPoint 1 isn't wrong; it's correct. There's a reason the name for this problem is "overfitting".