3 ms·

I wholeheartedly agree with the article critique. Every output of an LLM is a hallucination. If you try to argue anything else, you need the machine to be able

by _wire_ 2y ago

I wholeheartedly agree with the article critique. Every output of an LLM is a hallucination. If you try to argue anything else, you need the machine to be able to judge the validity of a response. But it does precisely the opposite: it expects you to judge the validity. Every output of the device is of a form terminated by an implied question mark. It is querying the operator. Yet, strangely, it seems none of the models can learn from the operator. So interaction is a circuit of hallucinations. Like a N-way Magic 8-ball it's fun, for a while. Then you begin to notice that you have to work just as hard to make sense of its help as you do to think for yourself. Being able to not know seems to me to be a crucial first step for sentience, followed closely by adaptability to not knowing— curiosity. An organism is a manifest endogenous dynamic with a suitability for exploration of the environment which gives rise to it. AI is an constructed exogenous dynamic with a suitability for replicating state based on a corpus of exposures and a context. An organism is distinguished by the feature that it keeps going. An AI is distinguished by the feature that it stops. That the human organism is now peering into the abyss of its recorded history via construction of the AI is a special event, full of strange possibilities. But the making of a reliable oracle doesn't look practical without the AI being able to govern itself, which it obviously cannot do. As to the value of an unreliable oracle, seems to be practical as a general-purpose time-wasting device.

yunwal 2y ago
There are many many questions where a correct/incorrect answer is quick and easy to verify, but difficult to create. If you understand which questions these are, you can use AI without wasting time.
- _wire_ 2y ago
  I think this point is fair given that "AI" is a highly provisional term. This point implicitly re-frames the capability as search engine, and steers away from any idea of thought as an innate capability the AI device. As long as as the oracle's answer is expected to require human verification, what's going on is a human thought amplification (bulldozer for for the mind, to quote a famous computational linguist). Even taking the point as an upside, I see tremendous hazards for human-mediated oracles (as already observed by many others): that the "well" of the commons can be polluted by AI output feeding back into the training sets, and that human hallucinations are amplified. From a POV of ethics, hallucination is absolutely intolerable in human decision-making in industrial settings. Yet industrial users seem not just tolerant of AI acting crazy but completely accepting and forgiving of AI craziness. It's routinely considered to be interesting and exciting to see it go wrong and tremendous effort attends trying to steer the craziness in new directions, whereas in a human setting the hallucinating agent would be disqualified from participation. The IT industry is taking very dangerous tacks without a second thought. I recently saw a job posting for a firm called UpCodes that invites junior web stack programmers to add LLM tech to building code compliance systems as follows: https://up.codes/careers?ashby_jid=1ed9b2d6-86fb-4ff0-9dff-4d215e5e08e4 https://up.codes/careers?ashby_jid=1ed9b2d6-86fb-4ff0-9dff-4... // EXAMPLE PROJECTS • Use an LLM to identify references to other sections in the text of the law • Create Rap Genius-style annotations on sections of building code, to create a more interactive user experience • Write algorithms to detect and display how versions of a code have changed through time, so users understand how regulations have evolved • Migrate our NLP algorithms over to use LLMs // WHAT COULD GO WRONG? And what qualifications, standards and practices are demanded of this developer position? // • 3+ years of experience building web products, ideally across the stack • Clean simple code, even when solving complex problems • Strong sense of UX • Good systems design • Ability to learn new tech quickly // DONE AND DONE
  - yunwal 2y ago
    I agree, AI output should effectively be treated similarly to untrusted user input. There are some situations where it's ok to use directly, because the consequences of getting it wrong are minimal. There's other cases where it makes sense with human review. For anything with real consequences, there ought to be the equivalent of an FDA drug trial showing that the trained model works before allowing its use without human review.

by _wire_ 2y ago

yunwal 2y ago

There are many many questions where a correct/incorrect answer is quick and easy to verify, but difficult to create. If you understand which questions these are, you can use AI without wasting time.

_wire_ 2y ago
I think this point is fair given that "AI" is a highly provisional term. This point implicitly re-frames the capability as search engine, and steers away from any idea of thought as an innate capability the AI device. As long as as the oracle's answer is expected to require human verification, what's going on is a human thought amplification (bulldozer for for the mind, to quote a famous computational linguist). Even taking the point as an upside, I see tremendous hazards for human-mediated oracles (as already observed by many others): that the "well" of the commons can be polluted by AI output feeding back into the training sets, and that human hallucinations are amplified. From a POV of ethics, hallucination is absolutely intolerable in human decision-making in industrial settings. Yet industrial users seem not just tolerant of AI acting crazy but completely accepting and forgiving of AI craziness. It's routinely considered to be interesting and exciting to see it go wrong and tremendous effort attends trying to steer the craziness in new directions, whereas in a human setting the hallucinating agent would be disqualified from participation. The IT industry is taking very dangerous tacks without a second thought. I recently saw a job posting for a firm called UpCodes that invites junior web stack programmers to add LLM tech to building code compliance systems as follows: https://up.codes/careers?ashby_jid=1ed9b2d6-86fb-4ff0-9dff-4d215e5e08e4 https://up.codes/careers?ashby_jid=1ed9b2d6-86fb-4ff0-9dff-4... // EXAMPLE PROJECTS • Use an LLM to identify references to other sections in the text of the law • Create Rap Genius-style annotations on sections of building code, to create a more interactive user experience • Write algorithms to detect and display how versions of a code have changed through time, so users understand how regulations have evolved • Migrate our NLP algorithms over to use LLMs // WHAT COULD GO WRONG? And what qualifications, standards and practices are demanded of this developer position? // • 3+ years of experience building web products, ideally across the stack • Clean simple code, even when solving complex problems • Strong sense of UX • Good systems design • Ability to learn new tech quickly // DONE AND DONE
- yunwal 2y ago
  I agree, AI output should effectively be treated similarly to untrusted user input. There are some situations where it's ok to use directly, because the consequences of getting it wrong are minimal. There's other cases where it makes sense with human review. For anything with real consequences, there ought to be the equivalent of an FDA drug trial showing that the trained model works before allowing its use without human review.