4 ms·
> The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being dec
by TeMPOraL 3y ago
> The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure.
My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious activity. My raw thought stream is about as good as LLM output - sounds right, but it's often full of nonsense. That's why I tend to say LLM output is akin to that unfiltered inner voice output.
My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things.
Also, fight-or-flight reaction may make you switch into full-on bullshitting mode, without it being intentional.
> they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate
And my argument is that this is the same thing a person's inner voice does. Deep understanding of a topic may make your inner voice output you more of the correct associations, but ultimately, most of the time, people have to have a back-and-forth with themselves in their own head to then say/write/do something correctly. With more complex (or less understood) tasks, this involves consciously performing an explicit step-by-step algorithm you track in your mind (and checklists in aviation and medicine are further externalization of that idea - to protect against you misremembering (or "hallucinating") the steps).
> It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output.
The output contains probability factors for predicted tokens, but chat interface doesn't expose it, and doesn't make a good use of it.
- starbugs 3y ago> My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things. There is a key difference between making up new facts and copying 'bullshit' picked up somewhere else. I agree that more than 50% of people don't put much effort into the accuracy of the material they recite, but that's not my point. My point is that LLMs obviously seem to make up completely new entities in generated code that do not have any merit in reality. I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Mine certainly does not. Does it generate non-sense that needs serious revision and filtering in general? Of course. But not in the style of new facts, just thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world.
- TeMPOraL 3y ago> I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Yes! Very often. That's because it's exactly "thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world". In my actual code writing practice, I've spotted this manifesting in several ways. One, my subconscious often autocompletes some function name that it feels should be there (or would be nice to have it), which is immediately checked by the actual autocomplete of the IDE. Could be something like: str.cou <-- oh, there's no count() str.size() <-- there we go! Or: str.split <-- ?!? ah, it's C++, std::string does not have a split() method. Same with enum values, and many other things. If I pay attention, I can observe my mind producing these kinds of completions, which are often immediately shot down by some other process (involving memory?), by looking at the typed out thing and feeling it's wrong, or worst case, by the IDE or the compiler. Another case is when I write new code and I know functions don't exist - I just write what my stream of thoughts provide, and either clean it out later, or actually implement the functions that it "hallucinated".
- starbugs 3y ago> str.cou <-- oh, there's no count() > str.size() <-- there we go! and > str.split <-- ?!? ah, it's C++, std::string > does not have a split() method. Both confuse two existing entities from different languages/frameworks, which is exactly my point. These entities exist, they are not made up at all. They are just out of context. Ironically, the LLMs I used did rarely run into this kind of mistake (confusing similar entities of a different context). As an example of what I got when asking a popular LLM for code that traverses the macOS accessibility UI element hierarchy: It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. Now I will admit that my mind is capable of doing the same. Under severe intoxication :) On a side note: std::string should totally have a split() method and count() is more appropriate (as in intuitive) than size(). So I would say your brain is more "right" than the language in this case :) Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject?