3 ms·

> My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being ac

> My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things. There is a key difference between making up new facts and copying 'bullshit' picked up somewhere else. I agree that more than 50% of people don't put much effort into the accuracy of the material they recite, but that's not my point. My point is that LLMs obviously seem to make up completely new entities in generated code that do not have any merit in reality. I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Mine certainly does not. Does it generate non-sense that needs serious revision and filtering in general? Of course. But not in the style of new facts, just thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world.

TeMPOraL 3y ago
> I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Yes! Very often. That's because it's exactly "thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world". In my actual code writing practice, I've spotted this manifesting in several ways. One, my subconscious often autocompletes some function name that it feels should be there (or would be nice to have it), which is immediately checked by the actual autocomplete of the IDE. Could be something like: str.cou <-- oh, there's no count() str.size() <-- there we go! Or: str.split <-- ?!? ah, it's C++, std::string does not have a split() method. Same with enum values, and many other things. If I pay attention, I can observe my mind producing these kinds of completions, which are often immediately shot down by some other process (involving memory?), by looking at the typed out thing and feeling it's wrong, or worst case, by the IDE or the compiler. Another case is when I write new code and I know functions don't exist - I just write what my stream of thoughts provide, and either clean it out later, or actually implement the functions that it "hallucinated".
- starbugs 3y ago
  > str.cou <-- oh, there's no count() > str.size() <-- there we go! and > str.split <-- ?!? ah, it's C++, std::string > does not have a split() method. Both confuse two existing entities from different languages/frameworks, which is exactly my point. These entities exist, they are not made up at all. They are just out of context. Ironically, the LLMs I used did rarely run into this kind of mistake (confusing similar entities of a different context). As an example of what I got when asking a popular LLM for code that traverses the macOS accessibility UI element hierarchy: It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. Now I will admit that my mind is capable of doing the same. Under severe intoxication :) On a side note: std::string should totally have a split() method and count() is more appropriate (as in intuitive) than size(). So I would say your brain is more "right" than the language in this case :) Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject?
  - TeMPOraL 3y ago
    > It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. I've seen this happening too, but it still feel more like a case of a generalized "there is something like this, or at least should be". In some cases, it was clear to me it was guessing at some reasonable patterns - like, upon seeing function calls like InitializeFoo() or AddBar(), it would assume existence of functions UninitializeFoo() and RemoveBar(). But in other cases, more similar to yours, it does invent stuff that doesn't exist (though perhaps should). I find the latter most common when asking for Emacs instructions or Emacs Lisp code. GPT-4 is prone to inventing functions based on the prompt, such as e.g. `org-timestamp-diff-day' (or something similar), when I asked it for the command that computes difference between Org Mode timestamps in days. This function does not exist, but there are a few functions like `org-timestamp-[something]', and even a few like `org-timestamp-[sth]-day' - and more generally, Emacs/ELisp code tends to be full of functions named like the very thing you're trying to do, like `kill-whole-line' or `move-end-of-line', `kill-comment' and `duplicate-line', so it's not that surprising GPT-4 would, the other day, give me something like `kill-whole-comment-and-duplicate-line'. > Now I will admit that my mind is capable of doing the same. Under severe intoxication :) Mine too. It's not a bad analogy. I've observed in myself that moderate intoxication does magic when it comes to letting my "inner voice" drive the conversation. I sometimes experienced the feeling of my conscious mind being too late to catch sentences produced by my "inner voice" before they exited my mouth. Another analogy I particularly like is, GPT-4 is actually quite like a ~4 year old kid in many ways. One that has somehow consumed half the internet, but still with child-like attention span and propensity to continue talking and making shit up even long after crossing the point of lack of knowledge. My daughter turned 4 lasts week. The way she makes stuff up is quite similar to LLM "hallucinations" (except it's easier to tell, because she's only heard so much in her life, so she's less good at making up stuff that sounds believable). And earlier than that - between the age of 2.5 and 4 - I could observe what's best described as a "rolling context window" growing. A little more than half a year ago, I could tell hers is about 30 seconds long - anything she invented and said once would be forgotten after about that time, unless it was repeated in between by someone, possibly herself. And believe it, she talked in an extremely repetitive way back then - ~every third sentence was 50-100% repeating the critical things said earlier - as if, intuitively, she was trying to overcome her own "context window" limit. > Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject? Sure. I suppose we should also agree on the same understanding of "real understanding". I'm only proposing that LLMs pick up the same kind of "understanding" your unconscious/subconscious mind does, and produce output of similar nature. This implies that, to replicate human reasoning/performance, we'll need to layer some additional models/systems on top of the LLM.

TeMPOraL 3y ago

> I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Yes! Very often. That's because it's exactly "thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world". In my actual code writing practice, I've spotted this manifesting in several ways. One, my subconscious often autocompletes some function name that it feels should be there (or would be nice to have it), which is immediately checked by the actual autocomplete of the IDE. Could be something like: str.cou <-- oh, there's no count() str.size() <-- there we go! Or: str.split <-- ?!? ah, it's C++, std::string does not have a split() method. Same with enum values, and many other things. If I pay attention, I can observe my mind producing these kinds of completions, which are often immediately shot down by some other process (involving memory?), by looking at the typed out thing and feeling it's wrong, or worst case, by the IDE or the compiler. Another case is when I write new code and I know functions don't exist - I just write what my stream of thoughts provide, and either clean it out later, or actually implement the functions that it "hallucinated".

starbugs 3y ago
> str.cou <-- oh, there's no count() > str.size() <-- there we go! and > str.split <-- ?!? ah, it's C++, std::string > does not have a split() method. Both confuse two existing entities from different languages/frameworks, which is exactly my point. These entities exist, they are not made up at all. They are just out of context. Ironically, the LLMs I used did rarely run into this kind of mistake (confusing similar entities of a different context). As an example of what I got when asking a popular LLM for code that traverses the macOS accessibility UI element hierarchy: It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. Now I will admit that my mind is capable of doing the same. Under severe intoxication :) On a side note: std::string should totally have a split() method and count() is more appropriate (as in intuitive) than size(). So I would say your brain is more "right" than the language in this case :) Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject?
- TeMPOraL 3y ago
  > It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. I've seen this happening too, but it still feel more like a case of a generalized "there is something like this, or at least should be". In some cases, it was clear to me it was guessing at some reasonable patterns - like, upon seeing function calls like InitializeFoo() or AddBar(), it would assume existence of functions UninitializeFoo() and RemoveBar(). But in other cases, more similar to yours, it does invent stuff that doesn't exist (though perhaps should). I find the latter most common when asking for Emacs instructions or Emacs Lisp code. GPT-4 is prone to inventing functions based on the prompt, such as e.g. `org-timestamp-diff-day' (or something similar), when I asked it for the command that computes difference between Org Mode timestamps in days. This function does not exist, but there are a few functions like `org-timestamp-[something]', and even a few like `org-timestamp-[sth]-day' - and more generally, Emacs/ELisp code tends to be full of functions named like the very thing you're trying to do, like `kill-whole-line' or `move-end-of-line', `kill-comment' and `duplicate-line', so it's not that surprising GPT-4 would, the other day, give me something like `kill-whole-comment-and-duplicate-line'. > Now I will admit that my mind is capable of doing the same. Under severe intoxication :) Mine too. It's not a bad analogy. I've observed in myself that moderate intoxication does magic when it comes to letting my "inner voice" drive the conversation. I sometimes experienced the feeling of my conscious mind being too late to catch sentences produced by my "inner voice" before they exited my mouth. Another analogy I particularly like is, GPT-4 is actually quite like a ~4 year old kid in many ways. One that has somehow consumed half the internet, but still with child-like attention span and propensity to continue talking and making shit up even long after crossing the point of lack of knowledge. My daughter turned 4 lasts week. The way she makes stuff up is quite similar to LLM "hallucinations" (except it's easier to tell, because she's only heard so much in her life, so she's less good at making up stuff that sounds believable). And earlier than that - between the age of 2.5 and 4 - I could observe what's best described as a "rolling context window" growing. A little more than half a year ago, I could tell hers is about 30 seconds long - anything she invented and said once would be forgotten after about that time, unless it was repeated in between by someone, possibly herself. And believe it, she talked in an extremely repetitive way back then - ~every third sentence was 50-100% repeating the critical things said earlier - as if, intuitively, she was trying to overcome her own "context window" limit. > Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject? Sure. I suppose we should also agree on the same understanding of "real understanding". I'm only proposing that LLMs pick up the same kind of "understanding" your unconscious/subconscious mind does, and produce output of similar nature. This implies that, to replicate human reasoning/performance, we'll need to layer some additional models/systems on top of the LLM.