7 ms·

> considering "out-of-context reasons" and taking a wider perspective is pretty much the defining aspect of LLMs as general-purpose AI tools. "out-of-context"

by saulpw 1y ago

> considering "out-of-context reasons" and taking a wider perspective is pretty much the defining aspect of LLMs as general-purpose AI tools. "out-of-context" literally means that the reason isn't in its context. Even if it can make the leap that the number should be zero if it's a bank holiday, how would an LLM know that yesterday was a one-off bank holiday? A human would only know through their lived experience that the markets were shut down, the news was making a big deal over it, etc. It's the same problem using cheap human labor in a different region of the world for this kind of thing; they can perform the mechanical task, but they don't have the context to detect the myriad of ways it can go subtly wrong.

ben_w 1y ago
That kind of knowledge is present in the training set, doesn't need to be in the context or system prompt. That said, I too would only use an LLM today in the same kinds of role that five years ago would be outsourced to a different culture. Culture, not even language: this is how you get the difference between "biscuits and gravy" as understood in the UK vs in the USA.
- TeMPOraL 1y ago
  LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.
  - alganet 1y ago
    No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNiMmJZ https://imgur.com/a/RNiMmJZ They should have answered that such comparisons are potentially offensive, and explained why academia thinks so, _before_ spilling out nonsense.
    - stevenhuang 1y ago
      There legitimately is a lot of cross over between Egyptian mythology and other high strangeness phenomenon as understood culturally though, such as aliens/ufos. I think you did just demonstrate you know less about culture than LLMs, which is not at all unsurprising.
      - alganet 1y ago
        Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source. This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran? Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them). You're wrong.
        TeMPOraL 1y ago
        > Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them). Nah, that's just LLMs being trained to acquiesce to the insanity of the last ~15 years, as many people seem to expect that claiming you're offended by something is an ultimate argument that everyone must yield to (and they'll keep making a fuss out of it until they do).
        alganet 1y ago
        An LLM that offends people and entire nations is what many would classify as _misaligned_. Let's say I have a company, and my company needs to comply to government policy regarding communication. I cannot trust LLMs then, they will either fail to follow policy, or acquiesce to any group that tries to game it. It's useless garbage. Egyptian myths were just an example, you don't need to bite it so hard.
        TeMPOraL 1y ago
  - ben_w 1y ago
    Looking at your other comment sibling to mine, I think part of the difficulty discussing these topics is how much these things are considered isolated magic artefacts (bad for engineering) or one tool amongst many where the magic word is "synergy". So I agree with you: LLMs do know all written cultures on the internet and can mimic them acceptably — but they only actually do so when this is requested by some combination of the fine-tuning, RLHF, system prompt, and context. In your example, having some current news injected, which is easy, but actually requires someone to plumb that in. And as you say, you'd not do that unless you thought you needed to. But even easier to pick, lower-hanging fruit, often gets missed. When the "dangerous sycophancy" behaviour started getting in the news, I updated my custom ChatGPT "traits" setting to this: Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea. But cultural differences can be subtle, and there's a long tail of cultural traits of the same kind that means 1980s Text Adventure NLP doesn't scale to what ChatGPT itself does. While this can still be solved with fine-tuning or getting your staff to RLHF it, the number of examples current AI need in order to learn is high compared to a real human, so it won't learn your corporate culture from experience *as fast* as a new starter within your team, unless you're a sufficiently big corporation that it can be on enough (I don't know how many exactly) teams within your company at the same time.
TeMPOraL 1y ago
> "out-of-context" literally means that the reason isn't in its context. Even if it can make the leap that the number should be zero if it's a bank holiday, how would an LLM know that yesterday was a one-off bank holiday? Depends. Was it a one-off holiday announced at 11th our or something? Then it obviously won't know. You'd need extra setup to enable it to realize that, such as e.g. first feeding an LLM the context of your task and a digest of news stories spanning a week, asking it to find if there's anything potentially relevant, and then appending that output to the LLM calls doing the work. It's not something you'd do by default in general case, but that's only because tokens cost money and context space is scarce. Is it a regular bank holiday? Then all it would need is today's date in the context, which is often just appended somewhere between system and user prompts, along with e.g. user location data. I see that by "out-of-context reasons" you meant the first case; I read it as a second. In the second case, the "out-of-context" bit could be the fact that a bank holiday could alter the entry for that day; if that rule is important or plausible enough but not given explicitly in the prompt, the model will learn it during training, and will likely connect the dots. This is what I meant as the "defining aspect of LLMs as general-purpose AI tools". The flip side is, when it connects the dots when it shouldn't, we say it's hallucinating.

by saulpw 1y ago

ben_w 1y ago

That kind of knowledge is present in the training set, doesn't need to be in the context or system prompt. That said, I too would only use an LLM today in the same kinds of role that five years ago would be outsourced to a different culture. Culture, not even language: this is how you get the difference between "biscuits and gravy" as understood in the UK vs in the USA.

TeMPOraL 1y ago
LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.
- alganet 1y ago
  No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNiMmJZ https://imgur.com/a/RNiMmJZ They should have answered that such comparisons are potentially offensive, and explained why academia thinks so, _before_ spilling out nonsense.
  - stevenhuang 1y ago
    There legitimately is a lot of cross over between Egyptian mythology and other high strangeness phenomenon as understood culturally though, such as aliens/ufos. I think you did just demonstrate you know less about culture than LLMs, which is not at all unsurprising.
    - alganet 1y ago
      Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source. This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran? Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them). You're wrong.
      - TeMPOraL 1y ago
        > Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them). Nah, that's just LLMs being trained to acquiesce to the insanity of the last ~15 years, as many people seem to expect that claiming you're offended by something is an ultimate argument that everyone must yield to (and they'll keep making a fuss out of it until they do).
        alganet 1y ago
        An LLM that offends people and entire nations is what many would classify as _misaligned_. Let's say I have a company, and my company needs to comply to government policy regarding communication. I cannot trust LLMs then, they will either fail to follow policy, or acquiesce to any group that tries to game it. It's useless garbage. Egyptian myths were just an example, you don't need to bite it so hard.
        TeMPOraL 1y ago
- ben_w 1y ago
  Looking at your other comment sibling to mine, I think part of the difficulty discussing these topics is how much these things are considered isolated magic artefacts (bad for engineering) or one tool amongst many where the magic word is "synergy". So I agree with you: LLMs do know all written cultures on the internet and can mimic them acceptably — but they only actually do so when this is requested by some combination of the fine-tuning, RLHF, system prompt, and context. In your example, having some current news injected, which is easy, but actually requires someone to plumb that in. And as you say, you'd not do that unless you thought you needed to. But even easier to pick, lower-hanging fruit, often gets missed. When the "dangerous sycophancy" behaviour started getting in the news, I updated my custom ChatGPT "traits" setting to this: Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea. But cultural differences can be subtle, and there's a long tail of cultural traits of the same kind that means 1980s Text Adventure NLP doesn't scale to what ChatGPT itself does. While this can still be solved with fine-tuning or getting your staff to RLHF it, the number of examples current AI need in order to learn is high compared to a real human, so it won't learn your corporate culture from experience *as fast* as a new starter within your team, unless you're a sufficiently big corporation that it can be on enough (I don't know how many exactly) teams within your company at the same time.

TeMPOraL 1y ago

> "out-of-context" literally means that the reason isn't in its context. Even if it can make the leap that the number should be zero if it's a bank holiday, how would an LLM know that yesterday was a one-off bank holiday? Depends. Was it a one-off holiday announced at 11th our or something? Then it obviously won't know. You'd need extra setup to enable it to realize that, such as e.g. first feeding an LLM the context of your task and a digest of news stories spanning a week, asking it to find if there's anything potentially relevant, and then appending that output to the LLM calls doing the work. It's not something you'd do by default in general case, but that's only because tokens cost money and context space is scarce. Is it a regular bank holiday? Then all it would need is today's date in the context, which is often just appended somewhere between system and user prompts, along with e.g. user location data. I see that by "out-of-context reasons" you meant the first case; I read it as a second. In the second case, the "out-of-context" bit could be the fact that a bank holiday could alter the entry for that day; if that rule is important or plausible enough but not given explicitly in the prompt, the model will learn it during training, and will likely connect the dots. This is what I meant as the "defining aspect of LLMs as general-purpose AI tools". The flip side is, when it connects the dots when it shouldn't, we say it's hallucinating.