4 ms·

But is it a step to greater rigor? Or is it an illusion of rigor? They talk about improving tokenization but I don't believe that's the fundamental problem of

But is it a step to greater rigor? Or is it an illusion of rigor? They talk about improving tokenization but I don't believe that's the fundamental problem of controlling LLMs. The problem with LLMs is all the data comes in as (tokenized) language and the result is nothing but in-context predicted output. That's where all the "prompt-injection" exploits come from - as well as the hallucinations, "temper tantrums" and so-forth.

startupsfail 3y ago
It is not a step towards greater rigor. They literally have magical thinking and “biblical” quotes from GPT 11:4 all other the place, mixing code and religion. And starting prompts with “You”? Seriously. Can we at least drop that as a start?
- quenix 3y ago
  > And starting prompts with “You”? Seriously. Can we at least drop that as a start? What is wrong with this?
  - startupsfail 3y ago
    “You” is completely unnecessary. What needs to be defined is the content of the language being modeled, not the model itself. And if there is an attempt to define the model itself, then this definition should be correct, should not contradict anything and should be useful. Otherwise it’s just dead code, waiting to create problems.
    - kurisufag 3y ago
      > Otherwise it’s just dead code, waiting to create problems it's very possible that the pretense improves results: most recorded interactions /are/ between two people, after all.
      - dolmen 3y ago
        Examples: HNN, StackOverflow, Reddit...
    - jameshart 3y ago
      I definitely agree with this. When a language model is dealing with a paragraph of text that says something like: You are standing in an open field west of a white house, with a boarded front door. There is a small mailbox here. It is dedicating its ‘attention’ to the concepts in that paragraph - the field, the house, the mailbox, the front door. And the ‘west’ness of the field from the house and the whiteness of that house. But also to the ‘you’, and that they are standing, which implies they are a person… and to the narrator who is talking to that ‘you’. That that narrator is speaking in English in second person present tense, in a style reminiscent of a text adventure… All sorts of connotations from this text activating neurons with different weights making it more or less likely to think that the word ‘xyzzy’ or ‘grue’ might be appropriate to output soon. Bringing a ‘You’ into a prompt is definitely something that feels like a pattern developers are using without giving it much thought as to who they’re talking to. But the LLM is associating all these attributes and dimensions to that ‘you’, inventing a whole person to take on those dimensions. Is that the best use of its scarce attention? Does it help the prompt produce the desired output? Does the LLM think it’s outputting text from an adventure game? Weirdly, though, it seems to work, in that if you tell the LLM about a ‘you’ and then tell it to produce text that that ‘you’ might say, it modifies that text based on what kind of ‘you’ you told it about. But that is a weird way to proceed. There must be others.
    - TeMPOraL 3y ago
      > “You” is completely unnecessary. It isn't, for at least two main reasons: 1) In LLMs, every token has some degree of influence on the output. Starting the prompt with "You" and writing it in second person attracts the model towards specific volumes in the latent space. This can have good or bad impact on the output, depending on the model. 2) Instruct-type models are fine-tuned to respond to second-person prompts. "You"-prompts are what those models expect. If you're working with a model that isn't instruction-tuned, use whatever you want.
      - startupsfail 3y ago
        Have you tried removing it and checking the results? Could it be that this is a cargo cult, people using You, simply because it was present in the ChatGPT prompt at the time it got leaked?
  - pxtail 3y ago
    I'm not interested in pleasant, formal "conversation" with the thing roleplaying as human and wasting, time, keystrokes and money, I want data as fast and condensed as possible without dumb fluff. Yes, it's funny for few first times but not much after that
    - Nevermark 3y ago
      If you come across a model that gives you better results with pleasant wordier prompts, then just create a polite standard pre-prompt that lets the model know the conversation is to be terse, clear, factual, and direct as possible, without any unnecessary social or creative flourishes. I mean, whatever gets the best results is what gets the best results, right? It's not a question of "funny" or "fluff".
jameshart 3y ago
The result is actually richer than ‘predicted output’ - it’s a probability distribution over all possible output. Having richer ways to consume that probability distribution than just ‘take the most likely thing, after adding some noise’ is more conducive to using LLMs to generate output that can be further processed - in rigorous ways. Like by running it through a compiler. Think about how when you’re coding, autocomplete suggestions help you pick the right ‘next token’ with greater accuracy.
- cubefox 3y ago
  Note that for any fine-tuned models (like GPT-4, where the foundation model has not been made accessible) the model does no longer give the "probabilities" of the next tokens, but rather their "goodness". Where the numbers say how good a token would be relative to the aims the model inferred from its fine-tuning.
  - brookst 3y ago
    Isn’t that the same thing? The non-fine-tuned models also have assumptions based on corpus and training. I don’t think there’s such a thing as a purely objective probability of the next token.
    - cubefox 3y ago
      It's very different. We don't know exactly what the model consideres good after fine-tuning (which can lead to surprising cases of misalignment), while the probability that something is the next token in the training distribution is very clear. I don't know how they measure it, but they can apparently measure the "loss" which (I think) says how close the model is to some sort of real probability.
      - brookst 3y ago
        What I meant was, fine tuning is not substantially different from training. It seems odd to use different words for the resulting systems.
two_in_one 3y ago
> That's where all the "prompt-injection" exploits come Giving access to LLM is like giving access to console, or any other application. Not safe in general. The application by itself should be limited and sandboxed. Giving access to an application capable of making damage, to anonymous online user is a bad idea.

startupsfail 3y ago

It is not a step towards greater rigor. They literally have magical thinking and “biblical” quotes from GPT 11:4 all other the place, mixing code and religion. And starting prompts with “You”? Seriously. Can we at least drop that as a start?

quenix 3y ago
> And starting prompts with “You”? Seriously. Can we at least drop that as a start? What is wrong with this?
- startupsfail 3y ago
  “You” is completely unnecessary. What needs to be defined is the content of the language being modeled, not the model itself. And if there is an attempt to define the model itself, then this definition should be correct, should not contradict anything and should be useful. Otherwise it’s just dead code, waiting to create problems.
  - kurisufag 3y ago
    > Otherwise it’s just dead code, waiting to create problems it's very possible that the pretense improves results: most recorded interactions /are/ between two people, after all.
    - dolmen 3y ago
      Examples: HNN, StackOverflow, Reddit...
  - jameshart 3y ago
    I definitely agree with this. When a language model is dealing with a paragraph of text that says something like: You are standing in an open field west of a white house, with a boarded front door. There is a small mailbox here. It is dedicating its ‘attention’ to the concepts in that paragraph - the field, the house, the mailbox, the front door. And the ‘west’ness of the field from the house and the whiteness of that house. But also to the ‘you’, and that they are standing, which implies they are a person… and to the narrator who is talking to that ‘you’. That that narrator is speaking in English in second person present tense, in a style reminiscent of a text adventure… All sorts of connotations from this text activating neurons with different weights making it more or less likely to think that the word ‘xyzzy’ or ‘grue’ might be appropriate to output soon. Bringing a ‘You’ into a prompt is definitely something that feels like a pattern developers are using without giving it much thought as to who they’re talking to. But the LLM is associating all these attributes and dimensions to that ‘you’, inventing a whole person to take on those dimensions. Is that the best use of its scarce attention? Does it help the prompt produce the desired output? Does the LLM think it’s outputting text from an adventure game? Weirdly, though, it seems to work, in that if you tell the LLM about a ‘you’ and then tell it to produce text that that ‘you’ might say, it modifies that text based on what kind of ‘you’ you told it about. But that is a weird way to proceed. There must be others.
  - TeMPOraL 3y ago
    > “You” is completely unnecessary. It isn't, for at least two main reasons: 1) In LLMs, every token has some degree of influence on the output. Starting the prompt with "You" and writing it in second person attracts the model towards specific volumes in the latent space. This can have good or bad impact on the output, depending on the model. 2) Instruct-type models are fine-tuned to respond to second-person prompts. "You"-prompts are what those models expect. If you're working with a model that isn't instruction-tuned, use whatever you want.
    - startupsfail 3y ago
      Have you tried removing it and checking the results? Could it be that this is a cargo cult, people using You, simply because it was present in the ChatGPT prompt at the time it got leaked?
- pxtail 3y ago
  I'm not interested in pleasant, formal "conversation" with the thing roleplaying as human and wasting, time, keystrokes and money, I want data as fast and condensed as possible without dumb fluff. Yes, it's funny for few first times but not much after that
  - Nevermark 3y ago
    If you come across a model that gives you better results with pleasant wordier prompts, then just create a polite standard pre-prompt that lets the model know the conversation is to be terse, clear, factual, and direct as possible, without any unnecessary social or creative flourishes. I mean, whatever gets the best results is what gets the best results, right? It's not a question of "funny" or "fluff".

jameshart 3y ago

The result is actually richer than ‘predicted output’ - it’s a probability distribution over all possible output. Having richer ways to consume that probability distribution than just ‘take the most likely thing, after adding some noise’ is more conducive to using LLMs to generate output that can be further processed - in rigorous ways. Like by running it through a compiler. Think about how when you’re coding, autocomplete suggestions help you pick the right ‘next token’ with greater accuracy.

cubefox 3y ago
Note that for any fine-tuned models (like GPT-4, where the foundation model has not been made accessible) the model does no longer give the "probabilities" of the next tokens, but rather their "goodness". Where the numbers say how good a token would be relative to the aims the model inferred from its fine-tuning.
- brookst 3y ago
  Isn’t that the same thing? The non-fine-tuned models also have assumptions based on corpus and training. I don’t think there’s such a thing as a purely objective probability of the next token.
  - cubefox 3y ago
    It's very different. We don't know exactly what the model consideres good after fine-tuning (which can lead to surprising cases of misalignment), while the probability that something is the next token in the training distribution is very clear. I don't know how they measure it, but they can apparently measure the "loss" which (I think) says how close the model is to some sort of real probability.
    - brookst 3y ago
      What I meant was, fine tuning is not substantially different from training. It seems odd to use different words for the resulting systems.

two_in_one 3y ago

> That's where all the "prompt-injection" exploits come Giving access to LLM is like giving access to console, or any other application. Not safe in general. The application by itself should be limited and sandboxed. Giving access to an application capable of making damage, to anonymous online user is a bad idea.