2 ms·

That's just a No True Prompt Engineer fallacy. You can get substantially improved generation results for either LLMs or Diffusion models with proper prompt engi

That's just a No True Prompt Engineer fallacy. You can get substantially improved generation results for either LLMs or Diffusion models with proper prompt engineering, even with GPT-4. You can do prompt term weighting with offline LLMs (e.g. compel: https://github.com/damian0815/compel https://github.com/damian0815/compel ) or averaging the embed tokens yourself and passing the embedding matrix to the model, but unlike with image generation where the results of prompt weighting are more obvious, there isn't as much of a need for it for LLMs.

Der_Einzige 3y ago
Sorry, but compel seems to only work with diffusion models. Note that they have no examples of running this with modern LLMs. It only supports a fraction of the techniques that, for example Automatic1111 supports for "real" prompt engineering. Even if I'm wrong and compel does work with, any transformer model, it still only supports a small fraction of all the "real" prompt engineering techniques. I shouldn't have to "average the embed tokens myself". I know you're a big deal in the industry, but you're SO WRONG about the idea that "there isn't as much of a need for it for LLMs". I hate that the NLP community has such huge blindspots for this kind of stuff. Anything that gives further levers of control has massive improvements to the capabilities of an LLM. My github gist proves that all of these techniques found in automatic1111 work with NLP models, yet no one implements it. I think it's because of the potential for breaking alignment techniques with them. And yes, I will fight and die on this hill even if Yann Lecun and Christopher Manning tell me I'm wrong. I know I'm right.
- minimaxir 3y ago
  > My github gist proves that all of these techniques found in automatic1111 work with NLP models No it doesn't, your gist just shows that it's possible to implement, which I'm not disputing. And even then, your example with GPT-2 has the comment "GPT2 is very tempermental with this technique" and requires a temperature of 20 to behave which is an accommodation that can never be used in a real application. If you can create test cases and demos where prompt weighting in a LLM results in a distinct observable improvement as it does with diffusion models, that would be a different story. I'd love to be proven wrong but your gist doesn't do it.

Der_Einzige 3y ago

Sorry, but compel seems to only work with diffusion models. Note that they have no examples of running this with modern LLMs. It only supports a fraction of the techniques that, for example Automatic1111 supports for "real" prompt engineering. Even if I'm wrong and compel does work with, any transformer model, it still only supports a small fraction of all the "real" prompt engineering techniques. I shouldn't have to "average the embed tokens myself". I know you're a big deal in the industry, but you're SO WRONG about the idea that "there isn't as much of a need for it for LLMs". I hate that the NLP community has such huge blindspots for this kind of stuff. Anything that gives further levers of control has massive improvements to the capabilities of an LLM. My github gist proves that all of these techniques found in automatic1111 work with NLP models, yet no one implements it. I think it's because of the potential for breaking alignment techniques with them. And yes, I will fight and die on this hill even if Yann Lecun and Christopher Manning tell me I'm wrong. I know I'm right.

minimaxir 3y ago
> My github gist proves that all of these techniques found in automatic1111 work with NLP models No it doesn't, your gist just shows that it's possible to implement, which I'm not disputing. And even then, your example with GPT-2 has the comment "GPT2 is very tempermental with this technique" and requires a temperature of 20 to behave which is an accommodation that can never be used in a real application. If you can create test cases and demos where prompt weighting in a LLM results in a distinct observable improvement as it does with diffusion models, that would be a different story. I'd love to be proven wrong but your gist doesn't do it.