3 ms·

Context is 8k and it's quadratic. It "sees" everything in that window. If you want to have a long conversation try Claude or some of the 32k models. Claude uses

Context is 8k and it's quadratic. It "sees" everything in that window. If you want to have a long conversation try Claude or some of the 32k models. Claude uses a strange kind of attention that isn't always as precise but it's very good at finding key information in huge documents.

wokwokwok 3y ago
Key information is not relevant for generating code. You have to generate specific api calls with specific semantics not “high level summary” and “key information” . You have to generate token sequences that are functionally equivalent to an exact segment of the input prompt. A lower precision is less useful for code generation. How do you maintain existing code functionality when you give code + “refactor for x” as an input? I’m skeptical you’ve tried this and know what you’re talking about.
- inciampati 3y ago
  You are welcome to be skeptical and I will happily continue to use these tools for drafting code and writing every day. I usually provide documentation for the systems I want to use on input. Then I include source code of anything I'm hacking on. If none exists I build it up using "let's do this step by step" where the first step is outlining or architecture and subsequent steps fill out components. The same pattern works for long form prose. Having 100k tokens is almost enough. It means the model can attend to a ton of relevant information. Also, this works well when the system gives you patches. I wouldn't say "rewrite this entire thing to make this one change" because if it's anything non trivial it will shift. Again I see the same patterns for code and text. Ask for small changes, contained updates. Guide the model to work in small increments, using abstraction to deal with high level broad topics.
  - wokwokwok 3y ago
    Seriously? You wouldn’t use it refactor because the implementation will drift, but you’d use it to generate a patch to apply to the code? Hm. Well, I don’t think that style is relevant or practical for large scale code generation as shown even in this ~600 line example. …but, I’d love to see the details of how it could done, end to end, by someone who has done something like that successfully and is willing to prove it by sharing the prompts and outputs.
    - inciampati 3y ago
      It's very hard to ask one of these systems to copy an entire piece of text and then change one or two things in it. They will often make small errors. So it's easier to ask for patches and apply them manually to edit the code or text that's being worked on. Happy to talk more how do I find you?
- deleted 3y ago
  [deleted]

wokwokwok 3y ago

Key information is not relevant for generating code. You have to generate specific api calls with specific semantics not “high level summary” and “key information” . You have to generate token sequences that are functionally equivalent to an exact segment of the input prompt. A lower precision is less useful for code generation. How do you maintain existing code functionality when you give code + “refactor for x” as an input? I’m skeptical you’ve tried this and know what you’re talking about.

inciampati 3y ago
You are welcome to be skeptical and I will happily continue to use these tools for drafting code and writing every day. I usually provide documentation for the systems I want to use on input. Then I include source code of anything I'm hacking on. If none exists I build it up using "let's do this step by step" where the first step is outlining or architecture and subsequent steps fill out components. The same pattern works for long form prose. Having 100k tokens is almost enough. It means the model can attend to a ton of relevant information. Also, this works well when the system gives you patches. I wouldn't say "rewrite this entire thing to make this one change" because if it's anything non trivial it will shift. Again I see the same patterns for code and text. Ask for small changes, contained updates. Guide the model to work in small increments, using abstraction to deal with high level broad topics.
- wokwokwok 3y ago
  Seriously? You wouldn’t use it refactor because the implementation will drift, but you’d use it to generate a patch to apply to the code? Hm. Well, I don’t think that style is relevant or practical for large scale code generation as shown even in this ~600 line example. …but, I’d love to see the details of how it could done, end to end, by someone who has done something like that successfully and is willing to prove it by sharing the prompts and outputs.
  - inciampati 3y ago
    It's very hard to ask one of these systems to copy an entire piece of text and then change one or two things in it. They will often make small errors. So it's easier to ask for patches and apply them manually to edit the code or text that's being worked on. Happy to talk more how do I find you?
deleted 3y ago
[deleted]