4 ms·

I am a heavy user of GPT4, and Phind was surprisingly able to match GPT4 on several initial programming tasks I gave it. Given the large context window of Phind

by drcode 3y ago

I am a heavy user of GPT4, and Phind was surprisingly able to match GPT4 on several initial programming tasks I gave it. Given the large context window of Phind, it will likely be able to outperform GPT4 for some tasks. That is quite an accomplishment, I am impressed

rushingcreek 3y ago
[dead]
iandanforth 3y ago
FWIW The default context window of GPT-4 via ChatGPT is about to change to 32k.
- drcode 3y ago
  that would put them significantly ahead again, for my use cases
  - rushingcreek 3y ago
    We will eventually increase the Phind Model to 100K tokens -- the RoPE embeddings in Code Llama were designed for this.
    - arugulum 3y ago
      > the RoPE embeddings in Code Llama were designed for this. The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE (e.g. position interpolation) are post-hoc tweaks (with optional tuning) to an entirely vanilla RoPE implementation.
    - m3kw9 3y ago
      Is it “100k” or really 100k there are so many ways to do context, I remember seeing 100k before but it was doing some cheap trick to get it
    - antupis 3y ago
      100k tokens and good ide support would be great. Copy pasting back and forth with browser and IDE is kinda annoying and you always miss some context. I think model is now good enough but what is kinda missing is good developer experience eg what to load in that context window and how model integrates to IDE. But this is kinda missing with copilot and chatgpt4 as well.
    - razodactyl 3y ago
      What about ALiBi and Sliding Window Attention? Additionally Apple researchers seem to be playing with "Attention Free" variants.
- ComplexSystems 3y ago
  This would be great if true. Any source for this?
  - iandanforth 3y ago
    https://twitter.com/DataChaz/status/1719660354743976342 https://twitter.com/DataChaz/status/1719660354743976342
- jeswin 3y ago
  Given the number of times it just fails with large prompts on 32k contexts, I'm not sure if they'ready for this. In my experience, if you're consuming 20k+ tokens failure rate is more than 50%.
  - ComputerGuru 3y ago
    Well over 50%, at least via the api, for me.
- jstummbillig 3y ago
  Source?

by drcode 3y ago

rushingcreek 3y ago

[dead]

iandanforth 3y ago

FWIW The default context window of GPT-4 via ChatGPT is about to change to 32k.

drcode 3y ago
that would put them significantly ahead again, for my use cases
- rushingcreek 3y ago
  We will eventually increase the Phind Model to 100K tokens -- the RoPE embeddings in Code Llama were designed for this.
  - arugulum 3y ago
    > the RoPE embeddings in Code Llama were designed for this. The RoPE embeddings were not "designed" for that. The original RoPE was not designed with length extrapolation in mind. Subsequent tweaks to extrapolate RoPE (e.g. position interpolation) are post-hoc tweaks (with optional tuning) to an entirely vanilla RoPE implementation.
  - m3kw9 3y ago
    Is it “100k” or really 100k there are so many ways to do context, I remember seeing 100k before but it was doing some cheap trick to get it
  - antupis 3y ago
    100k tokens and good ide support would be great. Copy pasting back and forth with browser and IDE is kinda annoying and you always miss some context. I think model is now good enough but what is kinda missing is good developer experience eg what to load in that context window and how model integrates to IDE. But this is kinda missing with copilot and chatgpt4 as well.
  - razodactyl 3y ago
    What about ALiBi and Sliding Window Attention? Additionally Apple researchers seem to be playing with "Attention Free" variants.
ComplexSystems 3y ago
This would be great if true. Any source for this?
- iandanforth 3y ago
  https://twitter.com/DataChaz/status/1719660354743976342 https://twitter.com/DataChaz/status/1719660354743976342
jeswin 3y ago
Given the number of times it just fails with large prompts on 32k contexts, I'm not sure if they'ready for this. In my experience, if you're consuming 20k+ tokens failure rate is more than 50%.
- ComputerGuru 3y ago
  Well over 50%, at least via the api, for me.
jstummbillig 3y ago
Source?