4 ms·

I just spent a few minutes doing a comparison between Phind and GPT-4 for a very high-level question on a distributed job queue. I gave them both the same fairl

I just spent a few minutes doing a comparison between Phind and GPT-4 for a very high-level question on a distributed job queue. I gave them both the same fairly vague sketch of a kind of system I would like to build. Here are my impressions: In the positives of Phind: * Phind was able, even eager, to recommend specific libraries relevant to the implementation. The recommendations matched my own research. GPT-4 takes some coaxing to get it to recommend libraries. Phind also provided sample code using the libraries it recommended. * Phind provides copious relevant sources including github, stackoverflow and others. This is a major advantage, especially if you use these AI assistants as a jumping off ground for further research. * Phind provides recommendations for follow on questions that were very good. One suggestion to the Phind team: don't remove the alternate follow on questions once I select one. A couple of times it recommended a few really good follow up questions but as soon as I selected one the others disappear. In the positives of GPT-4: * GPT-4 gave better answers. This is my subjective opinion (obviously) but if I was interviewing two candidates for a job position and using my question as the basis for a systems-design interview then GPT-4 was just overall better. In many cases it added context beyond my question, recommending things like logging and metrics for example. It seemed to intuit the "question behind the question" in a much better way than the literal interpretation of Phind. This is probably highly case-dependent, sometimes I just want an answer to my explicit question. But GPT-4 seemed to understand the broader context of the question and replied with that in mind leading to an overall more relevant response. * GPT-4 handled follow-up questions better. This is similar to the previous point - but GPT-4 gave me the impression of narrowing down the scope of the discussion based on the context of my follow-up question. It seemed to "understand" the direction of the conversation in a way that felt like it was following context. NOTE: this was not a test on coding capability (e.g. implementing algorithms) but on using these AI coding assistants as sounding boards for high-level design and architecture decisions.

X6S1x6Okd1st 3y ago
> * Phind provides copious relevant sources including github, stackoverflow and others. This is a major advantage, especially if you use these AI assistants as a jumping off ground for further research. Did you find them to be correct?
- pbhjpbhj 3y ago
  I don't use Phind for coding, except occasionally, but I like it best for generalised tech search because each para has a reference and there's a list of references down the side -- often the references would really be sufficient for me on their own. I've had one glaring error, I can't quite remember the details, but it switched the names/characteristics of two different processes (ie was exactly opposite in what it said); it was something to do with instruction caching and TLB, IIRC. I assumed you'd was a problem with the input corpus not allowing antonyms to be disambiguated. Anyway, for me it's the best of the LLM tools I have access to and had mostly replaced search engine (Google, Dukgo) for my tech-related work. I've only used chat.openai.com (free), bing chat, HuggingChat.
- zoogeny 3y ago
  I don't think "correct" is the right word since these were open ended systems design type questions. There are many ways to accomplish the same task. I also spent about 20 minutes on this which is why I mentioned this is a first impression. I'll leave it to researchers to develop a "relevancy" metric and objectively apply it. In my experience, the sources were sufficiently relevant based on its responses. They were about as relevant as equivalent Google queries. Some tiny, tiny niggles, like I was explicit I wanted it to recommend approaches in Go and for one reference I recall related to distributed locking mechanisms it provided a reference to an implementation in Java. However, that is completely fine for me since the context was more about the locking on the database side and not really the implementation in a specific language.
  - foobarbecue 3y ago
    And the sources actually existed? i.e. there weren't any made-up ones?
    - zoogeny 3y ago
      The sources are urls to the cited page (e.g. stackoverflow.com, pkg.go.dev). In the side-bar next to the answer is a more standard search-result style link list with pulled quotes from the pages (like a Google search). I didn't click every single link (as I mentioned, the citations are copious) but the few I did follow went to relevant articles. I just went back and randomly clicked several more and they all went to pages that exist and mostly relate to the content of the answer. The inline citations seem a bit more on-topic compared to the side bar which does seem more like the links were lifted directly from a search engine. To be fair there are some lower-quality blog-spammy kinda stuff - more or less the same kind of thing you would get out of Google. But compared to GPT-4, which provides no sources whatsoever, it is an advantage IMO.
fthd 3y ago
mind providing some of the prompts you use to question them?
- zoogeny 3y ago
  Here are the conversation logs: https://chat.openai.com/share/867ff0c4-d4cf-4af9-a785-31a599059d4a https://chat.openai.com/share/867ff0c4-d4cf-4af9-a785-31a599... https://www.phind.com/search?cache=ej8pn1dfjjwfr1tgc6ybwhlg https://www.phind.com/search?cache=ej8pn1dfjjwfr1tgc6ybwhlg NOTE: there are a few more question/answer blocks in the phind conversation since I was testing out the follow up question feature.
webappguy 3y ago
Do you have custom instructions? Everyone needs to mention and post prompts else entirely antidotal
- rushingcreek 3y ago
  We support custom instructions at https://phind.com/profile https://phind.com/profile.
  - bredren 3y ago
    I’m trying to get it to answer only in executable Python. I used the template with instructions I use for my system prompt on gpt4. And I tried using the additional context field for the same. It gets to writing the expected code but it still wants to include formatted headings instead of commenting those out so the entire response is executable Python. As a follow up I provided an example heading with the hash out front. It didn’t work. Any ideas on how to get it to this? Fwiw, gpt4 often if ignores this request, but only about half the time. When it does it is typically a single block of explanatory text. For that, I include prose detection and commenting as part of my post processing. Also, I don’t see it easily, but do you have an API for this or is it intended to be run by the user?
    - rushingcreek 3y ago
      Getting it to not output additional text is not something that it can do super well at the moment, unfortunately. We'll work on that.
    - soulofmischief 3y ago
      My trick for this has been one-shot training + regex. I tell the model to produce executable code within triple backticks suffixed by a keyword, like: ```keyword // code ``` and then I just ignore anything outside of those blocks.
- zoogeny 3y ago
  I did not have custom instructions for either assistant. You can see the full conversation logs which I posted as a reply to another comment.
m3kw9 3y ago
The “give context” part has a lot to do with prompting well based on the model. To have a fair comparison there should be just code and see what that come up with
fartasanelk 3y ago
would you be able to share your prompt(s)? Edit: they are already posted as comment in this thread.
idonotknowwhy 3y ago
This is a good point about GPT-4, it can intuit the "question behind the question" really well compared with other models. And it's been profoundly useful for me with the most random tasks I knew nothing about prior (like fixing a wall in my house), etc.
- Ghexor 3y ago
  That's probably because OpenAi can train on the (succesfull) conversations we have with ChatGPT!

X6S1x6Okd1st 3y ago

> * Phind provides copious relevant sources including github, stackoverflow and others. This is a major advantage, especially if you use these AI assistants as a jumping off ground for further research. Did you find them to be correct?

pbhjpbhj 3y ago
I don't use Phind for coding, except occasionally, but I like it best for generalised tech search because each para has a reference and there's a list of references down the side -- often the references would really be sufficient for me on their own. I've had one glaring error, I can't quite remember the details, but it switched the names/characteristics of two different processes (ie was exactly opposite in what it said); it was something to do with instruction caching and TLB, IIRC. I assumed you'd was a problem with the input corpus not allowing antonyms to be disambiguated. Anyway, for me it's the best of the LLM tools I have access to and had mostly replaced search engine (Google, Dukgo) for my tech-related work. I've only used chat.openai.com (free), bing chat, HuggingChat.
zoogeny 3y ago
I don't think "correct" is the right word since these were open ended systems design type questions. There are many ways to accomplish the same task. I also spent about 20 minutes on this which is why I mentioned this is a first impression. I'll leave it to researchers to develop a "relevancy" metric and objectively apply it. In my experience, the sources were sufficiently relevant based on its responses. They were about as relevant as equivalent Google queries. Some tiny, tiny niggles, like I was explicit I wanted it to recommend approaches in Go and for one reference I recall related to distributed locking mechanisms it provided a reference to an implementation in Java. However, that is completely fine for me since the context was more about the locking on the database side and not really the implementation in a specific language.
- foobarbecue 3y ago
  And the sources actually existed? i.e. there weren't any made-up ones?
  - zoogeny 3y ago
    The sources are urls to the cited page (e.g. stackoverflow.com, pkg.go.dev). In the side-bar next to the answer is a more standard search-result style link list with pulled quotes from the pages (like a Google search). I didn't click every single link (as I mentioned, the citations are copious) but the few I did follow went to relevant articles. I just went back and randomly clicked several more and they all went to pages that exist and mostly relate to the content of the answer. The inline citations seem a bit more on-topic compared to the side bar which does seem more like the links were lifted directly from a search engine. To be fair there are some lower-quality blog-spammy kinda stuff - more or less the same kind of thing you would get out of Google. But compared to GPT-4, which provides no sources whatsoever, it is an advantage IMO.

fthd 3y ago

mind providing some of the prompts you use to question them?

zoogeny 3y ago
Here are the conversation logs: https://chat.openai.com/share/867ff0c4-d4cf-4af9-a785-31a599059d4a https://chat.openai.com/share/867ff0c4-d4cf-4af9-a785-31a599... https://www.phind.com/search?cache=ej8pn1dfjjwfr1tgc6ybwhlg https://www.phind.com/search?cache=ej8pn1dfjjwfr1tgc6ybwhlg NOTE: there are a few more question/answer blocks in the phind conversation since I was testing out the follow up question feature.

webappguy 3y ago

Do you have custom instructions? Everyone needs to mention and post prompts else entirely antidotal

rushingcreek 3y ago
We support custom instructions at https://phind.com/profile https://phind.com/profile.
- bredren 3y ago
  I’m trying to get it to answer only in executable Python. I used the template with instructions I use for my system prompt on gpt4. And I tried using the additional context field for the same. It gets to writing the expected code but it still wants to include formatted headings instead of commenting those out so the entire response is executable Python. As a follow up I provided an example heading with the hash out front. It didn’t work. Any ideas on how to get it to this? Fwiw, gpt4 often if ignores this request, but only about half the time. When it does it is typically a single block of explanatory text. For that, I include prose detection and commenting as part of my post processing. Also, I don’t see it easily, but do you have an API for this or is it intended to be run by the user?
  - rushingcreek 3y ago
    Getting it to not output additional text is not something that it can do super well at the moment, unfortunately. We'll work on that.
  - soulofmischief 3y ago
    My trick for this has been one-shot training + regex. I tell the model to produce executable code within triple backticks suffixed by a keyword, like: ```keyword // code ``` and then I just ignore anything outside of those blocks.
zoogeny 3y ago
I did not have custom instructions for either assistant. You can see the full conversation logs which I posted as a reply to another comment.

m3kw9 3y ago

The “give context” part has a lot to do with prompting well based on the model. To have a fair comparison there should be just code and see what that come up with

fartasanelk 3y ago

would you be able to share your prompt(s)? Edit: they are already posted as comment in this thread.

idonotknowwhy 3y ago

This is a good point about GPT-4, it can intuit the "question behind the question" really well compared with other models. And it's been profoundly useful for me with the most random tasks I knew nothing about prior (like fixing a wall in my house), etc.

Ghexor 3y ago
That's probably because OpenAi can train on the (succesfull) conversations we have with ChatGPT!