4 ms·
I'm impressed with the speed, really impressed, but not so much with the quality of the responses. This is a prompt I usually try with new LLMs: > Acting as an
by jamesponddotco 3y ago
I'm impressed with the speed, really impressed, but not so much with the quality of the responses. This is a prompt I usually try with new LLMs:
> Acting as an expert Go developer, write a RoundTripper that retries failed HTTP requests, both GET and POST ones.
GPT-4 takes a few tries but usually takes the POST part into account, saving the body for new retries and whatnot. Phind in the other hand, in the two or three times I tried, ignores the POST part and focus on GET only.
Maybe that problem is just too hard for LLMs? Or the prompt sucks? I'll see how it handle other things since I still have a few tries left.
- rushingcreek 3y agoThanks, can you send the cached link please? I'd also suggest trying Chat mode for questions like this, where there are unlikely to benefit from an internet search. Just tried your query now and it seemed to work well -- what are your thoughts? https://www.phind.com/search?cache=tvyrul1spovzcpwtd8phgegj https://www.phind.com/search?cache=tvyrul1spovzcpwtd8phgegj
- jamesponddotco 3y agoHere you go: https://www.phind.com/search?cache=k56i132ekpg43zdc7j5z1h1x https://www.phind.com/search?cache=k56i132ekpg43zdc7j5z1h1x I'll give chat mode a try. Didn't see that it existed until now. EDIT Chat mode didn't do much better: https://www.phind.com/agent?cache=clsxpl4t80002l008v3vjqw5j https://www.phind.com/agent?cache=clsxpl4t80002l008v3vjqw5j For the record, this is the interface I asked it to implement: https://pkg.go.dev/net/http#RoundTripper https://pkg.go.dev/net/http#RoundTripper
- rushingcreek 3y agoThanks for the links. It seems like it switched to Phind-34B, which is worse. Phind-70B seems to be able to get the right interface every time. Please make sure that it says Phind-70B at the top of the page while it's generating.
- coder543 3y ago“RoadTripper”? Or “RoundTripper”?
- jamesponddotco 3y agoOps, haha. Interesting that GPT-4 still got it right though. Phind still forgot about POST, but at least now it got the interface right. https://www.phind.com/search?cache=ipu8z1tb3bnn7nfgfibcix38 https://www.phind.com/search?cache=ipu8z1tb3bnn7nfgfibcix38
- coder543 3y agoI'm not sure what you mean that it "forgot" about POST? Even as an experienced Go developer, I looked at the code and thought it would probably work for both GET and POST. I couldn't easily see a problem, yet I had not forgotten about POST being part of the request. It's just not an obvious problem. This is absolutely what I would classify as a "brain teaser". It's a type of problem that makes an interviewer feel clever, but it's not great for actually evaluating candidates. Only on running the code did I realize that it wasn't doing anything to handle the problem of the request body, where it works on the first attempt, but the ReadCloser is empty on subsequent attempts. It looks like Phind-70B corrected this issue once it was pointed out. I've seen GPT-4 make plenty of small mistakes when generating code, so being iterative seems normal, even if GPT-4 might have this one specific brain teaser completely memorized. I am not at the point where I expect any LLM to blindly generate perfect code every time, but if it can usually correct issues with feedback from an error message, then that's still quite good.
- NicoJuicy 3y agoThat's because it's better at classifying than at generating. Eg. Tree of thoughts, ...
- xyzzy_plugh 3y agoThis isn't a brain teaser at all. It's a direct test of domain knowledge/experience. There are countless well-documented RoundTripper implementations that handle this case correctly. This is the sort of thing you whip up in three minutes and move along. To me it seems like a perfect test of LLMs. I don't need an injection of something that's worse than stackoverflow polluting the code I work on.
- shapenamer 3y agoI'm a human and I don't have the slightest idea what you're asking for.
- Powdering7082 3y agoDo you use Go? It makes sense to me
- viraptor 3y agoThe RoundTripper throws me off if anything. RetryRequest, RetryOnFailure, anything could be more descriptive.
- pmarreck 3y agoDoes anyone outside the Go community call it a "RoundTripper"? I know what a retry is (and things like exponential backoff) and what GET and POST are, but not that, but I also hate Go, so... EDIT: ah, followup replies elucidated me, it's just a goofy name for a Go-only thing