4 ms·
How do you prefer to validate if a model is actually useful for you in practice outside of solving toy problems? Are you asking these models to solve reasoning
by ASinclair 3y ago
How do you prefer to validate if a model is actually useful for you in practice outside of solving toy problems? Are you asking these models to solve reasoning problems like this to get any benefit for yourself in your day to day use? Or do you even care if the models are useful for day to day tasks?
- jonplackett 3y agoI also tried it with a bunch of my previous got4 requests and it didn’t even understand. A few of them that gpt4 was very helpful with
- ASinclair 3y agoGot it. Thanks.
- kolinko 3y agoIt’s the simplest metric to see how much of the model is reasoning and hiw much is just rehashing. If it can’t solve puzzles then it’s more likely to make mistakes when understanding questions that are out of its training set.
- TillE 3y agoYeah I get the instinct to poke at LLMs, they're fun toys, but it's always weird to see so much focus on stuff like logic problems. I've used Bard for creative brainstorming, for real factual questions, for translating .pot files, etc, and it's done pretty well.
- amanzi 3y agoI was going to ask the same question... I've been using Bard for everyday tasks for a while now and it's as good and sometimes better that GPT-4 (I pay for a Pro subscription). Someone ruling out an LLM because it couldn't answer one question, speaks more to them than the LLM capabilities. Just yesterday I was using both GPT-4 and Bard to figure out an IPv6 routing issue. I pasted in the same questions and the same troubleshooting logs to both, and Bard was able to help me solve it quicker than GPT-4.
- huytersd 3y agoFor me the validation process is to use it for a few weeks and then I have a good handle on what it can handle and what it can’t.
- p1esk 3y agoI asked it the same question I was asking GPT4 yesterday (to explain details of a complicated ML model quantization method, based on a screenshot image of equations from a pdf). I actually need this for work - I'm trying to design an algorithm. I felt Ultra provided a very similar quality answer to GPT4.