3 ms·
What kinds of problems in what domains did you test o1 models with? I found that they are good at logic and math problems but still hallucinate. I didn’t try t
by nopinsight 2y ago
What kinds of problems in what domains did you test o1 models with?
I found that they are good at logic and math problems but still hallucinate. I didn’t try to stretch test them with hard problems though.
- llm_trw 2y agoFinding a path between two vertices when given an itinerary of all the edges in a general graph, exactly what I said in the OP.
- mkl 2y agoDid you try asking them to write a program to do it?
- andrepd 2y agoGP is trying to test the ability of LLMs to perform mathematical tasks, not their ability to store geeks4geeks pages.
- llm_trw 2y agoNot sure why you're being downvoted that is exactly why I'm using that simple problem to benchmark LLMs. If an LLM can't figure out how to traverse a graph in its working memory then it has no hope of figuring out how to structure a proof. Under natural deduction all proofs are sub trees of the graph which is induced by the inference rules from the premise. Right now LLMs can't even do a linear proof if it gets too long when given all the induced vertices.