4 ms·
What do you use to benchmark?
by bloomingkales 2y ago
What do you use to benchmark?
- noosphr 2y agoIt was a SAT solver with the symbolic expression fed into the LLM. Using the SAT solver I'd rate how well the LLM was able to solve a given boolean formula, if it got it wrong it would get a second prompt to ask it to split the boolean expression at the main connective into two sub expressions. If it got that right it would be asked to solve the sub expressions etc.. Then reward/punish based on how well it did overall. There was meant to be a lot more to the system than that, but I never had the training budget to do anything but the first draft of the system.
- bloomingkales 2y agoI was hoping for something more reproducible. Free work, I know.
- noosphr 2y agoThis was done as part of a contract and like I said I never got permission to write a paper about it, let alone share code.
- bloomingkales 2y agoNot even a hint?