3 ms·
The paper doesn't make this point at all, but one thing you could do here is an AlphaGeometry-style[1] synthetic benchmark, where you have a geometry engine cra
by sbierwagen 2y ago
The paper doesn't make this point at all, but one thing you could do here is an AlphaGeometry-style[1] synthetic benchmark, where you have a geometry engine crank out a hundred million word problems, and have an LLM try to solve them.
Geometry problems have the nice property that they're easy to generate and solve mechanically, but there's no particular reason why a vanilla Transformer LLM would be any good at them, and you can have absolutely huge scale. (Unlike, say, the HumanEval benchmark, which only has 164 problems, which resulted in lots of accusations that LLMs can simply memorize the answers)
1: https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/ https://deepmind.google/discover/blog/alphageometry-an-olymp...
- darby_nine 2y agoYou'd have the second problem of trying to figure out how to relay geometry as a sequence of tokens when surely how you would encode this would affect what things you might reasonably expect an LLM to draw from it.
- jiggawatts 2y agoImage input.
- energy123 2y agoOnly if your purpose is to create the best geometry solver. If you're trying to improve the general intelligence of a frontier LLM, you're probably better off feeding in the synthetic data as some combination of raw images and text (as part of its existing tokenisation).