3 ms·
its hard to judge how deep and unique your conjectures were. I did similar testing of GPT4, and my observation is that it starts failing after 3-4 levels of re
by riku_iki 3y ago
its hard to judge how deep and unique your conjectures were.
I did similar testing of GPT4, and my observation is that it starts failing after 3-4 levels of reasoning depth.
- tux1968 3y ago> failing after 3-4 levels of reasoning depth. That sounds more like an implementation or resource limitation, rather than an inherent limitation of the technique in general.
- riku_iki 3y agoit is not obvious to me how you came to such conclusion. LLMs got lots of investments: 10s of billions of dollars and tons of compute, maybe more than any other tech in history, and can't crack 3 steps reasoning. It sounds like tech limitation..
- naasking 3y agoNone of these systems or their training sets have been specifically tailored to tackle abstract reasoning or math, so that seems like a premature conclusion. The fact that they're decent at programming despite that is interesting.
- vikramkr 3y agoThey're also brand new and at some undetermined part of the sigmoid curve. Trying to predict where you are on the curve while in the middle of a sigmoid is a fools errand, the best you can do is make random predictions and hope you are accidentally correct so you can become a pundit later
- golol 3y agoNice to see number of levels of reasoning depth mentioned. I personally believe the size of a (well-trained) LLM determines how many steps of reasoning in sequence it can approximate. Newer models get deeper and deeper, giving them deeper reasoning context windows. My hypothesis is that you don't need infinite reasoning depth, just a bit more than GPT-4 has. I think once you can tie your output together with thinking in terms of ~10+ reasoning steps you'll be very close to hunan performance.