I've seen a paper (I forget which, might respond below later) about GPT-4 and tool use i.e. it reliably knows when it's hit the limit of what it can do on its o

I've seen a paper (I forget which, might respond below later) about GPT-4 and tool use i.e. it reliably knows when it's hit the limit of what it can do on its own, and can successfully emit commands for external APIs to handle those aspects for it (calculator, character manipulation, etc). My hunch is if you read that, you could get "raw GPT-4" to defer to WolframAlpha via a similar mechanism (with a couple dozen lines of glue code).