3 ms·
That ChatGPT is good at truthfully answering one kind of prompt (analytic, where the facts are contained in the prompt, so the task is translation) and bad at t
by williamcotton 4y ago
That ChatGPT is good at truthfully answering one kind of prompt (analytic, where the facts are contained in the prompt, so the task is translation) and bad at truthfully answering another kind of prompt (synthetic, where the facts are not contained in the prompt, so the task is synthesis), tells me something about the nature of language.
It turns out that setting up solutions to math problems is mainly a language translation task, followed by computation task as long as you have a way to eval() the resulting thunk:
Do not perform calculations. Do not compute the answer. Use Javascript Math and Date to perform calculations. Wait for further questions. Show your work in the DESCRIPTION. The number of steps in the %%%THUNK%%% should match the number of steps in the %%%DESCRIPTION%%%. Always answer with Javascript compatible code in the %%%THUNK%%%, including numerical form without commas (eg, 238572348723). Always answer with this JSON compatible object form, eg, {"key":%%%VALUE%%%}: { "question": "4 days a week, Laura practices martial arts for 1.5 hours. Considering a week is 7 days, what is her average practice time per day each week?", "description":["Multiplying 1.5 and 4", "Dividing the step 1 answer by 7"], "thunk": "(function() { const step1 = 1.5 * 4; const answer = step1 / 7; return answer; })()" }
Analytic augmentation is the procedure by which a question is augmented in a manner that promotes translation over synthesis. That this can be tested empirically is definitely something novel with regards to the philosophy of language!
- lamontcg 4y agoI've asked ChatGPT to take the derivative of an expression and the one it gave back to me was off by a minus sign. I tried to get it to calculate the expression for the height of a rocket flying vertically in constant gravity and the answer it gave me was just the rocket equation with h = instead of v = and I then got into a curious discussion with it about dimensional analysis where it just gaslit the crap out of me. If you think it is good at "analytic" tasks you haven't challenged it much.
- williamcotton 4y agoAnd I’ve applied these principles of analytic augmentation to make the results of otherwise synthetic prompts more reliably truthful and in a testable and reproducible manner. You seem to be thrashing around wildly while using these tools and I can’t imagine why anyone would be interested in whatever conclusions result from such lack of methodology.