3 ms·

I've personally found frameworks like this to get in the way of quality COT: It's rare for a prompt that takes great advantage of the LLM's reasoning to fit in

I've personally found frameworks like this to get in the way of quality COT: It's rare for a prompt that takes great advantage of the LLM's reasoning to fit in the format these generators encourage A friend mentioned how terrible most cold email generators are at actually generating natural feeling emails. It just took asking him questions about how actual people in marketing come up with emails to come up with a chain of thought that produces intentionally uncanny emails for a wide range of inputs: https://rentry.co/54hbz https://rentry.co/54hbz It's not like you can't technically fit what I described into bunch of comments (or an obnoxiously long multiline comment), but it'd be bulky and not conducive to general happiness of anyone involved. I much prefer repurposing Nunjucks templates to keep all of that a separate document that's easy to manage with version control

jackmpcollins 3y ago
With magentic you could do chain-of-thought in two or more steps: one function that generates a string output containing the chain-of-thought reasoning and answer, and a second that takes that output and converts it to the final answer object. I agree though that this is not encouraged or made obvious by the framework. The approach I'm encouraging with this is to write many functions to achieve your goal. So in the case of your email writing example you might have some of the following prompt-functions - write key bullet points for email about xyz -> list[str] - write email based on bullet points -> str - generate feedback for email to meet criteria abc -> str - update email based on feedback -> str - does email meet all criteria abc -> bool And between these you could have regular python code check things like blacklist/whitelist of keywords, length of paragraphs, and even add hardcoded strings to the feedback based on these checks.
- BoorishBears 3y ago
  Why would you add a second function for the answer object when you can return an answer object in the same response as the chain of thought? Overall your second approach makes for really terrible UX and dramatically weakens the performance at the task unless you go and repeat every single definition along the way: ensuring you now have X copies of the prompt spread across the code base and have blown up your token count. Once you get to that level of granularity between calls, you've pretty much fall back into doing a slower more expensive version of NLP pre-ChatGPT.

jackmpcollins 3y ago

With magentic you could do chain-of-thought in two or more steps: one function that generates a string output containing the chain-of-thought reasoning and answer, and a second that takes that output and converts it to the final answer object. I agree though that this is not encouraged or made obvious by the framework. The approach I'm encouraging with this is to write many functions to achieve your goal. So in the case of your email writing example you might have some of the following prompt-functions - write key bullet points for email about xyz -> list[str] - write email based on bullet points -> str - generate feedback for email to meet criteria abc -> str - update email based on feedback -> str - does email meet all criteria abc -> bool And between these you could have regular python code check things like blacklist/whitelist of keywords, length of paragraphs, and even add hardcoded strings to the feedback based on these checks.

BoorishBears 3y ago
Why would you add a second function for the answer object when you can return an answer object in the same response as the chain of thought? Overall your second approach makes for really terrible UX and dramatically weakens the performance at the task unless you go and repeat every single definition along the way: ensuring you now have X copies of the prompt spread across the code base and have blown up your token count. Once you get to that level of granularity between calls, you've pretty much fall back into doing a slower more expensive version of NLP pre-ChatGPT.