3 ms·

Sometimes it helps to have it outline the task first with a prompt like "Suggest good guidelines for [task]" and then "Follow those guidelines for [task]" once

by Kye 2y ago

Sometimes it helps to have it outline the task first with a prompt like "Suggest good guidelines for [task]" and then "Follow those guidelines for [task]" once you check to make sure the guidelines are good. LLMs aren't good at inventing processes on the fly, but they are good at writing plausible-sounding and often even correct processes, which they then follow. Applied to ChatGPT's 4o: > "Suggest good guidelines for generating ~500 filenames given a (human written) summary of each document’s contents" > "Pretend you have this data and create 10 example file names." > "Financial_Report_Q1_2024.xlsx (Summary: "Quarterly financial performance for Q1 2024, including revenue and expenses.")" > "Marketing_Strategy_SocialMedia_2024.docx (Summary: "Comprehensive marketing strategy focusing on social media growth for 2024.")" > "HR_Employee_Handbook_v2.1.pdf (Summary: "Updated version of the company employee handbook with revised policies.")" And so on. In general, you have to either create the process or have it create a process and refine it if it isn't quite there. You can even use standard processes and guidelines when they're available. Then it usually does a great job.

chefandy 2y ago
Interesting strategy— I’ll check it out. I get a bit annoyed that I have to essentially trick a tool into giving me useful answers. What’s the chance that a nontechnical user that was just sold a subscription to a Magic answer-generating machine would even realize they needed to figure out how to do things like that? LLM chatbots are amazing but they’re lagging behind the marketing inferences, and that’s a recipe for frustration that providers need to address. I think a lot of these problems need to be fixed with product design and communication.
- Kye 2y ago
  It does seem like some of the messaging gets away from their actual documentation: https://platform.openai.com/docs/guides/reasoning-best-practices https://platform.openai.com/docs/guides/reasoning-best-pract... The reasoning models improve on this kind of thing, but that just means they do even better when provided with or asked to create and follow a process. o3-mini-high, since it shows its work, didn't just spit out a process. It showed the process of creating the process in its reasoning block. It considered the format above in there and removed and added things while considering different levels of detail before the final version before producing anything.

by Kye 2y ago

chefandy 2y ago

Interesting strategy— I’ll check it out. I get a bit annoyed that I have to essentially trick a tool into giving me useful answers. What’s the chance that a nontechnical user that was just sold a subscription to a Magic answer-generating machine would even realize they needed to figure out how to do things like that? LLM chatbots are amazing but they’re lagging behind the marketing inferences, and that’s a recipe for frustration that providers need to address. I think a lot of these problems need to be fixed with product design and communication.

Kye 2y ago
It does seem like some of the messaging gets away from their actual documentation: https://platform.openai.com/docs/guides/reasoning-best-practices https://platform.openai.com/docs/guides/reasoning-best-pract... The reasoning models improve on this kind of thing, but that just means they do even better when provided with or asked to create and follow a process. o3-mini-high, since it shows its work, didn't just spit out a process. It showed the process of creating the process in its reasoning block. It considered the format above in there and removed and added things while considering different levels of detail before the final version before producing anything.