3 ms·

Comprehensive and practical write-up that aligns with most of my experiences. One controversial point that has led to discussions in my team is this: > A comm

Comprehensive and practical write-up that aligns with most of my experiences. One controversial point that has led to discussions in my team is this: > A common anti-pattern/code smell in software is the “God Object,” where we have a single class or function that does everything. The same applies to prompts too. In theory, a monolithic agent/prompt with infinite context size, a large toolset, and perfect attention would be ideal. Multi-agent systems will always be less effective and more error-prone than monolithic systems on a given problem because of less context of the overall problem. Individual agents work best when they have entirely different functionalities. I wrote down my thoughts about agent architectures here: https://www.kadoa.com/blog/ai-agents-hype-vs-reality https://www.kadoa.com/blog/ai-agents-hype-vs-reality

tedsanders 2y ago
As an OpenAI employee who has worked with dozens of API customers, I mostly agree with the article's tip to break up tasks into smaller, more reliable subtasks. If each step of your task requires knowledge of the big picture, then yeah it ought to help to put all your context into a single API call. But if you can decompose your task into relatively independent subtasks, then it helps to use a custom prompt/custom model for each of those steps. Extraneous context and complexity are just opportunities for the model to make mistakes, and the more you can strip those out, the better. 3 steps with 99% reliability are better than 1 step with 90% reliability. Of course, it all depends on what you're trying to do. I'd say single, big API calls are better when: - Much of the information/substeps are interrelated - You want immediate output for a user-facing app, without having to wait for intermediate steps Multiple, sequenced API calls are better when: - You can decompose the task into smaller steps, each of which do not require full context - There's a tree or graph of steps, and you want to prune irrelevant branches as you proceed from the root - You want to have some 100% reliabile logic live outside of the LLM in parsing/routing code - You want to customize the prompts based on results from previous steps
- umangrathi 2y ago
  smaller tasks also helps in choosing smaller models to work with, instead of waiting for a large model to respond (really not usable when doing customer facing work)
- hubraumhugo 2y ago
  Agree, that's a very good summary. Would love to see some benchmarks for the two approaches.
- 7d7n 2y ago
  100% agree with Ted's take. One of the authors wrote about splitting up prompts here too: https://eugeneyan.com/writing/prompting/#split-catch-all-prompts-into-multiple-smaller-ones https://eugeneyan.com/writing/prompting/#split-catch-all-pro...
sjducb 2y ago
I wonder how helpful “prompt unit tests” would be here? Write the initial prompt, and write some tests to validate the output of the prompt. Then as the prompt grows you can observe the decline in performance at the initial task. Then you can decide if the new larger prompt is worth the decline in performance at it’s initial task. It might not work for all tasks, but a good candidate would be - write SQL queries from natural language.

tedsanders 2y ago

As an OpenAI employee who has worked with dozens of API customers, I mostly agree with the article's tip to break up tasks into smaller, more reliable subtasks. If each step of your task requires knowledge of the big picture, then yeah it ought to help to put all your context into a single API call. But if you can decompose your task into relatively independent subtasks, then it helps to use a custom prompt/custom model for each of those steps. Extraneous context and complexity are just opportunities for the model to make mistakes, and the more you can strip those out, the better. 3 steps with 99% reliability are better than 1 step with 90% reliability. Of course, it all depends on what you're trying to do. I'd say single, big API calls are better when: - Much of the information/substeps are interrelated - You want immediate output for a user-facing app, without having to wait for intermediate steps Multiple, sequenced API calls are better when: - You can decompose the task into smaller steps, each of which do not require full context - There's a tree or graph of steps, and you want to prune irrelevant branches as you proceed from the root - You want to have some 100% reliabile logic live outside of the LLM in parsing/routing code - You want to customize the prompts based on results from previous steps

umangrathi 2y ago
smaller tasks also helps in choosing smaller models to work with, instead of waiting for a large model to respond (really not usable when doing customer facing work)
hubraumhugo 2y ago
Agree, that's a very good summary. Would love to see some benchmarks for the two approaches.
7d7n 2y ago
100% agree with Ted's take. One of the authors wrote about splitting up prompts here too: https://eugeneyan.com/writing/prompting/#split-catch-all-prompts-into-multiple-smaller-ones https://eugeneyan.com/writing/prompting/#split-catch-all-pro...

sjducb 2y ago

I wonder how helpful “prompt unit tests” would be here? Write the initial prompt, and write some tests to validate the output of the prompt. Then as the prompt grows you can observe the decline in performance at the initial task. Then you can decide if the new larger prompt is worth the decline in performance at it’s initial task. It might not work for all tasks, but a good candidate would be - write SQL queries from natural language.