4 ms·

I use DSPy often, and it’s the only framework that I have much interest in using professionally. Evaluations are first class and have a natural place in optimi

I use DSPy often, and it’s the only framework that I have much interest in using professionally. Evaluations are first class and have a natural place in optimization. I still usually spend some time adjusting initial prompts, but more time doing traditional ML things… like working with SMEs, building training sets, evaluating models and developing the pipeline. If you’re an ML engineer that’s frustrated by the “loose” nature of developing applications with LLMs, I recommend trying it out. With assertions and suggestions, there’s also additional pathways you can use to enforce constraints on the output and build in requirements from your customer.

qeternity 2y ago
What do you actually use it for? I've never been able to actually get it to perform on anything remotely close to what it claims. Sure, it can help optimize few shot prompting...but what else can it reliably do?
- deepsquirrelnet 2y ago
  It isn’t for every application, but I’ve used it for tasks like extraction, summarization and generating commands where you have specific constraints you’re trying to meet. Most important to me is that I can write evaluations based on feedback from the team and build them into the pipeline using suggestions and track them with LLM as a judge (and other) metrics. With some of the optimizers, you can use stronger models to help propose and test new instructions for your student model to follow, as well as optimize the N shot examples to use in the prompt (MIPROv2 optimizer). It’s not that a lot of that can’t be done other ways, but as a framework it provides a non-trivial amount of value to me when I’m trying to keep track of requirements that grow over time instead of playing the whack a mole game in the prompt.
  - qeternity 2y ago
    Yeah so like I said, I get the in-context optimization bit…which is nice, but pretty limited. I have had precisely zero success with the LLM-prompt-writer elements. I would love to be wrong, but DSpy makes huge promises and falls painfully short on basically all of them. I do not see any reason to use it.
huevosabio 2y ago
Every time I check the docs, I feel like it obfuscates so many things that it puts me off and I decide to just not try it out. Behind the scenes it's using LLM's to find the proper prompting. I find that it uses a terminology and abstraction that is way too complicated for what it is.

qeternity 2y ago

What do you actually use it for? I've never been able to actually get it to perform on anything remotely close to what it claims. Sure, it can help optimize few shot prompting...but what else can it reliably do?

deepsquirrelnet 2y ago
It isn’t for every application, but I’ve used it for tasks like extraction, summarization and generating commands where you have specific constraints you’re trying to meet. Most important to me is that I can write evaluations based on feedback from the team and build them into the pipeline using suggestions and track them with LLM as a judge (and other) metrics. With some of the optimizers, you can use stronger models to help propose and test new instructions for your student model to follow, as well as optimize the N shot examples to use in the prompt (MIPROv2 optimizer). It’s not that a lot of that can’t be done other ways, but as a framework it provides a non-trivial amount of value to me when I’m trying to keep track of requirements that grow over time instead of playing the whack a mole game in the prompt.
- qeternity 2y ago
  Yeah so like I said, I get the in-context optimization bit…which is nice, but pretty limited. I have had precisely zero success with the LLM-prompt-writer elements. I would love to be wrong, but DSpy makes huge promises and falls painfully short on basically all of them. I do not see any reason to use it.

huevosabio 2y ago

Every time I check the docs, I feel like it obfuscates so many things that it puts me off and I decide to just not try it out. Behind the scenes it's using LLM's to find the proper prompting. I find that it uses a terminology and abstraction that is way too complicated for what it is.