9 ms·

DSPy – Programming–not prompting–LMs

189 points by ulrischa 2y ago

byefruit 2y ago
I've seen a couple of talks on DSPy and tried to use it for one of my projects but the structure always feels somewhat strained. It seems to be suited for tasks that are primarily show, don't tell but what do you do when you have significant prior instruction you want to tell? e.g Tests I want applied to anything retrieved from the database. What I'd like is to optimise the prompt around those (or maybe even the tests themselves) but I can't seem to express that in DSPy signatures.
- thatsadude 2y ago
  You can optimize prompt with MIPROv2 without examples (set the max number of examples to 0)
  - vergessenmir 2y ago
    And have the chat history end up as an input field instead of each request response being prepared individually for the LLM endpoint? I'm new to the library but from what I can see the Chat adapter will do this automatically if I use the forward call Quoting the docs: "Though rarely needed, you can write custom LMs by inheriting from dspy.BaseLM. Another advanced layer in the DSPy ecosystem is that of adapters, which sit between DSPy signatures and LMs. A future version of this guide will discuss these advanced features, though you likely don't need them." I could be wrong, I could be looking for complexity where there is none. Have a fundamentally misunderstood how this all works, it sometimes feels like I have?
thom 2y ago
There are companies that will charge you 6-7 figures for what you can do in a few dozen lines with DSPy, but I guess that’s true of many things.
- 3abiton 2y ago
  That is a big claim though, I am not sure about that from my experience. What am I missing?
behnamoh 2y ago
Every time I've seen a dspy article, I end up thinking: ok, but what does it do exactly? I've been using guidance, outlines, GBF grammars, etc. What advantage does dspy have over those alternatives? I've learnt that the best package to use LLMs is just Python. These "LLM packages" just make it harder to do customizations as they all make opinionated assumptions and decisions.
- dr_kiszonka 2y ago
  Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?
  - gradys 2y ago
    I assume you know how to program in Python? I would start with just the client libraries of the model providers you want to use. LLMs are conceptually simple when treated as black boxes. String in, string out. You don't necessarily need a framework.
  - monkmartinez 2y ago
    Not the person you asked, but I will take a shot at your question. The best python package for getting what you want out of these systems is to use original gangster python with libraries to help with your goals. For your example; Write a python script with requests that hits the OpenAI API. You can even hardcode the API key because its just a script on your computer! Now you have the GPT-4proLight-mini-deluxe response in JSON. You can pipe that into a bazzillion and one different places including another API request to Anthropic. Once that returns, you can now have TWO llm responses to analyze. I tried haystack, langchain, txtai, langroid, CrewAI, Autogen, and more that I am forgetting. One day while I was reading r/Localllama someone wrote; "All these packages are TRASH, just write python!"... Lightbulb moment for me. Duh! Now I don't need to learn a massive framework to only use 1/363802983th of it while cursing that I can't figure out how to make it do what I want it to do. Just write python. I tell you that has been massive for my usage of these LLM's outside of the chat interfaces like LibreChat and OpenWebUI. You can even have claude or deepseek write the script for you. That often gets me within striking distance of what I really want to achieve at that moment.
scosman 2y ago
Can someone explain what DSPy does that fine tuning doesn’t? Structured IO, optimized to better results. Sure. But why just just go straight to weights, instead of trying to optimize the few-shot space?
- flakiness 2y ago
  It has multiple optimization strategies. One is optimizing the few shot list. Another is to let the model write prompts and pick the best one based on the given eval. I doubt latter much more intriguing although I have no idea how practical it is.
- choppaface 2y ago
  The main idea behind DSPy is that you can’t modify the weights, but you can perhaps modify the prompts. DSPy’s original primary customer was multi-llm-agent systems where you have a chain / graph of LLM calls (perhaps mostly or all to OpenAI GPT) and you have some metric (perhaps vague) that you want to increase. While the idea may seem a bit weird, there have been various success stories, such as a UoT team winning medical-notes-oriented competition using DSPy https://arxiv.org/html/2404.14544v1 https://arxiv.org/html/2404.14544v1
deepsquirrelnet 2y ago
I use DSPy often, and it’s the only framework that I have much interest in using professionally. Evaluations are first class and have a natural place in optimization. I still usually spend some time adjusting initial prompts, but more time doing traditional ML things… like working with SMEs, building training sets, evaluating models and developing the pipeline. If you’re an ML engineer that’s frustrated by the “loose” nature of developing applications with LLMs, I recommend trying it out. With assertions and suggestions, there’s also additional pathways you can use to enforce constraints on the output and build in requirements from your customer.
- qeternity 2y ago
  What do you actually use it for? I've never been able to actually get it to perform on anything remotely close to what it claims. Sure, it can help optimize few shot prompting...but what else can it reliably do?
  - deepsquirrelnet 2y ago
    It isn’t for every application, but I’ve used it for tasks like extraction, summarization and generating commands where you have specific constraints you’re trying to meet. Most important to me is that I can write evaluations based on feedback from the team and build them into the pipeline using suggestions and track them with LLM as a judge (and other) metrics. With some of the optimizers, you can use stronger models to help propose and test new instructions for your student model to follow, as well as optimize the N shot examples to use in the prompt (MIPROv2 optimizer). It’s not that a lot of that can’t be done other ways, but as a framework it provides a non-trivial amount of value to me when I’m trying to keep track of requirements that grow over time instead of playing the whack a mole game in the prompt.
    - qeternity 2y ago
      Yeah so like I said, I get the in-context optimization bit…which is nice, but pretty limited. I have had precisely zero success with the LLM-prompt-writer elements. I would love to be wrong, but DSpy makes huge promises and falls painfully short on basically all of them. I do not see any reason to use it.
beepbooptheory 2y ago
How does it work? Like I can see the goal and the results, but is it in fact the case that its still here "LLMs all the way down"? That is, is there a supplement bot here thats fine tuned to DSPy syntax, doing the actual work of turning the code to prompt? Trying to figure out how else it would work.. But if that is the case, this really feels like a Wizard of Oz behind the curtain thing.
aaronvg 2y ago
I found it interesting how DSPy created the Signatures concept: https://dspy.ai/learn/programming/signatures/ https://dspy.ai/learn/programming/signatures/ We took this kind of concept all the way to making a DSL called BAML, where prompts look like literal functions, with input and output types. Playground link here https://www.promptfiddle.com/ https://www.promptfiddle.com/ https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml (tried pasting code but the formatting is completely off here, sorry). We think we could run some optimizers on this as well in the future! We'll definitely use DSPy as inspiration!
- dcreater 2y ago
  Good dx? BAML looks even worse than the current API call based paradigm. Even your toy examples look bad - wouldn't want to see what an actual program would look like. Hopefully this, dspy and the like that have poor design, inelegant won't become common standards
  - aaronvg 2y ago
    How do you organize your prompts? Do you use a templating language like jinja? How complex are your prompts? Do you have any open source examples? I’m genuinely curious since if we can convince someone like you that BAML is amazing we’re on a good track. We’ve helped people remove really ugly concatenated strings or raw yaml files with json schemas just by using our prompt format (which uses jinja2!)
- vergessenmir 2y ago
  Wouldn't try and convince the nay sayers. I have a few comments (or opinions rather) This coming from a recent dspy user hitting its pain points. - The GitHub page is very busy - A clear example should come up early on the page. It's only when I got to the fiddle I could see a motivating example i.e the extractions, functions and tests. - Then a section for running tests/evaluations -Then deployment or run with/without the baml cli - I do wonder if all the functions have to be so tightly coupled with the model. In dspy my modules are model agnostic and I can evaluate behaviour across different models.
coffeephoenix 2y ago
I tried using it but one of the hard parts is defining a good metric that the underlying optimizer can use. Came up with an approach for that here: https://colab.research.google.com/drive/1obuS9cEWN9MT-MIv5aL1sDIYuGZEKo4r?usp=sharing https://colab.research.google.com/drive/1obuS9cEWN9MT-MIv5aL...
jmugan 2y ago
As far as I can tell, Google VertexAI prompt optimizer does similar things. I find their documentation more comprehensible. https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer https://cloud.google.com/vertex-ai/generative-ai/docs/learn/...
Imanari 2y ago
Can someone explain how it works?
thomasahle 2y ago
Here is a simple example of dspy: classify = dspy.Predict(f"text -> label:Literal{CLASSES}") optimized = dspy.BootstrapFewShot(metric=(lambda x, y: x.label == y.label)) .compile(classify, trainset=load_dataset('Banking77')) label = optimized(text="What does a pending cash withdrawal mean?").label What this does is optimize a prompt given a dataset (here Banking77). The optimizer, BootstrapFewShot, simply selects a bunch of random subsets from the training set, and measures which gives the best performance on the rest of the dataset when used as few-shot examples. There are also more fancy optimizers, including ones that first optimize the prompt, and then use the improved model as a teacher to optimize the weights. This has the advantage that you don't need to pay for a super long prompt on every inference call. dspy has more cool features, such as the ability to train a large composite LLM program "end to end", similar to backprop. The main advantage, imo, is just not having "stale" prompts everywhere in your code base. You might have written some neat few-shot examples for the middle layers of your pipeline, but then you change something at the start, and you have to manually rewrite all the examples for every other module. With dspy you just keep your training datasets around, and the rest is automated. (Note, the example above is taken from the new website: https://dspy.ai/#__tabbed_3_3 https://dspy.ai/#__tabbed_3_3 and simplified a bit)
thatsadude 2y ago
My go to framework. I wish we can use global metrics in DSPy, for examples, F1 score over the whole evaluation set (instead of a single query at the moment). The recent async support has been life saver.
dcreater 2y ago
DSPy seems unnecessarily convoluted, inelegant or am I just stupid?
- th0ma5 2y ago
  I think you read it right. It is in my mind a kind of wish casting that adding other modeling to LLMs can improve their use, but the ideas all sound like playing with your food at best, and deliberately confusing people to prey on their excitement at the worst.
  - edmundsauto 2y ago
    I'm torn - I like the promise and people are getting value out of it. I need to try it myself on a toy project! What experiences/evidence do you have that informed your opinion? It sounds like you've had pretty negative experiences.
kittikitti 2y ago
I've found DSPy to be very useful especially among larger teams working on LMs. It makes communicating parts of the AI infrastructure easier.
vergessenmir 2y ago
DSPy is both great and frustrating at times. It's designed dort prompt and response. Textgrad and DSPy share this design decision and this is where the frustration begins. I'd like the modules (dspy speak for inference ) to be aware of the chat history. Without this I'm not able to test the accuracy of the responses in the various chat contexts they appear. It's been asked for in the TextGrad & Dspy GitHub issues.
- saturatedfat 2y ago
  why can’t you feed the chat history back in at every forward call?
saturatedfat 2y ago
wrote a blog post on using DSPy and why it’s powerful. tried to keep simple, understandable, concrete. https://www.darinkishore.com/posts/mcp#building-tools-that-learn https://www.darinkishore.com/posts/mcp#building-tools-that-l...

189 points by ulrischa 2y ago

I've seen a couple of talks on DSPy and tried to use it for one of my projects but the structure always feels somewhat strained. It seems to be suited for tasks that are primarily show, don't tell but what do you do when you have significant prior instruction you want to tell? e.g Tests I want applied to anything retrieved from the database. What I'd like is to optimise the prompt around those (or maybe even the tests themselves) but I can't seem to express that in DSPy signatures.

thatsadude 2y ago
You can optimize prompt with MIPROv2 without examples (set the max number of examples to 0)
- vergessenmir 2y ago
  And have the chat history end up as an input field instead of each request response being prepared individually for the LLM endpoint? I'm new to the library but from what I can see the Chat adapter will do this automatically if I use the forward call Quoting the docs: "Though rarely needed, you can write custom LMs by inheriting from dspy.BaseLM. Another advanced layer in the DSPy ecosystem is that of adapters, which sit between DSPy signatures and LMs. A future version of this guide will discuss these advanced features, though you likely don't need them." I could be wrong, I could be looking for complexity where there is none. Have a fundamentally misunderstood how this all works, it sometimes feels like I have?

thom 2y ago

There are companies that will charge you 6-7 figures for what you can do in a few dozen lines with DSPy, but I guess that’s true of many things.

3abiton 2y ago
That is a big claim though, I am not sure about that from my experience. What am I missing?

behnamoh 2y ago

Every time I've seen a dspy article, I end up thinking: ok, but what does it do exactly? I've been using guidance, outlines, GBF grammars, etc. What advantage does dspy have over those alternatives? I've learnt that the best package to use LLMs is just Python. These "LLM packages" just make it harder to do customizations as they all make opinionated assumptions and decisions.

dr_kiszonka 2y ago
Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?
- gradys 2y ago
  I assume you know how to program in Python? I would start with just the client libraries of the model providers you want to use. LLMs are conceptually simple when treated as black boxes. String in, string out. You don't necessarily need a framework.
- monkmartinez 2y ago
  Not the person you asked, but I will take a shot at your question. The best python package for getting what you want out of these systems is to use original gangster python with libraries to help with your goals. For your example; Write a python script with requests that hits the OpenAI API. You can even hardcode the API key because its just a script on your computer! Now you have the GPT-4proLight-mini-deluxe response in JSON. You can pipe that into a bazzillion and one different places including another API request to Anthropic. Once that returns, you can now have TWO llm responses to analyze. I tried haystack, langchain, txtai, langroid, CrewAI, Autogen, and more that I am forgetting. One day while I was reading r/Localllama someone wrote; "All these packages are TRASH, just write python!"... Lightbulb moment for me. Duh! Now I don't need to learn a massive framework to only use 1/363802983th of it while cursing that I can't figure out how to make it do what I want it to do. Just write python. I tell you that has been massive for my usage of these LLM's outside of the chat interfaces like LibreChat and OpenWebUI. You can even have claude or deepseek write the script for you. That often gets me within striking distance of what I really want to achieve at that moment.

scosman 2y ago

Can someone explain what DSPy does that fine tuning doesn’t? Structured IO, optimized to better results. Sure. But why just just go straight to weights, instead of trying to optimize the few-shot space?

flakiness 2y ago
It has multiple optimization strategies. One is optimizing the few shot list. Another is to let the model write prompts and pick the best one based on the given eval. I doubt latter much more intriguing although I have no idea how practical it is.
choppaface 2y ago
The main idea behind DSPy is that you can’t modify the weights, but you can perhaps modify the prompts. DSPy’s original primary customer was multi-llm-agent systems where you have a chain / graph of LLM calls (perhaps mostly or all to OpenAI GPT) and you have some metric (perhaps vague) that you want to increase. While the idea may seem a bit weird, there have been various success stories, such as a UoT team winning medical-notes-oriented competition using DSPy https://arxiv.org/html/2404.14544v1 https://arxiv.org/html/2404.14544v1

deepsquirrelnet 2y ago

I use DSPy often, and it’s the only framework that I have much interest in using professionally. Evaluations are first class and have a natural place in optimization. I still usually spend some time adjusting initial prompts, but more time doing traditional ML things… like working with SMEs, building training sets, evaluating models and developing the pipeline. If you’re an ML engineer that’s frustrated by the “loose” nature of developing applications with LLMs, I recommend trying it out. With assertions and suggestions, there’s also additional pathways you can use to enforce constraints on the output and build in requirements from your customer.

qeternity 2y ago
What do you actually use it for? I've never been able to actually get it to perform on anything remotely close to what it claims. Sure, it can help optimize few shot prompting...but what else can it reliably do?
- deepsquirrelnet 2y ago
  It isn’t for every application, but I’ve used it for tasks like extraction, summarization and generating commands where you have specific constraints you’re trying to meet. Most important to me is that I can write evaluations based on feedback from the team and build them into the pipeline using suggestions and track them with LLM as a judge (and other) metrics. With some of the optimizers, you can use stronger models to help propose and test new instructions for your student model to follow, as well as optimize the N shot examples to use in the prompt (MIPROv2 optimizer). It’s not that a lot of that can’t be done other ways, but as a framework it provides a non-trivial amount of value to me when I’m trying to keep track of requirements that grow over time instead of playing the whack a mole game in the prompt.
  - qeternity 2y ago
    Yeah so like I said, I get the in-context optimization bit…which is nice, but pretty limited. I have had precisely zero success with the LLM-prompt-writer elements. I would love to be wrong, but DSpy makes huge promises and falls painfully short on basically all of them. I do not see any reason to use it.

beepbooptheory 2y ago

How does it work? Like I can see the goal and the results, but is it in fact the case that its still here "LLMs all the way down"? That is, is there a supplement bot here thats fine tuned to DSPy syntax, doing the actual work of turning the code to prompt? Trying to figure out how else it would work.. But if that is the case, this really feels like a Wizard of Oz behind the curtain thing.

aaronvg 2y ago

I found it interesting how DSPy created the Signatures concept: https://dspy.ai/learn/programming/signatures/ https://dspy.ai/learn/programming/signatures/ We took this kind of concept all the way to making a DSL called BAML, where prompts look like literal functions, with input and output types. Playground link here https://www.promptfiddle.com/ https://www.promptfiddle.com/ https://github.com/BoundaryML/baml https://github.com/BoundaryML/baml (tried pasting code but the formatting is completely off here, sorry). We think we could run some optimizers on this as well in the future! We'll definitely use DSPy as inspiration!

dcreater 2y ago
Good dx? BAML looks even worse than the current API call based paradigm. Even your toy examples look bad - wouldn't want to see what an actual program would look like. Hopefully this, dspy and the like that have poor design, inelegant won't become common standards
- aaronvg 2y ago
  How do you organize your prompts? Do you use a templating language like jinja? How complex are your prompts? Do you have any open source examples? I’m genuinely curious since if we can convince someone like you that BAML is amazing we’re on a good track. We’ve helped people remove really ugly concatenated strings or raw yaml files with json schemas just by using our prompt format (which uses jinja2!)
vergessenmir 2y ago
Wouldn't try and convince the nay sayers. I have a few comments (or opinions rather) This coming from a recent dspy user hitting its pain points. - The GitHub page is very busy - A clear example should come up early on the page. It's only when I got to the fiddle I could see a motivating example i.e the extractions, functions and tests. - Then a section for running tests/evaluations -Then deployment or run with/without the baml cli - I do wonder if all the functions have to be so tightly coupled with the model. In dspy my modules are model agnostic and I can evaluate behaviour across different models.

coffeephoenix 2y ago

I tried using it but one of the hard parts is defining a good metric that the underlying optimizer can use. Came up with an approach for that here: https://colab.research.google.com/drive/1obuS9cEWN9MT-MIv5aL1sDIYuGZEKo4r?usp=sharing https://colab.research.google.com/drive/1obuS9cEWN9MT-MIv5aL...

jmugan 2y ago

As far as I can tell, Google VertexAI prompt optimizer does similar things. I find their documentation more comprehensible. https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer https://cloud.google.com/vertex-ai/generative-ai/docs/learn/...

Imanari 2y ago

Can someone explain how it works?

thomasahle 2y ago

Here is a simple example of dspy: classify = dspy.Predict(f"text -> label:Literal{CLASSES}") optimized = dspy.BootstrapFewShot(metric=(lambda x, y: x.label == y.label)) .compile(classify, trainset=load_dataset('Banking77')) label = optimized(text="What does a pending cash withdrawal mean?").label What this does is optimize a prompt given a dataset (here Banking77). The optimizer, BootstrapFewShot, simply selects a bunch of random subsets from the training set, and measures which gives the best performance on the rest of the dataset when used as few-shot examples. There are also more fancy optimizers, including ones that first optimize the prompt, and then use the improved model as a teacher to optimize the weights. This has the advantage that you don't need to pay for a super long prompt on every inference call. dspy has more cool features, such as the ability to train a large composite LLM program "end to end", similar to backprop. The main advantage, imo, is just not having "stale" prompts everywhere in your code base. You might have written some neat few-shot examples for the middle layers of your pipeline, but then you change something at the start, and you have to manually rewrite all the examples for every other module. With dspy you just keep your training datasets around, and the rest is automated. (Note, the example above is taken from the new website: https://dspy.ai/#__tabbed_3_3 https://dspy.ai/#__tabbed_3_3 and simplified a bit)

thatsadude 2y ago

My go to framework. I wish we can use global metrics in DSPy, for examples, F1 score over the whole evaluation set (instead of a single query at the moment). The recent async support has been life saver.

dcreater 2y ago

DSPy seems unnecessarily convoluted, inelegant or am I just stupid?

th0ma5 2y ago
I think you read it right. It is in my mind a kind of wish casting that adding other modeling to LLMs can improve their use, but the ideas all sound like playing with your food at best, and deliberately confusing people to prey on their excitement at the worst.
- edmundsauto 2y ago
  I'm torn - I like the promise and people are getting value out of it. I need to try it myself on a toy project! What experiences/evidence do you have that informed your opinion? It sounds like you've had pretty negative experiences.

kittikitti 2y ago

I've found DSPy to be very useful especially among larger teams working on LMs. It makes communicating parts of the AI infrastructure easier.

vergessenmir 2y ago

DSPy is both great and frustrating at times. It's designed dort prompt and response. Textgrad and DSPy share this design decision and this is where the frustration begins. I'd like the modules (dspy speak for inference ) to be aware of the chat history. Without this I'm not able to test the accuracy of the responses in the various chat contexts they appear. It's been asked for in the TextGrad & Dspy GitHub issues.

saturatedfat 2y ago
why can’t you feed the chat history back in at every forward call?

saturatedfat 2y ago

wrote a blog post on using DSPy and why it’s powerful. tried to keep simple, understandable, concrete. https://www.darinkishore.com/posts/mcp#building-tools-that-learn https://www.darinkishore.com/posts/mcp#building-tools-that-l...