4 ms·

Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, w

Question from a casual AI user, if you have a minute. It seems to me that I could get much more productive by making my own personal AI "system". For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?

gradys 2y ago
I assume you know how to program in Python? I would start with just the client libraries of the model providers you want to use. LLMs are conceptually simple when treated as black boxes. String in, string out. You don't necessarily need a framework.
monkmartinez 2y ago
Not the person you asked, but I will take a shot at your question. The best python package for getting what you want out of these systems is to use original gangster python with libraries to help with your goals. For your example; Write a python script with requests that hits the OpenAI API. You can even hardcode the API key because its just a script on your computer! Now you have the GPT-4proLight-mini-deluxe response in JSON. You can pipe that into a bazzillion and one different places including another API request to Anthropic. Once that returns, you can now have TWO llm responses to analyze. I tried haystack, langchain, txtai, langroid, CrewAI, Autogen, and more that I am forgetting. One day while I was reading r/Localllama someone wrote; "All these packages are TRASH, just write python!"... Lightbulb moment for me. Duh! Now I don't need to learn a massive framework to only use 1/363802983th of it while cursing that I can't figure out how to make it do what I want it to do. Just write python. I tell you that has been massive for my usage of these LLM's outside of the chat interfaces like LibreChat and OpenWebUI. You can even have claude or deepseek write the script for you. That often gets me within striking distance of what I really want to achieve at that moment.
digdugdirk 2y ago
I've had good luck with a light "shim layer" library that handles the actual interfacing with the api and implements the plumbing on any fun new features that get introduced. I've settled on the Mirascope library (https://mirascope.com/ https://mirascope.com/), which suits my use cases and lets me implement structured inputs/outputs via pydantic models, which is nice. I really like using it, and the team behind it is really responsive and helpful. That being said, Pydantic just released an AI library of their own (https://ai.pydantic.dev/ https://ai.pydantic.dev/) that I haven't checked out, but I'd love to hear from someone who has! Given their track record, it's certainly worth keeping an eye on.
- dr_kiszonka 2y ago
  Mirascope looks great. Thanks!
pizza 2y ago
Yes, anthropic just released model context protocol and mcp is perfect for this kind of thing. I actually wrote an mcp server for claude to call out to openai just yesterday.
kingkongjaffa 2y ago
You could use the plain api libraries for each llm and ipython notebooks, conceptually each block could be a node or link in the prompt chain, and input/output of each block is printable and visible to check which part of the chain is the part that is failing or has sub optimal outputs.
sdesol 2y ago
> For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. I'm working on a naive approach to identify errors in LLM responses which I talk about at https://news.ycombinator.com/item?id=42313401#42313990 https://news.ycombinator.com/item?id=42313401#42313990, which can be used to scrutinize responses. It's written in Javascript though, but you will be able to create a new chat by calling a http endpoint. I'm hoping to have the system in place in a couple of weeks.
jackmpcollins 2y ago
I'm building magentic for use cases like this! https://github.com/jackmpcollins/magentic https://github.com/jackmpcollins/magentic It's based on pydantic and aims to make writing LLM queries as easy/compact as possible by using type annotations, including for structured outputs and streaming. If you use it please reach out!
d4rkp4ttern 2y ago
You can have a look at Langroid -- it's an agent-oriented LLM programming framework from CMU/UW-Madison researchers. We started building it in Apr 2023 out of frustration with the bloat of then-existing libs. In langroid you set up a ChatAgent class which encapsulates an LLM-interface plus any state you'd like. There's a Task class that wraps an Agent and allows inter-agent communication and tool-handling. We have devs who've found our framework easy to understand and extend for their purposes, and some companies are using it in production (some have endorsed us publicly). A quick tour gives a flavor of Langroid: https://langroid.github.io/langroid/tutorials/langroid-tour/ https://langroid.github.io/langroid/tutorials/langroid-tour/ Feel free to drop into our discord for help.
- dr_kiszonka 2y ago
  Neat. Thanks!
OxfordOutlander 2y ago
Also not OP but adding my 2c. I tried many python frameworks but the lack of customizations and observability limited the utility. Now, I only use Instructor (Jason Liu's library). That, and concurrent futures for parallel processing.

gradys 2y ago

I assume you know how to program in Python? I would start with just the client libraries of the model providers you want to use. LLMs are conceptually simple when treated as black boxes. String in, string out. You don't necessarily need a framework.

monkmartinez 2y ago

Not the person you asked, but I will take a shot at your question. The best python package for getting what you want out of these systems is to use original gangster python with libraries to help with your goals. For your example; Write a python script with requests that hits the OpenAI API. You can even hardcode the API key because its just a script on your computer! Now you have the GPT-4proLight-mini-deluxe response in JSON. You can pipe that into a bazzillion and one different places including another API request to Anthropic. Once that returns, you can now have TWO llm responses to analyze. I tried haystack, langchain, txtai, langroid, CrewAI, Autogen, and more that I am forgetting. One day while I was reading r/Localllama someone wrote; "All these packages are TRASH, just write python!"... Lightbulb moment for me. Duh! Now I don't need to learn a massive framework to only use 1/363802983th of it while cursing that I can't figure out how to make it do what I want it to do. Just write python. I tell you that has been massive for my usage of these LLM's outside of the chat interfaces like LibreChat and OpenWebUI. You can even have claude or deepseek write the script for you. That often gets me within striking distance of what I really want to achieve at that moment.

digdugdirk 2y ago

I've had good luck with a light "shim layer" library that handles the actual interfacing with the api and implements the plumbing on any fun new features that get introduced. I've settled on the Mirascope library (https://mirascope.com/ https://mirascope.com/), which suits my use cases and lets me implement structured inputs/outputs via pydantic models, which is nice. I really like using it, and the team behind it is really responsive and helpful. That being said, Pydantic just released an AI library of their own (https://ai.pydantic.dev/ https://ai.pydantic.dev/) that I haven't checked out, but I'd love to hear from someone who has! Given their track record, it's certainly worth keeping an eye on.

dr_kiszonka 2y ago
Mirascope looks great. Thanks!

pizza 2y ago

Yes, anthropic just released model context protocol and mcp is perfect for this kind of thing. I actually wrote an mcp server for claude to call out to openai just yesterday.

kingkongjaffa 2y ago

You could use the plain api libraries for each llm and ipython notebooks, conceptually each block could be a node or link in the prompt chain, and input/output of each block is printable and visible to check which part of the chain is the part that is failing or has sub optimal outputs.

sdesol 2y ago

> For example, write a simple pipeline where Claude would scrutinize OpenAI's answers and vice versa. I'm working on a naive approach to identify errors in LLM responses which I talk about at https://news.ycombinator.com/item?id=42313401#42313990 https://news.ycombinator.com/item?id=42313401#42313990, which can be used to scrutinize responses. It's written in Javascript though, but you will be able to create a new chat by calling a http endpoint. I'm hoping to have the system in place in a couple of weeks.

jackmpcollins 2y ago

I'm building magentic for use cases like this! https://github.com/jackmpcollins/magentic https://github.com/jackmpcollins/magentic It's based on pydantic and aims to make writing LLM queries as easy/compact as possible by using type annotations, including for structured outputs and streaming. If you use it please reach out!

d4rkp4ttern 2y ago

You can have a look at Langroid -- it's an agent-oriented LLM programming framework from CMU/UW-Madison researchers. We started building it in Apr 2023 out of frustration with the bloat of then-existing libs. In langroid you set up a ChatAgent class which encapsulates an LLM-interface plus any state you'd like. There's a Task class that wraps an Agent and allows inter-agent communication and tool-handling. We have devs who've found our framework easy to understand and extend for their purposes, and some companies are using it in production (some have endorsed us publicly). A quick tour gives a flavor of Langroid: https://langroid.github.io/langroid/tutorials/langroid-tour/ https://langroid.github.io/langroid/tutorials/langroid-tour/ Feel free to drop into our discord for help.

dr_kiszonka 2y ago
Neat. Thanks!

OxfordOutlander 2y ago

Also not OP but adding my 2c. I tried many python frameworks but the lack of customizations and observability limited the utility. Now, I only use Instructor (Jason Liu's library). That, and concurrent futures for parallel processing.