4 ms·

AIConfig – source control format for gen AI prompts, models and settings

97 points by saqadri 3y ago

saqadri 3y ago
Hi HN! OP here - this is our first open source project and we are really looking for your feedback about this direction, because we haven't seen something like aiconfig in generative AI developer tools yet. Our basic premise is that AI application development should be config-based, so you can track the prompts, models and model parameters being used more rigorously. Having this AI artifact then lets you iterate on it separately from your application code, and also set up evals that provide "test coverage" for the gen AI parts of your application. We were also inspired by the ipynb format for Jupyter notebooks, and you'll see parallels to that in the aiconfig format. Please ask any questions, and share your thoughts on config vs. code.
- zby 3y ago
  I wish there was some more documentation on dynamic parameters - how they interact with the Workbook editor? A related question - I want to have chains of function calls - so that the prompts should contain all previous function calls and function call results. The chains can have variable length and end when the LLM calls a 'finish' function. How can I do that with AIConfig?
  - deleted 3y ago
    [deleted]
smy20011 3y ago
Why not store them in code? Easier to read and source controlled.
- saqadri 3y ago
  We have a bit of context about this in the readme: https://github.com/lastmile-ai/aiconfig#what-problem-it-solves https://github.com/lastmile-ai/aiconfig#what-problem-it-solv.... The main issue with keeping it in code is that it tangles application code with prompts and model-specific logic. That makes it hard to evaluate the genAI parts of the application, and also iterating on the prompts is not as straightforward as opening up a playground. Having the config be the source of truth let's you connect it to your application code (and still source controlled), lets you evaluate the config as the AI artifact, and also lets you open the config in a playground to edit and iterate. For example, compare how much simpler openai function calling becomes with storing the stuff as a config: https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/Function-Calling-OpenAI/Function_Calling_with_OpenAI.ipynb https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/... vs using vanilla openai directly (https://github.com/openai/openai-node/blob/v4/examples/function-call-stream.ts https://github.com/openai/openai-node/blob/v4/examples/funct...)
  - throwanem 3y ago
    Yeah, looking at this just briefly it might be the wheel I've (pre)-invented like 1/3 of for one of my couch projects. Definitely can see the appeal of a conventional format here, I'll check it out in more detail when time permits!
    - saqadri 3y ago
      Please definitely let me know when you get a chance to try it out! The readme has a link to our discord too if you want to get in touch (and you can email me directly too)
      - throwanem 3y ago
        I mean don't get your hopes up too far, it's gonna be a minute :D But I'll pass along what I find when I can!
  - smy20011 3y ago
    Thank you. I think most things we store in the config are non-business logic. Like the connection string of database or some feature flags. However, the prompt is your business logic in most cases and put your business logic into a separate file make it harder to read and harder to maintain.
activescott 3y ago
Well done. Now that I've seen your getting started video (https://aiconfig.lastmileai.dev/docs/getting-started https://aiconfig.lastmileai.dev/docs/getting-started) this seems obvious. I feel really silly that I copy & paste from the browser back into a file just so I can save different prompts and results to see what I liked best. Thanks for sharing and for open sourcing!
- saqadri 3y ago
  Thanks! Would love your feedback both on the UX editor as well as the APIs for interacting with aiconfig. We are also working on a local editor so you can have a locally hosted "AI workbook" playground
jdwyah 3y ago
I published https://www.prefab.cloud/blog/dynamic-config-for-openai-python/ https://www.prefab.cloud/blog/dynamic-config-for-openai-pyth... a few weeks ago as a way to do config of AI params like temperature, etc. But I've had a few people start using it for prompts as well and I really curious to see where it goes. In particular for 1. teams that have complex slow deploys, but want to change prompt now 2. when there are data analyst types doing the prompts and people don't want them to be able to "break things". 3. being able to alpha test / rollout / target new prompts easily. Definitely an interesting question whether prompts is code or configuration.
thatxliner 3y ago
How does this compare to Ollama
- saqadri 3y ago
  I assume you're talking about the Modelfile in ollama. AFAIK it doesn't work with other models. It may also be a bit more low-level, since it has information about the LoRA adapter. That said, there are some things that are overlapping, and we'll reach out to the ollama team to see if an aiconfig integration makes sense.
kordlessagain 3y ago
I love this! Been working on something similar that uses containers and Jinja2 templates: https://github.com/kordless/SlothAI https://github.com/kordless/SlothAI. A demo is running at https://mitta.ai https://mitta.ai. No docs yet and the auth system is getting an upgrade.
- saqadri 3y ago
  Nice! Let me know if you try out aiconfig. We would love contributions
joshka 3y ago
Relatedly similar project: https://promptfile.org/ https://promptfile.org/

97 points by saqadri 3y ago

Hi HN! OP here - this is our first open source project and we are really looking for your feedback about this direction, because we haven't seen something like aiconfig in generative AI developer tools yet. Our basic premise is that AI application development should be config-based, so you can track the prompts, models and model parameters being used more rigorously. Having this AI artifact then lets you iterate on it separately from your application code, and also set up evals that provide "test coverage" for the gen AI parts of your application. We were also inspired by the ipynb format for Jupyter notebooks, and you'll see parallels to that in the aiconfig format. Please ask any questions, and share your thoughts on config vs. code.

zby 3y ago
I wish there was some more documentation on dynamic parameters - how they interact with the Workbook editor? A related question - I want to have chains of function calls - so that the prompts should contain all previous function calls and function call results. The chains can have variable length and end when the LLM calls a 'finish' function. How can I do that with AIConfig?
- deleted 3y ago
  [deleted]

smy20011 3y ago

Why not store them in code? Easier to read and source controlled.

saqadri 3y ago
We have a bit of context about this in the readme: https://github.com/lastmile-ai/aiconfig#what-problem-it-solves https://github.com/lastmile-ai/aiconfig#what-problem-it-solv.... The main issue with keeping it in code is that it tangles application code with prompts and model-specific logic. That makes it hard to evaluate the genAI parts of the application, and also iterating on the prompts is not as straightforward as opening up a playground. Having the config be the source of truth let's you connect it to your application code (and still source controlled), lets you evaluate the config as the AI artifact, and also lets you open the config in a playground to edit and iterate. For example, compare how much simpler openai function calling becomes with storing the stuff as a config: https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/Function-Calling-OpenAI/Function_Calling_with_OpenAI.ipynb https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/... vs using vanilla openai directly (https://github.com/openai/openai-node/blob/v4/examples/function-call-stream.ts https://github.com/openai/openai-node/blob/v4/examples/funct...)
- throwanem 3y ago
  Yeah, looking at this just briefly it might be the wheel I've (pre)-invented like 1/3 of for one of my couch projects. Definitely can see the appeal of a conventional format here, I'll check it out in more detail when time permits!
  - saqadri 3y ago
    Please definitely let me know when you get a chance to try it out! The readme has a link to our discord too if you want to get in touch (and you can email me directly too)
    - throwanem 3y ago
      I mean don't get your hopes up too far, it's gonna be a minute :D But I'll pass along what I find when I can!
- smy20011 3y ago
  Thank you. I think most things we store in the config are non-business logic. Like the connection string of database or some feature flags. However, the prompt is your business logic in most cases and put your business logic into a separate file make it harder to read and harder to maintain.

activescott 3y ago

Well done. Now that I've seen your getting started video (https://aiconfig.lastmileai.dev/docs/getting-started https://aiconfig.lastmileai.dev/docs/getting-started) this seems obvious. I feel really silly that I copy & paste from the browser back into a file just so I can save different prompts and results to see what I liked best. Thanks for sharing and for open sourcing!

saqadri 3y ago
Thanks! Would love your feedback both on the UX editor as well as the APIs for interacting with aiconfig. We are also working on a local editor so you can have a locally hosted "AI workbook" playground

jdwyah 3y ago

I published https://www.prefab.cloud/blog/dynamic-config-for-openai-python/ https://www.prefab.cloud/blog/dynamic-config-for-openai-pyth... a few weeks ago as a way to do config of AI params like temperature, etc. But I've had a few people start using it for prompts as well and I really curious to see where it goes. In particular for 1. teams that have complex slow deploys, but want to change prompt now 2. when there are data analyst types doing the prompts and people don't want them to be able to "break things". 3. being able to alpha test / rollout / target new prompts easily. Definitely an interesting question whether prompts is code or configuration.

thatxliner 3y ago

How does this compare to Ollama

saqadri 3y ago
I assume you're talking about the Modelfile in ollama. AFAIK it doesn't work with other models. It may also be a bit more low-level, since it has information about the LoRA adapter. That said, there are some things that are overlapping, and we'll reach out to the ollama team to see if an aiconfig integration makes sense.

kordlessagain 3y ago

I love this! Been working on something similar that uses containers and Jinja2 templates: https://github.com/kordless/SlothAI https://github.com/kordless/SlothAI. A demo is running at https://mitta.ai https://mitta.ai. No docs yet and the auth system is getting an upgrade.

saqadri 3y ago
Nice! Let me know if you try out aiconfig. We would love contributions

joshka 3y ago

Relatedly similar project: https://promptfile.org/ https://promptfile.org/