Building with LLMs does not have to be that different from building on top of any other APIs. (*) The two things that are crucial are: (1) Observability Being

Building with LLMs does not have to be that different from building on top of any other APIs. (*) The two things that are crucial are: (1) Observability Being able to see and store the flow of inputs and outputs for your pipelines. In development, this is invaluable for seeing under the hood and debugging. In production, it's essential for collecting data for eval and training. (2) Configurability Being able to quickly swap out and try different configurations -- whether it's "prompt templates", model parameters / providers, etc. To that end, I have my own internal framework & platform that tackles these problems in a holistic way. Here's a short example: - The basic building block https://www.loom.com/share/d09d38d2c316468fa9f38f5b386fc114 https://www.loom.com/share/d09d38d2c316468fa9f38f5b386fc114 - Tracing for more complex system: https://screenbud.com/shot/61dc59a6-5c73-4610-8168-753b79062fc3/image.png https://screenbud.com/shot/61dc59a6-5c73-4610-8168-753b79062... - Code example in TypeScript & Rust: https://gist.github.com/Palmik/42940e3887d6446244f8b74ce0243827 https://gist.github.com/Palmik/42940e3887d6446244f8b74ce0243... I recently wrote a bit more about this here: https://news.ycombinator.com/item?id=36760727 https://news.ycombinator.com/item?id=36760727 Would be happy to chat and brainstorm with folks working on similar problems. (*) I am basing this on experience from working on Google's Bard and now on my own products.