My colleague Pat just wrote up an article on LLM-based apps and how to use data engineering tools to help build them faster that blew my mind. It contains a co

My colleague Pat just wrote up an article on LLM-based apps and how to use data engineering tools to help build them faster that blew my mind. It contains a complete implementation - with scraping context data from a docs website - chunking it, getting embeddings via the openAI API - loading it into pinecone - and finally a simple Q&A interface with streamlit on top of it Here's a quick summary: - LangChain and LlamaIndex are great tools for quick exploration - But aren't perfect for production-grade use - I think we all know the "LangChain is pointless" debate, but there's a lot of real meat to it, and Pat describes a few of them (a lot of LangChains extractors are super basic, 2-3 liners without retries etc.) - LLM applications are all about moving data, extracting and enriching data (creating embeddings!) are the most expensive ones of those steps - A bunch of data engineering tools are out there that make these two steps much easier, versionable, robust, and reproducible. - Meltano is one such tool and Pat implemented the above described pipeline with it FWIW: The GitHub project that comes with the post is super easy to run and super modular. I just tested it and was able to modify everything for my own application within 30 mins.