## How did I build this? The backend is written in Rust, and the frontend is written in TypeScript with SvelteKit. The API is based on gPRC, using Buf Connect

## How did I build this? The backend is written in Rust, and the frontend is written in TypeScript with SvelteKit. The API is based on gPRC, using Buf Connect on the TS side. On the storage side, it’s using Postgres + Qdrant (self-hosted on dedicated servers) + R2. The reason I am self-hosting on dedicated servers are mostly the high CPU and RAM requirements for the main project which would otherwise make it cost prohibitive. For the LLM parts (which are central to the main project), I built my own language-agnostic framework and platform to facilitate building LLM powered services. This is based on some lessons learned while working on Google’s Bard and other LLM projects that preceded Bard (those were ultimatelly killed — long story :)). The framework and platform are currently private, but I might open it up in the future. To give you a glimpse: The main premise is that building apps that leverage LLMs should not be that different from building regular apps. The main difference is the need for configurability and observability. Each component (e.g. LLM call, Vector DB lookup, etc.) is just a regular RPC method, with well defined schema for its input & output. The platform allows you to configure each method, which just means specifying values for some subset of its inputs. The platform also tracks and stores the inputs and outputs of all the RPC calls and sub-calls. This is invaluable during development to understand and debug the methods, and in production to collect data for evaluation & training. Thanks to the well defined input & output schemas, all of this can be done universally, rather than creating a bespoke solution for each method. Here are some examples: - The “LLM” building block: https://www.loom.com/share/bc6fa6b27298420c82fcacca5f84d096 https://www.loom.com/share/bc6fa6b27298420c82fcacca5f84d096 - Higher level JSON → Format → LLM building block: https://www.loom.com/share/d09d38d2c316468fa9f38f5b386fc114 https://www.loom.com/share/d09d38d2c316468fa9f38f5b386fc114 - Example a complex trace from a summarization method: https://screenbud.com/shot/61dc59a6-5c73-4610-8168-753b79062fc3/image.png https://screenbud.com/shot/61dc59a6-5c73-4610-8168-753b79062... Example code for a simple retrieval augmented chat bot in TypeScript and Rust https://gist.github.com/Palmik/42940e3887d6446244f8b74ce0243827 https://gist.github.com/Palmik/42940e3887d6446244f8b74ce0243... — there’s still a lot of room for polish and boilerplate removal, but it gets the job done already. Happy to answer any questions.