Hi, I am running several LLM for a web app running locally (django framework + Ollama for the LLMs). One of the main constrains is the limited context window f

Hi, I am running several LLM for a web app running locally (django framework + Ollama for the LLMs). One of the main constrains is the limited context window for now (but getting better now with the newer models). This web app aims at keeping data safety first (IP and sensitive data). A good way to feed the local agents is to use markdown formatted data (PDF to mrkd using pymupdf4llm). I am running it on my work laptop (M2pro) but I am looking forward to test is on AMD AI Max 'strix halo' due to it unified memory. Mainstream GPU have a frustratingly limited amount of VRAM. best