>Does anyone know a secret that makes that site tractable? Its basically just a repo for models. Most original models are uploaded in fp16 format, with differe

>Does anyone know a secret that makes that site tractable? Its basically just a repo for models. Most original models are uploaded in fp16 format, with different parameter counts - higher parameter count = better performance. If you were to fine tune the model on your own data set, you have to keep the model in fp16, because gradients need higher resolution On the flip side, inference is pretty much statistically most likely token which can be obtained without such resolution. As such, these models are usually quantized with GPTQ (GPU first), GGUF (CPU first, born from the llama.cpp project, but supports ), and AWQ (new method, supposedly faster than GPTQ). Primer for quantization https://archive.ph/2023.11.21-144133/https://towardsdatascience.com/which-quantization-method-is-right-for-you-gptq-vs-gguf-vs-awq-c4cd9d77d5be https://archive.ph/2023.11.21-144133/https://towardsdatascie... When using the model, you generally want to use the model with the largest parameters, with the highest bit quantization that fits in your system if you are running this on personal hardware. Easiest way to do this is with ollama, because its basically just does what pytorch or llama.cpp do in terms of loading models onto gpu (or ram for apple silicon), and executing them with whatever hardware you have. It can auto download models (usually 4 bit quantized) as well, and integrates into vscode with Continue extension. >For an example of a question, for which I cannot find an answer, I am aware of a startup that has developed a chat client that, the startup says, can entirely replace a company's customer support team. Among the claims made by the startup is that when their chat client makes a mistake, it can be easily adjusted so it won't make that mistake any more. I am curious, what approaches are the engineers at that startup probably using to fix mistakes? Highly likely some form of prompt engineering. Langchain is a popular tool to utilize for this. Most companies pay for api access rather than set up their own hardware. >how are you keeping up with advances around LLMs and associated tools? Wait for a model to drop on ollama, try it out.