4 ms·

Thank you! So if I were to, say, build my own SaaS product that I wanted AI capabilities in, I could theoretically use NanoLLM to train data on my domain-specif

by ebbi 1y ago

Thank you! So if I were to, say, build my own SaaS product that I wanted AI capabilities in, I could theoretically use NanoLLM to train data on my domain-specific stuff to have a domain-specific trained LLM to use in my product without having recurring fees from an API provider like OpenAI?

fragmede 1y ago
Technically yes, but NanoLLM is stripped down and targeted more words educating AI researchers so I wouldn't recommend you use it for that (because it's output isnterrible compared to ChatGPT) (intentionally, it's a teaching tool). Nothing stopping you, but for that goal, I'd recommend starting with one of the downlodable permissibly license models like a newer Qwen3 and fine tune it. Google Collab has notebooks specifically for that. Once you have your fine tuned model, then you wouldn't be paying OpenAI to use it, but it would need to be run somewhere, and those somewheres range in quality and price. Models come in various shapes and sizes and the bigger the model, the beefier (and more expensive to rent) a computer you need to operate this SaaS business.
- ebbi 1y ago
  Thanks for this - learned a lot. I'll look into those.
simonw 1y ago
Training (or fine-tuning) a custom model to answer domain-specific questions is almost never the right solution. It's complicated, expensive, time-consuming and the results are rarely any good. I have yet to see a demo of someone doing this that I find convincing, at least for adding new knowledge to a model. If you want to teach an LLM to answer questions about private documents you should look into RAG or agentic search - techniques where the LLM can take a user's question and then look for additional information by searching some documents before answering. The good news is that these tricks work reasonably with small models that you can run on your own hardware - even a 4B or 8B model (a few GBs to download) can often handle these cases. But... even then, it's still usually cheaper to pay for the APIs from OpenAI and suchlike. Their API costs are so low that it's hard to save money by running your own model somewhere, since you have to pay to keep it in RAM the whole time while OpenAI share that cost between thousands of users.
- ebbi 1y ago
  Very helpful - thanks a lot!

by ebbi 1y ago

fragmede 1y ago

Technically yes, but NanoLLM is stripped down and targeted more words educating AI researchers so I wouldn't recommend you use it for that (because it's output isnterrible compared to ChatGPT) (intentionally, it's a teaching tool). Nothing stopping you, but for that goal, I'd recommend starting with one of the downlodable permissibly license models like a newer Qwen3 and fine tune it. Google Collab has notebooks specifically for that. Once you have your fine tuned model, then you wouldn't be paying OpenAI to use it, but it would need to be run somewhere, and those somewheres range in quality and price. Models come in various shapes and sizes and the bigger the model, the beefier (and more expensive to rent) a computer you need to operate this SaaS business.

ebbi 1y ago
Thanks for this - learned a lot. I'll look into those.

simonw 1y ago

Training (or fine-tuning) a custom model to answer domain-specific questions is almost never the right solution. It's complicated, expensive, time-consuming and the results are rarely any good. I have yet to see a demo of someone doing this that I find convincing, at least for adding new knowledge to a model. If you want to teach an LLM to answer questions about private documents you should look into RAG or agentic search - techniques where the LLM can take a user's question and then look for additional information by searching some documents before answering. The good news is that these tricks work reasonably with small models that you can run on your own hardware - even a 4B or 8B model (a few GBs to download) can often handle these cases. But... even then, it's still usually cheaper to pay for the APIs from OpenAI and suchlike. Their API costs are so low that it's hard to save money by running your own model somewhere, since you have to pay to keep it in RAM the whole time while OpenAI share that cost between thousands of users.

ebbi 1y ago
Very helpful - thanks a lot!