3 ms·
Fine tuning will disappear, no reason to invest so heavily in it. Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next
by ldjkfkdsjnv 3y ago
Fine tuning will disappear, no reason to invest so heavily in it. Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Anyone starting an LLM application startup is arguably wasting their time, wait until the next iteration is out. Then you will know whats possible.
- minimaxir 3y ago> Also, most LLM code being written right now (prompts, RAG, etc) will be obsolete once the next model comes out. Not true. Most prompt techniques that work on current modern LLM models will work on different or future models, although it will require a QA pass for any regressions.
- ldjkfkdsjnv 3y agoYeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. If you believe in the scaling theory, then writing LLM applications is non sensical.
- deleted 3y ago[deleted]
- mlyle 3y ago> Yeah they will work, but they will also be unnecessary. You will also bake all sorts of logic into your application that will be solved natively on the stronger model. I think that a whole lot of what I do in prompt engineering is what's necessary to fully specify the output that I want. A newer model may be less finicky, so I have a higher chance of getting it to work on the first try (and it's more reliable afterwards), but it's hard for me to imagine it needing a whole lot less prompt.
- Xenoamorphous 3y agoHow will a more powerful model be a substitute for RAG, which is usually used with private data that won’t be present in any training dataset?
- ldjkfkdsjnv 3y agoI just think that the capability of the model could radically change, such that however you structured your RAG pipeline, might need to be rewritten. More general problems could be solved by the model, that you were solving with some complicated contraption of prompts.
- kergonath 3y agoOne of the idea is to just stuff all the documents in the prompt, which still keeps them private but avoids having to faff around with chunking, embedding, and vector stores. That’s not really the end of RAG as a concept, but it would change all the current tooling and infrastructure we built for it. I don’t think RAG is going away, at least not because of this. But I expect new techniques to become available fairly regularly.
- kergonath 3y ago> If you believe in the scaling theory, then writing LLM applications is non sensical. But not doing it is an opportunity cost. You don’t built skills, tooling and experience, and you don’t get feedback on what works and where you should go. It’s like computers in the 1990s: there’s always a better one 6 months away, so if you wait for it to stabilise, then you don’t do anything for a decade. Just enjoy the ride, bearing in mind that things change very fast and some things will be obsolete next year.
- treprinum 3y agoOpenAI-related startups are likely using GPT-5 already. Waiting it out won't help other startups, they will be too far behind.
- rkagerer 3y ago"Don't buy a computer today, because the faster one is coming out tomorrow"
- simonw 3y agoDon't buy a computer today with a six month delivery lead time, because there's a company that releases computers with a same-day lead time with several improved models coming out next week.