The other problem I find is that LLMs are changing so fast, that what you evaluated 6-12 months ago, might be completely different now with newer models. So th

The other problem I find is that LLMs are changing so fast, that what you evaluated 6-12 months ago, might be completely different now with newer models. So the strengths and weaknesses quickly can become outdated as the strengths grow and weaknesses diminish. When the first batch of LLMs people tried in 2023 had a lot of weaknesses. At the end of 2024, we can see increases in performance in speed and the complexity of output. People are creating frameworks on top of the LLMs that further increase their value. We went from thousands of tokens in context to millions of tokens pretty fast. I can see myself dividing problems up into 4 groups: 1. LLMs currently solve the problem 2. It doesn't solve it now, but we are within a couple iteration of next generation models or frameworks to be able to solve it 3. LLMs are still years off from being able to solve this effectively so wait and implement it when it can. 4. LLMs will never solve this. I think a lot of people building products are in group 2 right now.