Nice overview, the parameter efficient fine tuning space is heating up. It’s been interesting to watch fine tuning go from a binary choice to a spectrum. With t

Nice overview, the parameter efficient fine tuning space is heating up. It’s been interesting to watch fine tuning go from a binary choice to a spectrum. With the context window also continuing to grow LLMs are entering a whole new era. I imagine down the road we may learn what to tune into models dynamically and how deep it needs to go. Or maybe the context window becomes a sort of rapidly trained thin NN layer on top like we’ve seen in diffusion models