4 ms·
Was the comparison done with or without code context (as obtained using RAG or letting Sonnet ask for files)?
by pcwelder 2y ago
Was the comparison done with or without code context (as obtained using RAG or letting Sonnet ask for files)?
- jeswin 2y agoIn the absense of other information, looks like a cherry-picked example to me.
- samatdav 2y agoWe used a single file for the context. It is a cherry-picked example, you are right. I wanted to demonstrate a simple visual change that our model did correctly unlike Sonnet-3.5. Since we are just getting started, we don't have many features like making changes across multiple files in the code editor so it would be harder to demo. Our premise is that a smaller fine-tuned works better than a large, general-purpose SOTA model. We plan to share more metrics and data in the future.