3 ms·
Part of my stack selection process these days is considering how well LLM tools like ChatGPT and Claude "know" the stack in question. Using libraries that have
by simonw 2y ago
Part of my stack selection process these days is considering how well LLM tools like ChatGPT and Claude "know" the stack in question.
Using libraries that have been around for a few years and have accumulated a lot of training data is a huge productivity boost here, because LLM tools can both write code that uses them and usefully explain and debug them when they go wrong.
It's weird and a little uncomfortable to have this as part of my selection criteria, but if I want to maximize the rate at which I can build things it's honestly a useful rule of thumb.
- hiatus 2y agoHow do you go about determining the aptitude of a particular LLM with a particular stack? What does your evaluation look like?
- simonw 2y ago"Vibes" Seriously. The only way that works for me is to spend a bunch of time messing around with a model trying out different things. I really want to have a more repeatable process for this. I've been tinkering with a way of setting up my own automated evals to run against different models but I'm a fair way off having something usable there. For programming libraries I find that my assumptions usually hold: if a live set is widely used and has been around for a few years it's likely to get good results. I picked the WTForm Python library recently for this reason.
- yodon 2y agoIsn't this identical to a dev from five years ago saying "I picked X stack because of the community"? ChatGPT learns from blog posts, forums, etc. If there's a vibrant and helpful community around a stack, ChatGPT has a lot to work with. If it's obscure, not so much.
- simonw 2y agoIt's similar to that, yes. The difference is that I didn't use to rate that as one of my highest selection criteria, because if a new library was well designed and well documented enough I wouldn't feel the need for large amounts of community support. Today if a new library is well designed and well documented I still might pick a more mature competitor purely for the LLM advantage.
- dysoco 2y agoI recently started writing a toy kernel in the Zig programming language and I came upon this realization. The Copilot support was really bad and it was trying to autocomplete with C code, which was correct semantically but wasn't valid Zig code. It might seem non-important to many, but I was writing a big amount of boilerplate code, tests, etc. and I have a feeling I would've been much more productive just sticking with C.