5 ms·
I've been finding this stuff genuinely useful for two years now, across Copilot and ChatGPT and Claude 3 Opus and similar tools. Either I'm a dimwit, easily co
by simonw 3y ago
I've been finding this stuff genuinely useful for two years now, across Copilot and ChatGPT and Claude 3 Opus and similar tools.
Either I'm a dimwit, easily conned by hype and shiny tools to the point that I can imagine benefits for two years that simply aren't there... or there's something to them.
- aimazon 3y agoI don’t think you’re a dimwit but I read your post[1] with an example and I am curious to whether you feel you’re losing something by telling the LLM it’s wrong and to try again, rather than going through the exploratory/iterative learning process yourself. For example, would you have known to ask about GeoJSON if you had not come across and learned about it pre-LLM? More succinctly: do you feel you’re learning more or less or an equal amount when using LLMs? [1] https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-case-study/ https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-cas...
- noduerme 3y agoYou seem to be writing from a lost world when people enjoyed coding because they enjoyed learning new things with every project. I think that's sadly something only enjoyed at work by a very small minority these days.
- simonw 3y agoI very much aim to learn something new with every project. The thing I like most about working with LLMs is that they let me take on MORE projects!
- eru 3y agoI don't think those 'good old days' ever existed. Or rather, the situation hasn't really gotten worse.
- noduerme 3y agoI probably wasn't clear I still live in that world, both professionally and in personal projects. But my perception is that the vast majority of devs and engineers do not.
- simonw 3y agoAbsolutely I could have learned more from that particular project if I'd spent more time with it rather than getting the LLM to do the work... but that's why I like it as an example: since it was effectively a distraction (a "side quest") the alternative wasn't learning more, it was not doing it at all (and learning nothing). I'm able to get really great results out of LLMs because I have 20+ years of experience helping me know what questions to ask of them. I do feel like my rate of learning has increased though, because I'm much more likely to try out a completely new technology when I know an LLM can flatten the learning curve for me a bit.
- gtirloni 3y ago> I do feel like my rate of learning has increased It's the same for me. Lots of experience knowing what to ask for. It does a better job in summarizing knowledge and getting me a relatively coherent explanation. Much faster than using Google Search to find and connect the dots from dozens of pages. I just don't see much benefit in its reasoning and code assistance features besides basic stuff.
- yawaramin 3y ago> I have 20+ years of experience helping me know what questions to ask of them. That seems rather unlikely given that LLMs didn't exist more than seven years ago, much less 20+.
- erhaetherth 3y agoI was trying to fetch key-value pairs out of a database using PHP+PDO the other day and I knew there was a nice easy to do it but I couldn't remember how. Something about fetchAll, maybe PDO::FETCH_GROUP|PDO::FETCH_COLUMN.... what was it? So I asked a couple LLMs. They wrote out loops for me to format the data how I wanted. I could have copy-and-pasted that in and it would probably have worked. But I felt there was something better yet, so back to Google I go. It's `PDO::FETCH_KEY_PAIR`. It's built-in. But oddly kind of hard to find unless you know the right thing to search for, and "key pair" was not springing to my mind. Point is, if you just let the LLMs do your work you won't even find these better ways of doing things. And I'm quite afraid of what the LLMs are going to do to Google, Stackoverflow and documentation in general. In a couple users it'll be ungoogleable too.
- fzeroracer 3y agoAs noted LLMs can only give you what you ask for, but for a lot of problems what you ask for isn't what you need; it's two or three steps removed. And LLMs can't tell you that you're doing something wrong; unlike curmudgeonly users on SO or in various forums/channels. My gut feeling is that we're going to enter into a 'dark age' of coding where a lot of previously available resources are going to be ransacked and made hard to find in favor of big corporation owned LLMs. It's already having an extremely bad effect on search in general; we're potentially only a few fights away from sites like SO having users leave en masse. That's why I think having a strong network of engineers to talk with will become more important than ever, almost a return to the IRC days.
- DrSiemer 3y agoStrong disagree. I have learned so many new things since I started using LLM's. Some of them I'm actually embarrassed to admit, because I should have known about them a decade ago. If you work in a small company and you are the most experienced developer, you don't often get feedback on how you can improve things. The trick is, quite simply: just ask. I regularly dump some code I wrote in a language model and then ask what can be done better. I would never do that in any online space, because first, I don't wait an answer maybe some day, I need an answer NOW. And second, I prefer to avoid being called a fool.
- anonzzzies 3y agoNah, it seems a lot of people are using them wrong? OP seems to also fall in that category. I sit next to many people in pair programming situations and it’s quite weird to see very smart programmers using gpt or copilot; they are the ones shouting ‘stochastic parrot’ on forums and yet expect some kind of mind reading magic when using these tools. When I show that you can put a comment for your function, like you should do anyway, and it writes the function, it usually clicks. Many still find it ‘faster and easier’ to write the function themselves and that’s fine, but, like OP, doing things like entering ???? and generally expecting a useful result will not be optimal.
- ern 3y agoYou make an interesting point. I once worked with someone who was brilliant, but had gotten into the field via a math degree, and we were building an enterprise-software product. We tried pair-programming, and he cracked within a few minutes. He couldn't articulate his thought processes verbally. Sounds similar to the scenario you're describing.
- blibble 3y agoI wouldn't feel too bad, there's orders more magnitude more money put into this con vs. the last one (NFTs)
- eru 3y agoYour counting of 'cons' seems pretty idiosyncratic. You know there have been lots of other technologies people have been working on in the meantime and concurrently? Some of them more questionable, some of them less.
- selcuka 3y agoI basically say the same when people ask me if the AI bubble is going to burst. While I agree that the current push for LLMs should be taken with a grain of salt, I don't think it's a bubble either. I have been finding it useful too, and I don't think I will stop using it because it will look less shiny in the future.
- BlueTemplar 3y agoThe dot com was a bubble that burst spectacularly enough, yet we still use the World Wide Web, and even some of those websites.
- tymscar 3y agoYou are clearly not a dimwit, and you don’t only have way more experience than most people here, you also have some amazing projects under your belt. However I can’t help but notice that the vast majority of blogposts, talks, tweets, and basically everything else you do now is around LLMs. Do you not think that’s indicative of this being “hyped” and “shiny tools”?
- ToValueFunfetti 3y agoIf hype just means people are excited about it and talking about it, it should be a positive signal about the merit of the thing. I think people get the wrong idea by measuring P(X is good | people are excited about X) and finding it to be low. But P(X is good | people are not excited about X) is vastly lower, and 'hype' as here used is not really discrediting.
- thaumasiotes 3y ago> I think people get the wrong idea by measuring P(X is good | people are excited about X) and finding it to be low. But P(X is good | people are not excited about X) is vastly lower Only if you're including things that don't exist. If you restrict yourself to things that people might conceivably talk about, that probability is actually very high, for the simple reason that bad things are exciting specifically because they are bad. Here are some boring things: - bread - sunlight - air - water - parents Note that starvation, darkness, suffocation, dehydration, and being orphaned are all much more exciting than their opposites.
- ToValueFunfetti 3y agoPoor phrasing on my part; 'good' should have been 'worth talking about'.
- tucnak 3y agoDeep learning is the most important thing to happen to philosophy since Socrates / Wittgenstein.
- thefaux 3y agoYou are creating a false dichotomy. Very few people are saying there are no benefits, but many have reasonable concerns both about the current efficacy of these tools as well as the expectation of continued exponential growth which is driving much of the current hype. What if we have already passed the inflection point where the exponential growth transitions to an s-curve? That would mean that this technology on its own would only get marginally better than is today. Maybe 2, 4 or even 10x better than it is today, but not 100x or 1000x. To break those barriers, we would need further innovations beyond just throwing more gpus and training data at the problem. I personally am short on LLMs because I believe it is much more likely that we have already crossed the inflection point or will soon. Again, they are impressive but ultimately I think that LLMs will be at best a footnote in history if they are even remembered at all in a few hundred years. But of course I could be wrong.
- simonw 3y agoI've been generally staying away from the whole "imagine what this stuff could do next!" side of the discourse. My personal opinion there is that if all research froze today it would still take us years to figure out all of the potential use-cases and applications for the models we have access to right now, and how best to apply them.
- vertis 3y agoNo I'm with you. I don't use copilot though for undirected autocomplete, I use tools that allow me to give instructions and diff the results, Cursor and Aider are my current defaults, both with Claude 3 now[0]. Always looking for alternatives though. I think it's always going to be a YMMV experience with LLMs. I'm an extreme generalist with 23 years of experience so it compliments my strengths and weaknesses. I know how to program across 20 odd languages but most of them I need to look stuff up if I'm not using it frequently enough. Now though, don't really need to look stuff up. For me there are two groups, those that want to use LLMs and are pushing it forward and those that are would prefer that LLMs not exist and want it to be hype that goes away. Having followed the hype cycles of VR/AR and crypto I can feel a difference. Both of those felt like a solution in search of a problem. The true believers wanting it to be something like Ready Player One with VR (fun story, Oculus/Facebook actually handed out copies of the novel at Oculus Connect 2 in 2015). The hype around LLMs seems different; like applying a new solution to all the existing problems to see where it helps, and that's just with the current iteration. [0]: I'd prefer to be using local/open equivalents but the capabilities are still lacking.