6 ms·
I’m doing an experiment in this in real time: I’ve got a bunch of top-flight junior folks, all former Jane and Google and Galois and shit, but all like 24. I’v
by benreesman 2y ago
I’m doing an experiment in this in real time: I’ve got a bunch of top-flight junior folks, all former Jane and Google and Galois and shit, but all like 24.
I’ve also been logging every interaction with an LLM and the exit status of the build on every mtime of every language mode file and all the metadata: I can easily plot when I lean on the thing and when I came out ahead, I can tag diffs that broke CI. I’m measuring it.
My conclusion is that I value LLMs for coding in exact the same way that the kids do: you have to break Google in order for me to give a fuck about Sonnet.
LLMs seem like magic unless you remember when search worked.
- dartos 2y ago> LLMs seem like magic unless you remember when search worked. Yikes. I didn’t even think about this, but it’s true. I’m looking for the kinds of answers that Google used to surface from stack overflow
- supaflybanzai 2y agoKagi… Fully switched over more than a year ago and never looked back.
- jansan 2y agoThe best way to get useful answers was (and for me still is) to ask Goggle for "How do I blah site:stackoverflow.com". Without the site filter, Google results suck or are just a mess, and stackoverflow's own search is crap.
- JamesBarney 2y agoGoogle used to be better but so was stack overflow. Now a lot of the answers are out-dated. And even more importantly they got rid of any questions where the answer was even a little bit subjective. Unfortunately for users that's almost all the most useful answers.
- skydhash 2y agoNowadays, I just read manuals, docs and books. I mostly use search as a quick online TOC or for that specific errors I’m in no mood to debug.
- nopinsight 2y agoWhat domain/type of software do you and they work on? Cursor has been quite effective for me and many others say the same. As long as one prompts it properly with sufficient context, reviews the generated code, and asks it to revise as needed, the productivity boost is significant in my experience.
- benreesman 2y agoI get value out of LLMs on stock Python or NextJS or whatever where that person was in fact a lossy channel from SO to my diff queue. If there’s no computation then there’s no computer science. It may be the case that Excel with attitude was a bubble in hiring. But Sonnet and 4o both suck at why CUDA isn’t detected on this SkyPilot resource.
- benreesman 2y agohttps://www.youtube.com/watch?v=FpG--q5u6ns https://www.youtube.com/watch?v=FpG--q5u6ns
- komali2 2y ago> But Sonnet and 4o both suck at why CUDA isn’t detected on this SkyPilot resource. I don't understand this sentence, should "both suck at why" be "both suck and why" or perhaps I'm just misunderstanding in general?
- benreesman 2y agoSkyPilot is an excellent piece of software attempting an impossible job: run your NVIDIA job on actively adversarial compute fabric who mark up the nastiest monopoly since the Dutch East India Company (look it up: the only people to run famine margins anywhere near NVIDIA are slave traders). To come out of the cloud “credits” game with your shirt on, you need stone cold pros. The kind of people on the Cursor team. Not the adoring fans who actually use their shit.
- valenterry 2y agoWell, the context is the problem. LLMs will really become useful if they 1.) understand the WHOLE codebase AND all it's context and THEN also understand the changes over time to it (local history and git history) and finally also use context from slack - and all of that updating basically in real time. That will be scary. Until then, it's basically just a better autocomplete for any competent developer.
- benreesman 2y agoTo be clear I’m a huge fan of the Cursor team: those folks are clearly great at their jobs and winning at life. They didn’t get ahead by selling you the same thing they do, if they did Continue would be parity.
- danielbln 2y agoI don't understand, are you using LLMs purely for information retrieval, like a database (or search index)? I mean sure that's one usecase, but for me the true power of LLMs comes from actually processing and transforming information, not just retrieving it.
- benreesman 2y agoI have my dots wired up where I basically fire off a completion request any time I select anything in emacs. I just spend any amount of tokens to build a database of how 4o behaves correlated to everything emacs knows, which is everything. I’m putting down tens of megabytes a day on what exact point they did whatever thing.
- benreesman 2y agoI’m actively data-mining OpenAI, they get a bunch of code that they have anyways because they have GitHub, I get arbitrary scope to plot their quantization or whatever with examples. Flip it on em. You’re the one being logged asshole. https://youtu.be/un3NkWnHl9Q?si=VOnH2krJkJLRA2BQ https://youtu.be/un3NkWnHl9Q?si=VOnH2krJkJLRA2BQ