6 ms·
This sounds to me like you have no idea how to properly integrate LLMs into your workflow. Just because they’re not useful for changing large code bases doesn’t
by valine 3y ago
This sounds to me like you have no idea how to properly integrate LLMs into your workflow. Just because they’re not useful for changing large code bases doesn’t mean they can’t be valuable. A few examples of things I’ve used LLMs for:
Writing bash scripts to automate various parts my workflow
Quickly interpreting complex regex
Tutoring me on how to use poorly documented APIs
Proof reading important emails
Everything copilot, writing java docs, auto completing all the cases in a switch statement, etc.
- l33tman 3y agoCould you? Do you have to? Would someone pay you to implement an FPGA design, given that you know nothing about it but have access to GPT-4? In practice, I've found that the advantage it might give you quickly ebbs out. It surely helps filling details and blanks though.
- DonsDiscountGas 3y ago>Tutoring me on how to use poorly documented APIs Can you elaborate on this? I would find it very useful.
- valine 3y agoMy current workflow is to copy a bunch of relevant code into GPT4 and then ask it questions about how to use the API. GPT4 is very good at inferring how to use an API, even from code snippets. The key is that you can paste large amounts of unstructured code and GPT4 will make sense of it.
- latexr 3y ago> Writing bash scripts to automate various parts my workflow Would you share some of those? Bash is fraught with gotchas and intricacies and seldom have I read Bash code where I couldn’t immediately spot flaws. There’s often something which left unchecked may come back to bite you later. I’m skeptical the LLM code would be any better, since it’s (presumably) trained on a corpus of subpar code. But I could be wrong.
- spudlyo 3y agoGive the LLM access to `shellcheck` and have it iterate on the warnings. I think promising results have been shown when you allow LLMs to reflect on their answers and give them access to tools.
- IshKebab 3y agoShellcheck can't check everything though. I think using LLMs on difficult to verify code like shell scripts and complex regexes is a pretty bad idea. I've found them useful as a "smart search engine", e.g. if I don't even know the magic terms to search for. Most of the time it gives me answers that are completely wrong, but are close enough that I know where to look to find the right answer. E.g. "Using the rapidcheck C++ test framework, how do you test a function that takes two vectors, one of which must be exactly 8 times as long as the other." It gets it wrong but it still helped. I definitely wouldn't want to use its output unquestioningly though.
- valine 3y agoDepends entirely on if the shell script is mission critical. I couldn't care less about the quality of my LLM shell scripts most of the time. My only success criteria is that is that it solves a problem. These are scripts I'm never going to touch again, things like batch converting files, or like one time I had it generate a script to titrate stable diffusion parameters. As long as I can skim the output and see that its correct I'm happy. You call it a bad idea, but it's saved me hours of work.
- nice_byte 3y ago> Tutoring me on how to use poorly documented APIs Good luck with that. GPT-4 is _incredibly_ bad at Metal (most under-documented GPU api I've ever worked with), but more importantly it's _even worse_ at Vulkan, which is complete opposite - it's meticulously documented. The stuff LLMs are good at is the stuff that a lot of people use and talk about all the time, in public. There's just not a lot of public discussion (with examples) about e.g. how to properly use Vulkan in a production application, and the LLM itself can't "reason" within the framework of the detailed specification, so most things it outputs is wrong. In the long run though, a very likely outcome is that tools that the LLMs have a poor "understanding" of just wither and die, or are relegated to being something only 3.5 people in the world use. I can see a future where literally all software is written in python just because that's the language that OpenAI's LLMs happens to know best.
- ilaksh 3y agoIt has an 8kb (or 32kb if you are lucky) context. Try giving it the relevant documentation examples before asking a question.
- nice_byte 3y agothe point the grandparent was making was that GPT-4 can teach you how to use an under-documented API. that means the relevant docs are either sparse or non-existent, otherwise there wouldn't be a need to ask GPT-4 in the first place.