10 ms·

I think this essay is right in many ways. LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing wor

I think this essay is right in many ways. LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". However at some point you have to admit the LLM does generate things that are good answers. They might be good answers that happen to pass the smell test, but they are nonetheless good answers. For instance when you ask it for a snippet of code and it gets it right. And here is the crucial thing: you need to already know what you're doing to know whether the LLM got it right. I'm no historian, and I can ask cGPT for an essay about the causes of the Great War. When I get the answer, it sounds right to me. I don't know if the essay talks about the things an actual historian would find important, all I know is that it gives me the vanilla answer that some layman who has read a little bit would think was the right answer. Now there's another issue this brings up. Most of us are experts in one field only. What is stopping the LLM from fooling me in every field that I don't know anything about? I best be wary of using it outside of my area of expertise. So in the current iteration, I think LLMs are a shortcutting tool for experts. I can tell when it spits out a snippet of code that is correct, and when it's wrong. Someone who wasn't working in my domain would get fooled.

falserum 3y ago
Agree. It is a language model, so it’s best use is language things (emails, motivational letter, summaries) For everything else, you need to be enough of an expert to validate the output.
mike_hock 3y ago
The same principle applies outside of LLMs. Ever seen mainstream media write something about your field of expertise that wasn't complete horseshit? No? Then why believe anything else they say.
indigochill 3y ago
> LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". Isn't the hint in the name? It's a language model, not a knowledge model. LLMs are exceptional at generating stuff that passes as coherent language (like, the parts of speech are all where one would expect them to be in the sentences). The trouble is people think it goes deeper when the knowledge modeling part only happens incidentally because language and knowledge modeling are so closely related.
- mvdtnz 3y ago
  > Isn't the hint in the name? It's a language model, not a knowledge model. Well yes, but how is this insight helpful? People aren't just looking at the name. Have your missed the last 10 months of ridiculous breathless coverage of LLMs? > The trouble is people think it goes deeper when the knowledge modeling part People are literally calling this shitty text prediction algorithm "AI". The discourse is completely out of hand.
  - simonw 3y ago
    It is "AI", using the definition of that term that was established when the term was coined in 1956. https://en.m.wikipedia.org/wiki/Dartmouth_workshop https://en.m.wikipedia.org/wiki/Dartmouth_workshop > An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. LLMs fit that definition well. They're not AGI, but I think it's fair to call them AI.
    - mvdtnz 3y ago
      There is no widely accepted definition of "AI" and even if I accepted this one (I absolutely do not) no LLM meets this bar in the general sense.
      - trealira 3y ago
        Game tree heuristic searches, like the minimax algorithm with alpha-beta pruning, has also been called AI, which is why programs that play chess are called AI. What I think is the problem is that when people hear "artificial intelligence," they don't think of chess programs anymore; they think of something like Skynet from the Terminator. Or, maybe now they think of ChatGPT.
      - simonw 3y ago
        That 1956 workshop is widely credited as the place where the term AI was coined. If they don't get to define "AI", who does?
jwestbury 3y ago
There are loads of examples of human-assisted algorithmic processes, or the inverse, algorithm-assisted human processes, most of which have better performance than either entity alone. I've had trouble finding it, but years ago, I read the abstract of a study which looked at chess performance, pitting people who had access to some basic chess software against those who didn't. What they found is that the strongest predictor of success was whether a player could utilise the software effectively, rather than pure chess skill. Another area where I don't think there are any published studies, but there's a lot of anecdotal evidence, is algorithmic trading, where successful firms almost always have human traders monitoring the algorithmic behaviour. This means that humans can spot when the algorithm is doing something a bit odd, or when conditions aren't conducive to the algorithm performing well, and they can tweak some parameters on the algorithm to get a better result, shut it down if it looks like it's lost the plot, etc. (And let's be honest, a lot of this oversight is probably a result of Knight Capital.) My point here is: You're entirely right, LLMs are currently a tool for people with experience in the field they're working with. But that's still a pretty big step forward -- just like top quant hedge funds consistently outperform human-managed funds (see: Rentech), smart people who learn how to leverage these tools are going to outperform. The key is learning where they perform well, where they perform poorly, and how to validate their output.
- tmorton 3y ago
  > I've had trouble finding it, but years ago, I read the abstract of a study which looked at chess performance, pitting people who had access to some basic chess software against those who didn't. What they found is that the strongest predictor of success was whether a player could utilise the software effectively, rather than pure chess skill. There was a time when this was true - a grandmaster and a computer was stronger than either alone. But that window closed pretty quickly. Today, even the strongest GMs' best strategy is to follow the computer blindly.
- ycombobreaker 3y ago
  > let's be honest, a lot of this oversight is probably a result of Knight Capital Gray-box trading and humans in a control loop over automated trading have both been happening for decades. Knight Capital makes for a pretty spectacular example, but that's about it.
3cats-in-a-coat 3y ago
> LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". That's NOT how LLMs operate. They're trained on what is supposed to be selected representative human text of some at least minimum quality, and ideally truthful, factual and relevant. And then their prediction is judged on how close to that ground truth it is. Not on whether it's "plausible" or "will this pass for a correct answer". There's no math function for pretending to be right. You have to be right. One reason why LLMs make up things sometimes is because they're incredibly tightly optimized. Like you have no idea how tight. GPT-4 has the comparable complexity of a mouse brain. A mouse brain that has to fit the world's knowledge. Obviously, something has to give, and you'll not be accurate sometimes when you have so limited space. You'll store some things approximately, generally, and then extrapolate answers from them, maximizing for correct output.
- lazy_moderator1 3y ago
  this is an interesting statement, are you involved in LLM research?
- ethanbond 3y ago
  > And then their prediction is judged on how close to that ground truth it is When/where/how?
- SAI_Peregrinus 3y ago
  Extant LLMs are trained on whatever data the trainers can get their hands on. There's no magical "check for truth" step, either at the input or at the output.
  - 3cats-in-a-coat 3y ago
    No, data is significantly filtered in order to bias it to accuracy. Of course if you feed an AI scientific papers and books of some quality, information will be of high quality. THAT... is how you "filter for truth". It's not supposed or required to be perfect. But of course you can filter the input. The fact you also add conversational input from various sources like Reddit and Twitter doesn't change this balance significantly. Also even those sources are filtered to exclude vulgarities, certain topics and so on. Also you're not thinking this through. If a question is on average answered incorrectly 'round the world, it means you're over 50% likely to also give the incorrect answer when asked. Who equipped you to know the truth?
coffeedan 3y ago
>> For instance when you ask it for a snippet of code and it gets it right. In my experience so far, more often than not, it doesn’t get it right. Brings to mind the old maxim, “a broken clock is right twice a day”.
- xp84 3y ago
  Personally, this just seems to me to model slight inexperience pretty well. But just like junior developers are not useless, I find GPT, copilot, etc. to be a force multiplier. It can generate, in 2 seconds, a 15-line function to do a simple thing which is nearly always either correct or needs one small tweak to satisfy me. This allows me to focus my brain 100% on the interesting parts of the problem domain. To me, it feels exactly like having a bright junior developer at a second keyboard twiddling his thumbs all day, waiting for me to need some trivial grunt work task.
garrickvanburen 3y ago
I like how you characterize the output as, “an answer that sounds right”. I was listening to a podcast yesterday where the purported expert gave an answer that sounded right but based on my direct experience - very wrong. Of course I could be wrong as well. Or the experts answer was the same kind of BS the OP was referring to and is so oft generated by LLMs.
- garrickvanburen 3y ago
  related: https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
add-sub-mul-div 3y ago
> So in the current iteration, I think LLMs are a shortcutting tool for experts My take has been that LLMs aren't valuable for me because I'm an expert and writing code is no longer very difficult. And searching for the knowledge when I need it is faster on my own than having a conversation with a computer because true expertise also involves having mastered and streamlined that process.
- empath-nirvana 3y ago
  i'm an experienced coder and co-pilot often writes code faster than I can think of it, and it's usually fine and follows the pattern I had established with previous code. A lot of code that everyone writes is just boilerplate. Until I used copilot, I didn't realize _how much_ i was writing that didn't really require much thought.
  - zerbinxx 3y ago
    Imo if you’re “doing coding right”, at a certain point, you should be thinking very little about what you’re doing. This can manifest as having easily copypastable design patterns, lots of tests, inversion of control, etc. Even among senior developers it can be shockingly rare for people to actually understand and implement something like DI/IoC in a correct and useful way.
api 3y ago
They're a form of lossy compression. I suspect they are doing less than we think they are. They might be fooling us into thinking there's more intelligence there than there really is in the same way fractals can fool us into thinking "the whole universe must be a big fractal!" by their uncanny ability to replicate complex shapes.
matheusmoreira 3y ago
Just yesterday ChatGPT thought it was possible to use sizeof in C preprocessor macros. I called it out on this and it doubled down on it. Edit: #if directives, like static_assert.
- pelagicAustral 3y ago
  I've got a colleague that would've double down on it as well...
- yorwba 3y ago
  Where's the problem? ~ $ cat sizeofmacro.c #include <stdio.h> #define pintsize(T) printf("sizeof(%s) is %zu. Big or small? You decide.\n", #T, sizeof(T)) int main(int argc, char **argv) { pintsize(int); pintsize(argv); return 0; } ~ $ gcc sizeofmacro.c && ./a.out sizeof(int) is 4. Big or small? You decide. sizeof(argv) is 8. Big or small? You decide.
  - blibble 3y ago
    the preprocessor isn't evaluating it, it's just pasting the tokens in the preprocessor doesn't know anything about structs or types
  - diydsp 3y ago
    I was curious, too. Looked it up. Still don't understand tho. >The sizeof in C is an operator, and all operators have been implemented at compiler level; therefore, you cannot implement sizeof operator in standard C as a macro or function. You can do a trick to get the size of a variable by pointer arithmetic.
- fanf2 3y ago
  You can use sizeof() in macros but not in #if expressions.
  - matheusmoreira 3y ago
    Yeah, I meant #if directives, like static_assert. I asked it to help me statically verify my assumptions about ELF files and it gave me some pretty insane code. Here's the excerpt: > Certainly, you can use a combination of sizeof and #error directives for conditional compilation to achieve a similar effect without static_assert. // Compile-time check for structure sizes #if sizeof(Elf32_Phdr) != sizeof(((Elf32_Ehdr*)0)->e_phentsize) #error "32-bit structure size mismatch" #endif #if sizeof(Elf64_Phdr) != sizeof(((Elf64_Ehdr*)0)->e_phentsize) #error "64-bit structure size mismatch" #endif > In this example, if the structure sizes don't match, the preprocessor will generate an error at compile time using the #error directive. > I didn't know sizeof could be used with the C preprocessor > Yes, sizeof can be used in certain contexts within the C preprocessor. > It's commonly employed in combination with the #if and #error directives to perform compile-time checks on sizes or other constants. > This usage allows you to catch potential issues during compilation rather than at runtime. I should probably go sleep a bit before I embarrass myself any further. After nearly 24 hours awake it seems I have lost the ability to tell macros and directives apart.
jackcviers3 3y ago
> You need to already know what you're doing to know whether the LLM got it right. I want to call out that this is not necessarily true. You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs to produce novel code that you don't know the structure of beforehand. You start with a set of example inputs and desired outputs, and prompt the llm for a function in your programming language of choice. You then take the LLM's response and feed it into an agent that executes the code against the inputs and outputs, and report the discrepancies back to the LLM. It responds with a new completion, which you feed to the agent until all your example inputs produce the expected outputs. Finally, you can use property-based testing to produce new examples for testing by the llm, resolving the answers until the produced code is correct to within some margin of acceptable correctness. You can do this to produce code without needing to know anything besides the desired properties of generated code and the properties of the inputs. You can further automate this by using a separate LLM to produce the examples instead of the typical generation and shrinking functions to produce examples. This doesn't require any prior knowledge of the target language. You can expand beyond programming into any domain for which you can produce a scoring function and automate input generation. I suspect that you can use control theory (PID loops, behavior choice loops scored by Eigenvalues) to model complex scenarios spanning multiple domains as well, choosing the evaluation agent as the behavior, each behavior of which has a separately defined scoring function. All of this can be automated by using the LLM generator without prior knowledge of the algorithmic structure of the solution and knowledge of all possible inputs to all possible behaviors. If that all sounds very familiar, it's because it's essentially just doing test-driven development, but with the LLM machine as the developer. It would be expensive to run an entire project development that way, due to the high costs of executing LLMs and/or querying LLM apis, but if the cost is less than that of employing a junior developer or teams of developers, it might be worth it. And human intervention can be kept in the loop at any stage, making it very feasible for rapid prototyping - where you don't necessarily need the entirely correct answer from the machine, just a good enough starting point to allow the human to take over to produce a result.
- falserum 3y ago
  > You can do this to produce code without needing to know anything besides properties … Reminder, you started with this: > You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs Maybe those are fancy words for simple things which I’m not familiar with or learning programming language seems a simpler option :)
photochemsyn 3y ago
There are methods for getting useful information from LLMs about subjects outside your own area of expertise, but it takes a lot of work. For the example, causes of the Great War, I'd first ask it to provide a list of the top ten reasons reputable historians cite as primary causes of World War One. Then, for each reason that looked interesting, I'd ask the LLM to provide arguments for and against, including one primary historical document in support of each side. Then, go to Google Scholar or similar and look up that document and see what different reputable historians have said about it. A few iterations of this process should get you to some reliable information even in a field you know nothing about. This is a time-consuming process, and requires active engagement rather than what many people have been trained to do by our atrocious education system and watching television, i.e. passive absorption of content without skepticism.
- bsenftner 3y ago
  I'm doing something similar with figuring out how to convert my home to solar power. Where I live, the sun is perfect for solar, and as a result our area if flooded with solar startups that are universally using hard sell tactics and undermining one another. After discussions with neighbors, everyone feels the available solar installers are all serving slanted bad deals. So I'm using ChatGPT4-API to create solar installation planning bots, financing evaluation bots, duties and responsibilities checklists for service people, and so on. I think my neighbors and I might just form our own startup or just give it all away because this situation here with the available service companies feels Orwellian; they can like there is no choice, they use fear and hard sell language.
- latexr 3y ago
  > I'd first ask it to provide a list of the top ten reasons reputable historians cite as primary causes of World War One. And right here you could bump into problems, because how would you know if what you got back is really the opinion of reputable historians? For all you know it could be giving you the opinions of a layman on Reddit. > Then, go to Google Scholar or similar and look up that document and see what different reputable historians have said about it. Looks like that could’ve been the first step. Identify the reputable historians and go from there.
SamBam 3y ago
It reminds me of the old philosophical debate about AI and consciousness. Many philosophers argue that knowing whether something which appears to be conscious really is conscious is a meaningless question. If there is no way to distinguish consciousness from non-consciousness from the outside, then you may as well just say that the thing is conscious. I think we may be approaching the same question with regard to truth. If it appears to be truth from the outside, is that the same thing as being truth? Is there any point in knowing wether it was produced from a thinking mind or not?
- shmageggy 3y ago
  Truthfulness is on totally different philosophical footing than consciousness, so the analogy doesn't hold up. There is no "hard problem of epistemology", but there is a hard problem of consciousness. There exist criteria for evaluating truthfulness (that we may reasonably disagree on) that can be applied to LLM output.
  - joelfried 3y ago
    I think the Gettier problems are at least epistemological problems with some weight, at least as long as you're in the Justified True Belief sort of camp (which I think many of us start out as). If you haven't seen Gettier problems before, I'd start with the generalized problem and work out from there: https://en.wikipedia.org/wiki/Gettier_problem#The_generalized_problem https://en.wikipedia.org/wiki/Gettier_problem#The_generalize...
  - SamBam 3y ago
    I think the whole point of the philosophical concept of "bullshit" is that it's orthogonal to the truth. It doesn't matter whether the words end up being correct or not correct, it's still bullshit because the speaker doesn't know or care if it's actually true.
broast 3y ago
Reminds me of me when I was a junior dev. Pretend I already know how to do the thing and form the answer as I google and speak.
aredox 3y ago
"when you ask for a snippet of code and it gets it right" Most examples I have seen are about basic code that is highly likely to be already present in the training data - making the language model an expert system, but with a worse rate of failure. Meanwhile it has a hard time creating something it hasn't already seen, or a close variant.
- simonw 3y ago
  > Meanwhile it has a hard time creating something it hasn't already seen, or a close variant. How much time have you spent with the strongest models, such as GPT-4? I've been using it on a daily basis for 6+ months now and your statement there doesn't fit my experience at all.
  - ryandvm 3y ago
    Indeed. I feel like people are not even talking about the same thing in here. I've been using GPT-4 daily for a couple months and it's not outputting code that "only looks correct". It is writing functional solutions that often work as is. Just yesterday I pasted in a hastily written function and told GPT-4 to "please evaluate this function and make sure it's bulletproof". It outputted a bulleted list of criticisms (mostly true) and then spit out a version of the function that was shorter, easier to understand, and was safer. That's not just the "next most likely token". Or if it is, then that's all any of us are doing.
    - rbranson 3y ago
      It’s probably more accurate to say that what your brain is doing when writing code is more like emulating an LLM than an LLM is emulating your brain. Converting human language to code is an ideal task for LLMs, which is part of why they are such an exciting development.
  - zerbinxx 3y ago
    If you ask it to start explaining/extending/implementing classes/interfaces, start asking it to reason about leaky abstractions or when it’s appropriate to use Mixins, it frequently generates code that either doesn’t work or just misses the point. It’s especially hard because some of these problems actually are not tractable in the language you’re taking about due to their features, or due to limitations of your code/your knowledge it may suggest you do something slightly worse because you haven’t prompted it on factories or other design patterns that you might want to use.
PaulHoule 3y ago
Maybe I should be impressed that an algorithm can sort a list correctly 50% of a time because it is better than chance (ex. you have 1 in 3.6 million odds of sorting a list of 10 items correctly at random) but conventionally the value of a 50% sorting algorithm is 0, not 50% that of a sort that works since the sort is part of a larger system that requires the sort to work 100% of the time. For a long time there have been "machine learning" techniques where getting the right answer 80% of the time is a win, what's interesting about LLMs is they sometimes "raise the bar" for the machine learning problems (get it right 90% of the time) but have also lowered the bar for other things that can be done algorithmically 100% of the time.
- pjerem 3y ago
  The thing is, if you want to sort a list, LLM is the wrong tool. However, if what you want is to generate a program that will sort a list, your LLM will be pretty performant. It’s a powerful tool but like any powerful tool, you will use it wrongly if you don’t acknowledge its strengths and weaknesses. Judging a LLM because he can’t act as a program is a fundamental error because it means you are wasting an enormous amount of resources for nothing. The novelty of LLMs and where they are good at is helping you thinking about things. Helping you to complete reasonings about your thoughts or about your data. They are not good at doing things but they are good at « feeling » things.
- FeepingCreature 3y ago
  A fast 50% sort could be very useful given the right circumstances.
  - jjk166 3y ago
    Yeah, so long as you have something else that can quickly verify that a list is sorted, just keep applying this "quick coin toss sort" until you get a sorted list. If a ML algorithm with a 90% success rate and a "sanity checker" might be much easier to implement than a ML algorithm with a 100% success rate.
    - 3y ago
sanp 3y ago
Completely agree. One thing to add - we should not be comparing LLMs to experts in any field (where expertise can be objectively determined). We should be comparing LLMs to an average (in some cases better than average) layperson. The same level skepticism is warranted but it does not mean that they are not useful.
dun44 3y ago
>However at some point you have to admit the LLM does generate things that are good answers. Hawing a noise generator sometimes generate something that looks like a valid signal doesn’t mean it’s not a noise generator.
jstummbillig 3y ago
> LLMs seem to operate on the principle of "will this pass for a correct answer" My new thing: Testing LLM critique against humans. The results are sobering.
- trimethylpurine 3y ago
  When an LLM fails, it needs you to tell it that it failed, and it needs you to tell it to try again. Conversely, we experience a desire to be thought of as just, and that drives us to seek information on our own. While it mimics the language, it doesn't mimic that honesty because it doesn't need to be honest. That said, it's impressive that the LLM fools you enough to be sobered by the comparison.
  - jstummbillig 3y ago
    Well, now I am mostly sad about the great density of human talent I have apparently missed out on. In my experience, there is a wild amount of finding absolutely nothing wrong with what is clearly going wrong and me having to prompt some variation of "so... what were you planning to do about that thing?" for something to happen.
    - trimethylpurine 3y ago
      And are you an LLM or are you the only human that feels this way? You've proven my point by replying to my comment.
trimethylpurine 3y ago
>For instance when you ask it for a snippet of code and it gets it right. Does this happen often for you? I've logged my success rate with Go and C. 32/37 responses failed to meet the requested criteria. 15 wouldn't even compile. Getting the bot to do it right takes longer than writing it myself. Maybe that's just me? I wouldn't think so. Word prediction and writing code aren't nearly the same thing, when I think about them. Sure, they both require language. But one of those also requires reasoning. As worded in this paragraph, could an LLM tell me which one? Sure. Sometimes...
- lordnacho 3y ago
  Particularly with copilot, it often just guesses what I want to do next and gets it right. Small snippets of code but it saves me time.
itsoktocry 3y ago
>Someone who wasn't working in my domain would get fooled. So they run the code, it breaks, they paste the error message back into chatGPT and try again. What's wrong with that? Don't know about you, but I don't get code right the first time, either.
- RandomLensman 3y ago
  Not all areas of work and life come with error messages that are a cheap to encounter. Coding might be somewhat special and the use case for LLMs there might be considerably stronger than in some other areas.
- deckard1 3y ago
  You're assuming bad code is code that throws errors. SQL injection is bad code that "works". Most security issues come from bad code that just "works". I can't believe I have to say this, on HN of all places, but you have to actually know what you are doing when writing code. Some of you scare the fucking shit out of me.
agentultra 3y ago
> I can tell when it spits out a snippet of code that is correct, and when it's wrong. Hah, sure. Okay. How? You just read it and think really hard? Bertrand Meyer, the inventor of Eiffel, who rants about software correctness all the time -- didn't notice an error in a 1-line expression in Eiffel that was generated by ChatGPT [0]. If you spend a little time formalizing correctness in a theorem prover or model checker you may not be so confident that you can read a snippet of code and know that it's correct. In order to know that you have to be able to write the specification precisely. Natural language is not precise enough. Update: You may be able to write the specification precisely either by modelling the system and checking the model or writing a theorem and proving it... but so far LLM's cannot reason on that level. [0] https://buttondown.email/hillelwayne/archive/programming-ais-worry-me/ https://buttondown.email/hillelwayne/archive/programming-ais...
setgree 3y ago
> However at some point you have to admit the LLM does generate things that are good answers. They might be good answers that happen to pass the smell test, but they are nonetheless good answers. For instance when you ask it for a snippet of code and it gets it right. +1, though it took some time to figure out what questions work well. Yesterday, I asked chatGPT to help me document some R functions, and in my opinion, it did a great job [0]. I asked it to summarize what my functions were doing, and it gave me nice, plain-language summaries, and then reformatted my notes into the the form that Roxygen2 [1] expects, and added some useful inline comments. Moreover, it was fast. It read and understood my code in a few seconds. No human can compete with that. I don't think this is bullshit. I think that chatGPT shows expertise with some formal conventions that can be kind of a pain to memorize and work with. You might think those conventions were BS in the first place, but regardless, they're what we settled on, and it's really nice to have a coding assistant do the tedious work. I don't think chatGPT could have written the functions in the first place. But who knows what GPT [5,6...N] will be capable of [0] https://github.com/setgree/sv-meta/commit/5f71e7c251b38e19817d57b0d1093076aa4b45f4 https://github.com/setgree/sv-meta/commit/5f71e7c251b38e1981... [1] https://cran.r-project.org/web/packages/roxygen2/vignettes/roxygen2.html https://cran.r-project.org/web/packages/roxygen2/vignettes/r...
ren_engineer 3y ago
ChatGPT and art stuff gets the attention and are seen as gimmicks, but that stuff is funding research that will actually drive an impact in less sexy areas. Here's Nvidia showing they can use LLMs to teach robots complex skills - https://eureka-research.github.io/ https://eureka-research.github.io/ lots of game changing technology starts off being used for toy or bullshit use cases, the GPUs being used to train AI were originally designed for allowing video games to have better graphics
elegantlie 3y ago
> What is stopping the LLM from fooling me in every field that I don't know anything about? What's stopping any source from doing this? In the case of encyclopedias or news websites, I guess you could say reputation. But that's hardly reliable either. So I guess that's my big pushback in regards to correctness complaints: 1) "Official" sources have always lied, or at least bent the truth, or at least pushed their subjective viewpoint. It has always been the readers job to think critically and cross-reference. On this point, LLM don't fundementally change anything, they just bring this long-running tension closer to our attention. 2) "LLM are inaccurate". Ok, inaccurate compared to what? Academic journals (see the reproducibility crisis), encylopedias (they tend to be accurate by nature of leaving out contentious facts), journalists (big laugh from me)? I think outright hallucinations are a valid concern to bring up, but I would refer to my point about cross-referencing. However, I'd still point out that oftentimes a person might have read a perfectly factually accurate encyclopedia article but remembered hallucinations via motivated reasoning. Is the end result (what the person thinks and remembers) really different between LLM and traditional sources? This seems like more of a human problem than a LLM problem. Do we have any factual evidance showing that people learning via LLM+traditional methods are actually less informed than people who learn from traditional methods alone? Right now, there's a lot of fear mongering and charged rhetoric, and not a lot of facts and studies. The burden of proof needs to fall on the people who want to regulate and restrict access to these models. Finally, what I think this is really about is the continued transfer from the "industrial age" to the "information age". 100 years ago, your average person wasn't expected or really even able to "do their own research", and instead relied on top-down elite driven institutions to disseminate their version of the information. De-industrialization, then the internet, then social media, and now LLM are cracking this (now) outdated social order, and our old elites are understandably threatened. I think this is another reason why we need to make this debate more rigorous and fact based: are LLM actually dangerous or is this just an example of elite preference?
- daedalus_j 3y ago
  LLM's hallucinate so often when I ask them about software development issues that I don't even bother to ask them about areas I'm unfamiliar with. I think the major difference that makes in not just "elite preference" is that the LLM output is just confidently wrong. If I read a report from an elite institution I'm at least reasonably confident that they're not going to base their entire argument on the premise that 2+2=5. Someone is going to call them on it, there may be reputational damage, there may even be legal repercussions in certain cases. LLMs have no such protections. GPT recently, when asked to "write a function using the ELixir programming language that..." wrote a function in Elixir syntax using Python libraries and function names. That's a class of error that makes it actually dangerous if you're asking it about anything you can't fully check on your own, and there's no checks-and-balances for the content it generates that's only ever visible to a single user. I agree with you that "official" sources can't be trusted and there are inaccuracies everywhere, but you have to admit that the craziest sources who say the craziest things (flat-earthers and such for example) get pretty easily dismissed and everything else they say becomes suspect. LLM's bypass that and can say the most outrageous things without ever getting caught or being forced to make a retraction/correction. That feels like a problem worth worrying about.
Biologist123 3y ago
I think what you and the writer of the article are complaining about is not the LLM but the dataset that the LLM is describing. My guess is that if you created a GPT of a well-regarded book on WW1, you’d be able to have an enlightening conversation about the topic.
antisthenes 3y ago
> So in the current iteration, I think LLMs are a shortcutting tool for experts. I can tell when it spits out a snippet of code that is correct, and when it's wrong. Someone who wasn't working in my domain would get fooled. The real horror is when there's going to be an entire generation that grew up not by relying on experts/historic facts and reality, but on something that AI generated that looks good enough and is a neat answer. Imagine this generation in positions of power over actual experts, or equating AI opinion with the opinion of an older-generation expert. Progress in all of human history has always hinged on our ability to disregard idiots in favor of experts. Now, the position of the idiot is stronger than ever, because anyone can type a prompt into GPT without real understanding. That's the real scary part.

falserum 3y ago

Agree. It is a language model, so it’s best use is language things (emails, motivational letter, summaries) For everything else, you need to be enough of an expert to validate the output.

mike_hock 3y ago

The same principle applies outside of LLMs. Ever seen mainstream media write something about your field of expertise that wasn't complete horseshit? No? Then why believe anything else they say.

indigochill 3y ago

> LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". Isn't the hint in the name? It's a language model, not a knowledge model. LLMs are exceptional at generating stuff that passes as coherent language (like, the parts of speech are all where one would expect them to be in the sentences). The trouble is people think it goes deeper when the knowledge modeling part only happens incidentally because language and knowledge modeling are so closely related.

mvdtnz 3y ago
> Isn't the hint in the name? It's a language model, not a knowledge model. Well yes, but how is this insight helpful? People aren't just looking at the name. Have your missed the last 10 months of ridiculous breathless coverage of LLMs? > The trouble is people think it goes deeper when the knowledge modeling part People are literally calling this shitty text prediction algorithm "AI". The discourse is completely out of hand.
- simonw 3y ago
  It is "AI", using the definition of that term that was established when the term was coined in 1956. https://en.m.wikipedia.org/wiki/Dartmouth_workshop https://en.m.wikipedia.org/wiki/Dartmouth_workshop > An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. LLMs fit that definition well. They're not AGI, but I think it's fair to call them AI.
  - mvdtnz 3y ago
    There is no widely accepted definition of "AI" and even if I accepted this one (I absolutely do not) no LLM meets this bar in the general sense.
    - trealira 3y ago
      Game tree heuristic searches, like the minimax algorithm with alpha-beta pruning, has also been called AI, which is why programs that play chess are called AI. What I think is the problem is that when people hear "artificial intelligence," they don't think of chess programs anymore; they think of something like Skynet from the Terminator. Or, maybe now they think of ChatGPT.
    - simonw 3y ago
      That 1956 workshop is widely credited as the place where the term AI was coined. If they don't get to define "AI", who does?

jwestbury 3y ago

There are loads of examples of human-assisted algorithmic processes, or the inverse, algorithm-assisted human processes, most of which have better performance than either entity alone. I've had trouble finding it, but years ago, I read the abstract of a study which looked at chess performance, pitting people who had access to some basic chess software against those who didn't. What they found is that the strongest predictor of success was whether a player could utilise the software effectively, rather than pure chess skill. Another area where I don't think there are any published studies, but there's a lot of anecdotal evidence, is algorithmic trading, where successful firms almost always have human traders monitoring the algorithmic behaviour. This means that humans can spot when the algorithm is doing something a bit odd, or when conditions aren't conducive to the algorithm performing well, and they can tweak some parameters on the algorithm to get a better result, shut it down if it looks like it's lost the plot, etc. (And let's be honest, a lot of this oversight is probably a result of Knight Capital.) My point here is: You're entirely right, LLMs are currently a tool for people with experience in the field they're working with. But that's still a pretty big step forward -- just like top quant hedge funds consistently outperform human-managed funds (see: Rentech), smart people who learn how to leverage these tools are going to outperform. The key is learning where they perform well, where they perform poorly, and how to validate their output.

tmorton 3y ago
> I've had trouble finding it, but years ago, I read the abstract of a study which looked at chess performance, pitting people who had access to some basic chess software against those who didn't. What they found is that the strongest predictor of success was whether a player could utilise the software effectively, rather than pure chess skill. There was a time when this was true - a grandmaster and a computer was stronger than either alone. But that window closed pretty quickly. Today, even the strongest GMs' best strategy is to follow the computer blindly.
ycombobreaker 3y ago
> let's be honest, a lot of this oversight is probably a result of Knight Capital Gray-box trading and humans in a control loop over automated trading have both been happening for decades. Knight Capital makes for a pretty spectacular example, but that's about it.

3cats-in-a-coat 3y ago

> LLMs seem to operate on the principle of "will this pass for a correct answer" rather than "this is how this thing works, so here's a reasonable opinion answering your question". That's NOT how LLMs operate. They're trained on what is supposed to be selected representative human text of some at least minimum quality, and ideally truthful, factual and relevant. And then their prediction is judged on how close to that ground truth it is. Not on whether it's "plausible" or "will this pass for a correct answer". There's no math function for pretending to be right. You have to be right. One reason why LLMs make up things sometimes is because they're incredibly tightly optimized. Like you have no idea how tight. GPT-4 has the comparable complexity of a mouse brain. A mouse brain that has to fit the world's knowledge. Obviously, something has to give, and you'll not be accurate sometimes when you have so limited space. You'll store some things approximately, generally, and then extrapolate answers from them, maximizing for correct output.

lazy_moderator1 3y ago
this is an interesting statement, are you involved in LLM research?
ethanbond 3y ago
> And then their prediction is judged on how close to that ground truth it is When/where/how?
SAI_Peregrinus 3y ago
Extant LLMs are trained on whatever data the trainers can get their hands on. There's no magical "check for truth" step, either at the input or at the output.
- 3cats-in-a-coat 3y ago
  No, data is significantly filtered in order to bias it to accuracy. Of course if you feed an AI scientific papers and books of some quality, information will be of high quality. THAT... is how you "filter for truth". It's not supposed or required to be perfect. But of course you can filter the input. The fact you also add conversational input from various sources like Reddit and Twitter doesn't change this balance significantly. Also even those sources are filtered to exclude vulgarities, certain topics and so on. Also you're not thinking this through. If a question is on average answered incorrectly 'round the world, it means you're over 50% likely to also give the incorrect answer when asked. Who equipped you to know the truth?

coffeedan 3y ago

>> For instance when you ask it for a snippet of code and it gets it right. In my experience so far, more often than not, it doesn’t get it right. Brings to mind the old maxim, “a broken clock is right twice a day”.

xp84 3y ago
Personally, this just seems to me to model slight inexperience pretty well. But just like junior developers are not useless, I find GPT, copilot, etc. to be a force multiplier. It can generate, in 2 seconds, a 15-line function to do a simple thing which is nearly always either correct or needs one small tweak to satisfy me. This allows me to focus my brain 100% on the interesting parts of the problem domain. To me, it feels exactly like having a bright junior developer at a second keyboard twiddling his thumbs all day, waiting for me to need some trivial grunt work task.

garrickvanburen 3y ago

I like how you characterize the output as, “an answer that sounds right”. I was listening to a podcast yesterday where the purported expert gave an answer that sounded right but based on my direct experience - very wrong. Of course I could be wrong as well. Or the experts answer was the same kind of BS the OP was referring to and is so oft generated by LLMs.

garrickvanburen 3y ago
related: https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

add-sub-mul-div 3y ago

> So in the current iteration, I think LLMs are a shortcutting tool for experts My take has been that LLMs aren't valuable for me because I'm an expert and writing code is no longer very difficult. And searching for the knowledge when I need it is faster on my own than having a conversation with a computer because true expertise also involves having mastered and streamlined that process.

empath-nirvana 3y ago
i'm an experienced coder and co-pilot often writes code faster than I can think of it, and it's usually fine and follows the pattern I had established with previous code. A lot of code that everyone writes is just boilerplate. Until I used copilot, I didn't realize _how much_ i was writing that didn't really require much thought.
- zerbinxx 3y ago
  Imo if you’re “doing coding right”, at a certain point, you should be thinking very little about what you’re doing. This can manifest as having easily copypastable design patterns, lots of tests, inversion of control, etc. Even among senior developers it can be shockingly rare for people to actually understand and implement something like DI/IoC in a correct and useful way.

api 3y ago

They're a form of lossy compression. I suspect they are doing less than we think they are. They might be fooling us into thinking there's more intelligence there than there really is in the same way fractals can fool us into thinking "the whole universe must be a big fractal!" by their uncanny ability to replicate complex shapes.

matheusmoreira 3y ago

Just yesterday ChatGPT thought it was possible to use sizeof in C preprocessor macros. I called it out on this and it doubled down on it. Edit: #if directives, like static_assert.

pelagicAustral 3y ago
I've got a colleague that would've double down on it as well...
yorwba 3y ago
Where's the problem? ~ $ cat sizeofmacro.c #include <stdio.h> #define pintsize(T) printf("sizeof(%s) is %zu. Big or small? You decide.\n", #T, sizeof(T)) int main(int argc, char **argv) { pintsize(int); pintsize(argv); return 0; } ~ $ gcc sizeofmacro.c && ./a.out sizeof(int) is 4. Big or small? You decide. sizeof(argv) is 8. Big or small? You decide.
- blibble 3y ago
  the preprocessor isn't evaluating it, it's just pasting the tokens in the preprocessor doesn't know anything about structs or types
- diydsp 3y ago
  I was curious, too. Looked it up. Still don't understand tho. >The sizeof in C is an operator, and all operators have been implemented at compiler level; therefore, you cannot implement sizeof operator in standard C as a macro or function. You can do a trick to get the size of a variable by pointer arithmetic.
fanf2 3y ago
You can use sizeof() in macros but not in #if expressions.
- matheusmoreira 3y ago
  Yeah, I meant #if directives, like static_assert. I asked it to help me statically verify my assumptions about ELF files and it gave me some pretty insane code. Here's the excerpt: > Certainly, you can use a combination of sizeof and #error directives for conditional compilation to achieve a similar effect without static_assert. // Compile-time check for structure sizes #if sizeof(Elf32_Phdr) != sizeof(((Elf32_Ehdr*)0)->e_phentsize) #error "32-bit structure size mismatch" #endif #if sizeof(Elf64_Phdr) != sizeof(((Elf64_Ehdr*)0)->e_phentsize) #error "64-bit structure size mismatch" #endif > In this example, if the structure sizes don't match, the preprocessor will generate an error at compile time using the #error directive. > I didn't know sizeof could be used with the C preprocessor > Yes, sizeof can be used in certain contexts within the C preprocessor. > It's commonly employed in combination with the #if and #error directives to perform compile-time checks on sizes or other constants. > This usage allows you to catch potential issues during compilation rather than at runtime. I should probably go sleep a bit before I embarrass myself any further. After nearly 24 hours awake it seems I have lost the ability to tell macros and directives apart.

jackcviers3 3y ago

> You need to already know what you're doing to know whether the LLM got it right. I want to call out that this is not necessarily true. You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs to produce novel code that you don't know the structure of beforehand. You start with a set of example inputs and desired outputs, and prompt the llm for a function in your programming language of choice. You then take the LLM's response and feed it into an agent that executes the code against the inputs and outputs, and report the discrepancies back to the LLM. It responds with a new completion, which you feed to the agent until all your example inputs produce the expected outputs. Finally, you can use property-based testing to produce new examples for testing by the llm, resolving the answers until the produced code is correct to within some margin of acceptable correctness. You can do this to produce code without needing to know anything besides the desired properties of generated code and the properties of the inputs. You can further automate this by using a separate LLM to produce the examples instead of the typical generation and shrinking functions to produce examples. This doesn't require any prior knowledge of the target language. You can expand beyond programming into any domain for which you can produce a scoring function and automate input generation. I suspect that you can use control theory (PID loops, behavior choice loops scored by Eigenvalues) to model complex scenarios spanning multiple domains as well, choosing the evaluation agent as the behavior, each behavior of which has a separately defined scoring function. All of this can be automated by using the LLM generator without prior knowledge of the algorithmic structure of the solution and knowledge of all possible inputs to all possible behaviors. If that all sounds very familiar, it's because it's essentially just doing test-driven development, but with the LLM machine as the developer. It would be expensive to run an entire project development that way, due to the high costs of executing LLMs and/or querying LLM apis, but if the cost is less than that of employing a junior developer or teams of developers, it might be worth it. And human intervention can be kept in the loop at any stage, making it very feasible for rapid prototyping - where you don't necessarily need the entirely correct answer from the machine, just a good enough starting point to allow the human to take over to produce a result.

falserum 3y ago
> You can do this to produce code without needing to know anything besides properties … Reminder, you started with this: > You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs Maybe those are fancy words for simple things which I’m not familiar with or learning programming language seems a simpler option :)

photochemsyn 3y ago

There are methods for getting useful information from LLMs about subjects outside your own area of expertise, but it takes a lot of work. For the example, causes of the Great War, I'd first ask it to provide a list of the top ten reasons reputable historians cite as primary causes of World War One. Then, for each reason that looked interesting, I'd ask the LLM to provide arguments for and against, including one primary historical document in support of each side. Then, go to Google Scholar or similar and look up that document and see what different reputable historians have said about it. A few iterations of this process should get you to some reliable information even in a field you know nothing about. This is a time-consuming process, and requires active engagement rather than what many people have been trained to do by our atrocious education system and watching television, i.e. passive absorption of content without skepticism.

bsenftner 3y ago
I'm doing something similar with figuring out how to convert my home to solar power. Where I live, the sun is perfect for solar, and as a result our area if flooded with solar startups that are universally using hard sell tactics and undermining one another. After discussions with neighbors, everyone feels the available solar installers are all serving slanted bad deals. So I'm using ChatGPT4-API to create solar installation planning bots, financing evaluation bots, duties and responsibilities checklists for service people, and so on. I think my neighbors and I might just form our own startup or just give it all away because this situation here with the available service companies feels Orwellian; they can like there is no choice, they use fear and hard sell language.
latexr 3y ago
> I'd first ask it to provide a list of the top ten reasons reputable historians cite as primary causes of World War One. And right here you could bump into problems, because how would you know if what you got back is really the opinion of reputable historians? For all you know it could be giving you the opinions of a layman on Reddit. > Then, go to Google Scholar or similar and look up that document and see what different reputable historians have said about it. Looks like that could’ve been the first step. Identify the reputable historians and go from there.

SamBam 3y ago

It reminds me of the old philosophical debate about AI and consciousness. Many philosophers argue that knowing whether something which appears to be conscious really is conscious is a meaningless question. If there is no way to distinguish consciousness from non-consciousness from the outside, then you may as well just say that the thing is conscious. I think we may be approaching the same question with regard to truth. If it appears to be truth from the outside, is that the same thing as being truth? Is there any point in knowing wether it was produced from a thinking mind or not?

shmageggy 3y ago
Truthfulness is on totally different philosophical footing than consciousness, so the analogy doesn't hold up. There is no "hard problem of epistemology", but there is a hard problem of consciousness. There exist criteria for evaluating truthfulness (that we may reasonably disagree on) that can be applied to LLM output.
- joelfried 3y ago
  I think the Gettier problems are at least epistemological problems with some weight, at least as long as you're in the Justified True Belief sort of camp (which I think many of us start out as). If you haven't seen Gettier problems before, I'd start with the generalized problem and work out from there: https://en.wikipedia.org/wiki/Gettier_problem#The_generalized_problem https://en.wikipedia.org/wiki/Gettier_problem#The_generalize...
- SamBam 3y ago
  I think the whole point of the philosophical concept of "bullshit" is that it's orthogonal to the truth. It doesn't matter whether the words end up being correct or not correct, it's still bullshit because the speaker doesn't know or care if it's actually true.

broast 3y ago

Reminds me of me when I was a junior dev. Pretend I already know how to do the thing and form the answer as I google and speak.

aredox 3y ago

"when you ask for a snippet of code and it gets it right" Most examples I have seen are about basic code that is highly likely to be already present in the training data - making the language model an expert system, but with a worse rate of failure. Meanwhile it has a hard time creating something it hasn't already seen, or a close variant.

simonw 3y ago
> Meanwhile it has a hard time creating something it hasn't already seen, or a close variant. How much time have you spent with the strongest models, such as GPT-4? I've been using it on a daily basis for 6+ months now and your statement there doesn't fit my experience at all.
- ryandvm 3y ago
  Indeed. I feel like people are not even talking about the same thing in here. I've been using GPT-4 daily for a couple months and it's not outputting code that "only looks correct". It is writing functional solutions that often work as is. Just yesterday I pasted in a hastily written function and told GPT-4 to "please evaluate this function and make sure it's bulletproof". It outputted a bulleted list of criticisms (mostly true) and then spit out a version of the function that was shorter, easier to understand, and was safer. That's not just the "next most likely token". Or if it is, then that's all any of us are doing.
  - rbranson 3y ago
    It’s probably more accurate to say that what your brain is doing when writing code is more like emulating an LLM than an LLM is emulating your brain. Converting human language to code is an ideal task for LLMs, which is part of why they are such an exciting development.
- zerbinxx 3y ago
  If you ask it to start explaining/extending/implementing classes/interfaces, start asking it to reason about leaky abstractions or when it’s appropriate to use Mixins, it frequently generates code that either doesn’t work or just misses the point. It’s especially hard because some of these problems actually are not tractable in the language you’re taking about due to their features, or due to limitations of your code/your knowledge it may suggest you do something slightly worse because you haven’t prompted it on factories or other design patterns that you might want to use.

PaulHoule 3y ago

Maybe I should be impressed that an algorithm can sort a list correctly 50% of a time because it is better than chance (ex. you have 1 in 3.6 million odds of sorting a list of 10 items correctly at random) but conventionally the value of a 50% sorting algorithm is 0, not 50% that of a sort that works since the sort is part of a larger system that requires the sort to work 100% of the time. For a long time there have been "machine learning" techniques where getting the right answer 80% of the time is a win, what's interesting about LLMs is they sometimes "raise the bar" for the machine learning problems (get it right 90% of the time) but have also lowered the bar for other things that can be done algorithmically 100% of the time.

pjerem 3y ago
The thing is, if you want to sort a list, LLM is the wrong tool. However, if what you want is to generate a program that will sort a list, your LLM will be pretty performant. It’s a powerful tool but like any powerful tool, you will use it wrongly if you don’t acknowledge its strengths and weaknesses. Judging a LLM because he can’t act as a program is a fundamental error because it means you are wasting an enormous amount of resources for nothing. The novelty of LLMs and where they are good at is helping you thinking about things. Helping you to complete reasonings about your thoughts or about your data. They are not good at doing things but they are good at « feeling » things.
FeepingCreature 3y ago
A fast 50% sort could be very useful given the right circumstances.
- jjk166 3y ago
  Yeah, so long as you have something else that can quickly verify that a list is sorted, just keep applying this "quick coin toss sort" until you get a sorted list. If a ML algorithm with a 90% success rate and a "sanity checker" might be much easier to implement than a ML algorithm with a 100% success rate.
  - 3y ago

sanp 3y ago

Completely agree. One thing to add - we should not be comparing LLMs to experts in any field (where expertise can be objectively determined). We should be comparing LLMs to an average (in some cases better than average) layperson. The same level skepticism is warranted but it does not mean that they are not useful.

dun44 3y ago

>However at some point you have to admit the LLM does generate things that are good answers. Hawing a noise generator sometimes generate something that looks like a valid signal doesn’t mean it’s not a noise generator.

jstummbillig 3y ago

> LLMs seem to operate on the principle of "will this pass for a correct answer" My new thing: Testing LLM critique against humans. The results are sobering.

trimethylpurine 3y ago
When an LLM fails, it needs you to tell it that it failed, and it needs you to tell it to try again. Conversely, we experience a desire to be thought of as just, and that drives us to seek information on our own. While it mimics the language, it doesn't mimic that honesty because it doesn't need to be honest. That said, it's impressive that the LLM fools you enough to be sobered by the comparison.
- jstummbillig 3y ago
  Well, now I am mostly sad about the great density of human talent I have apparently missed out on. In my experience, there is a wild amount of finding absolutely nothing wrong with what is clearly going wrong and me having to prompt some variation of "so... what were you planning to do about that thing?" for something to happen.
  - trimethylpurine 3y ago
    And are you an LLM or are you the only human that feels this way? You've proven my point by replying to my comment.

trimethylpurine 3y ago

>For instance when you ask it for a snippet of code and it gets it right. Does this happen often for you? I've logged my success rate with Go and C. 32/37 responses failed to meet the requested criteria. 15 wouldn't even compile. Getting the bot to do it right takes longer than writing it myself. Maybe that's just me? I wouldn't think so. Word prediction and writing code aren't nearly the same thing, when I think about them. Sure, they both require language. But one of those also requires reasoning. As worded in this paragraph, could an LLM tell me which one? Sure. Sometimes...

lordnacho 3y ago
Particularly with copilot, it often just guesses what I want to do next and gets it right. Small snippets of code but it saves me time.

itsoktocry 3y ago

>Someone who wasn't working in my domain would get fooled. So they run the code, it breaks, they paste the error message back into chatGPT and try again. What's wrong with that? Don't know about you, but I don't get code right the first time, either.

RandomLensman 3y ago
Not all areas of work and life come with error messages that are a cheap to encounter. Coding might be somewhat special and the use case for LLMs there might be considerably stronger than in some other areas.
deckard1 3y ago
You're assuming bad code is code that throws errors. SQL injection is bad code that "works". Most security issues come from bad code that just "works". I can't believe I have to say this, on HN of all places, but you have to actually know what you are doing when writing code. Some of you scare the fucking shit out of me.

agentultra 3y ago

> I can tell when it spits out a snippet of code that is correct, and when it's wrong. Hah, sure. Okay. How? You just read it and think really hard? Bertrand Meyer, the inventor of Eiffel, who rants about software correctness all the time -- didn't notice an error in a 1-line expression in Eiffel that was generated by ChatGPT [0]. If you spend a little time formalizing correctness in a theorem prover or model checker you may not be so confident that you can read a snippet of code and know that it's correct. In order to know that you have to be able to write the specification precisely. Natural language is not precise enough. Update: You may be able to write the specification precisely either by modelling the system and checking the model or writing a theorem and proving it... but so far LLM's cannot reason on that level. [0] https://buttondown.email/hillelwayne/archive/programming-ais-worry-me/ https://buttondown.email/hillelwayne/archive/programming-ais...

setgree 3y ago

> However at some point you have to admit the LLM does generate things that are good answers. They might be good answers that happen to pass the smell test, but they are nonetheless good answers. For instance when you ask it for a snippet of code and it gets it right. +1, though it took some time to figure out what questions work well. Yesterday, I asked chatGPT to help me document some R functions, and in my opinion, it did a great job [0]. I asked it to summarize what my functions were doing, and it gave me nice, plain-language summaries, and then reformatted my notes into the the form that Roxygen2 [1] expects, and added some useful inline comments. Moreover, it was fast. It read and understood my code in a few seconds. No human can compete with that. I don't think this is bullshit. I think that chatGPT shows expertise with some formal conventions that can be kind of a pain to memorize and work with. You might think those conventions were BS in the first place, but regardless, they're what we settled on, and it's really nice to have a coding assistant do the tedious work. I don't think chatGPT could have written the functions in the first place. But who knows what GPT [5,6...N] will be capable of [0] https://github.com/setgree/sv-meta/commit/5f71e7c251b38e19817d57b0d1093076aa4b45f4 https://github.com/setgree/sv-meta/commit/5f71e7c251b38e1981... [1] https://cran.r-project.org/web/packages/roxygen2/vignettes/roxygen2.html https://cran.r-project.org/web/packages/roxygen2/vignettes/r...

ren_engineer 3y ago

ChatGPT and art stuff gets the attention and are seen as gimmicks, but that stuff is funding research that will actually drive an impact in less sexy areas. Here's Nvidia showing they can use LLMs to teach robots complex skills - https://eureka-research.github.io/ https://eureka-research.github.io/ lots of game changing technology starts off being used for toy or bullshit use cases, the GPUs being used to train AI were originally designed for allowing video games to have better graphics

elegantlie 3y ago

> What is stopping the LLM from fooling me in every field that I don't know anything about? What's stopping any source from doing this? In the case of encyclopedias or news websites, I guess you could say reputation. But that's hardly reliable either. So I guess that's my big pushback in regards to correctness complaints: 1) "Official" sources have always lied, or at least bent the truth, or at least pushed their subjective viewpoint. It has always been the readers job to think critically and cross-reference. On this point, LLM don't fundementally change anything, they just bring this long-running tension closer to our attention. 2) "LLM are inaccurate". Ok, inaccurate compared to what? Academic journals (see the reproducibility crisis), encylopedias (they tend to be accurate by nature of leaving out contentious facts), journalists (big laugh from me)? I think outright hallucinations are a valid concern to bring up, but I would refer to my point about cross-referencing. However, I'd still point out that oftentimes a person might have read a perfectly factually accurate encyclopedia article but remembered hallucinations via motivated reasoning. Is the end result (what the person thinks and remembers) really different between LLM and traditional sources? This seems like more of a human problem than a LLM problem. Do we have any factual evidance showing that people learning via LLM+traditional methods are actually less informed than people who learn from traditional methods alone? Right now, there's a lot of fear mongering and charged rhetoric, and not a lot of facts and studies. The burden of proof needs to fall on the people who want to regulate and restrict access to these models. Finally, what I think this is really about is the continued transfer from the "industrial age" to the "information age". 100 years ago, your average person wasn't expected or really even able to "do their own research", and instead relied on top-down elite driven institutions to disseminate their version of the information. De-industrialization, then the internet, then social media, and now LLM are cracking this (now) outdated social order, and our old elites are understandably threatened. I think this is another reason why we need to make this debate more rigorous and fact based: are LLM actually dangerous or is this just an example of elite preference?

daedalus_j 3y ago
LLM's hallucinate so often when I ask them about software development issues that I don't even bother to ask them about areas I'm unfamiliar with. I think the major difference that makes in not just "elite preference" is that the LLM output is just confidently wrong. If I read a report from an elite institution I'm at least reasonably confident that they're not going to base their entire argument on the premise that 2+2=5. Someone is going to call them on it, there may be reputational damage, there may even be legal repercussions in certain cases. LLMs have no such protections. GPT recently, when asked to "write a function using the ELixir programming language that..." wrote a function in Elixir syntax using Python libraries and function names. That's a class of error that makes it actually dangerous if you're asking it about anything you can't fully check on your own, and there's no checks-and-balances for the content it generates that's only ever visible to a single user. I agree with you that "official" sources can't be trusted and there are inaccuracies everywhere, but you have to admit that the craziest sources who say the craziest things (flat-earthers and such for example) get pretty easily dismissed and everything else they say becomes suspect. LLM's bypass that and can say the most outrageous things without ever getting caught or being forced to make a retraction/correction. That feels like a problem worth worrying about.

Biologist123 3y ago

I think what you and the writer of the article are complaining about is not the LLM but the dataset that the LLM is describing. My guess is that if you created a GPT of a well-regarded book on WW1, you’d be able to have an enlightening conversation about the topic.

antisthenes 3y ago

> So in the current iteration, I think LLMs are a shortcutting tool for experts. I can tell when it spits out a snippet of code that is correct, and when it's wrong. Someone who wasn't working in my domain would get fooled. The real horror is when there's going to be an entire generation that grew up not by relying on experts/historic facts and reality, but on something that AI generated that looks good enough and is a neat answer. Imagine this generation in positions of power over actual experts, or equating AI opinion with the opinion of an older-generation expert. Progress in all of human history has always hinged on our ability to disregard idiots in favor of experts. Now, the position of the idiot is stronger than ever, because anyone can type a prompt into GPT without real understanding. That's the real scary part.