4 ms·

> It has been shown that they are good at doing a lot of things that might not have been immediately obvious Could you point me to some of the places/articles

> It has been shown that they are good at doing a lot of things that might not have been immediately obvious Could you point me to some of the places/articles where this is being shown? I'm definitely amongst those who have bought in to the common cop out you are rebutting here

geoduck14 2y ago
Not OP, but I have been poking around with LLMs for a year and I'd like to add to the conversation. In my experience, LLMs are word predictors, and the impacts of that fact are not immediately obvious. LLMs are capable of "explaining" what code does. What it is doing under the hood is pattern matching: I've seen code that looks like X, with an explanation that looks like Y LLMs are capable of formatting text. It has seen English written like X, that is reformatted to look like Y One resounding fact my team has found over and over is that "the things we think are hard for LLMs aren't necessarily hard; the things we think are easy aren't necessarily easy"
- dTal 2y ago
  Worth noting that pattern matching and term rewriting are a sufficiently general combination that it forms the basis for Mathematica.
  - psb217 2y ago
    Trouble sneaks in when the pattern matching is only correct most of the time. Eg, if some code for regex-based search missed anywhere from 0.1% to 10% of matches, with the miss rate depending on the regex and no obvious way to know which regexes have worse miss rates, the utility of your regex-based search would be limited. LLMs are like this, but their generality makes them useful in spite of this limitation.
  - 60654 2y ago
    I would just add that Mathematica-style term rewriting (e.g. analytical integration, or equation solving) is done with _semantics-preserving_ symbolic solvers, which are hand-made and human-reviewed to guarantee correctness. LLM style pattern matching and rewriting does not preserve semantics, except accidentally due to an overwhelming amount of examples.

geoduck14 2y ago

Not OP, but I have been poking around with LLMs for a year and I'd like to add to the conversation. In my experience, LLMs are word predictors, and the impacts of that fact are not immediately obvious. LLMs are capable of "explaining" what code does. What it is doing under the hood is pattern matching: I've seen code that looks like X, with an explanation that looks like Y LLMs are capable of formatting text. It has seen English written like X, that is reformatted to look like Y One resounding fact my team has found over and over is that "the things we think are hard for LLMs aren't necessarily hard; the things we think are easy aren't necessarily easy"

dTal 2y ago
Worth noting that pattern matching and term rewriting are a sufficiently general combination that it forms the basis for Mathematica.
- psb217 2y ago
  Trouble sneaks in when the pattern matching is only correct most of the time. Eg, if some code for regex-based search missed anywhere from 0.1% to 10% of matches, with the miss rate depending on the regex and no obvious way to know which regexes have worse miss rates, the utility of your regex-based search would be limited. LLMs are like this, but their generality makes them useful in spite of this limitation.
- 60654 2y ago
  I would just add that Mathematica-style term rewriting (e.g. analytical integration, or equation solving) is done with _semantics-preserving_ symbolic solvers, which are hand-made and human-reviewed to guarantee correctness. LLM style pattern matching and rewriting does not preserve semantics, except accidentally due to an overwhelming amount of examples.