3 ms·

Yeah, I disagree on the latter point by a lot :-). Really, is it truly "orthogonal"? Not really, you will understand the details better by writing them yourself

Yeah, I disagree on the latter point by a lot :-). Really, is it truly "orthogonal"? Not really, you will understand the details better by writing them yourself. On your first point, they are pretty good at a lot of stuff, given that they have seen it before. A lot of the time, I'm writing code that no LLM has seen before. That sounds super smug, but it's "da tru tru".

soulofmischief 2mo ago
I welcome you to prove this conjecture, otherwise it's just vibes. > A lot of the time, I'm writing code that no LLM has seen before I hear this tired point over and over from people who cannot fathom that others who use LLMs successfully could possibly also be working in a specialized domain. Frontier models are excelling at difficult, long-horizon tasks now. I write all sorts of esoteric stuff, and I can confidently hand a frontier model specification for a language it's never even seen before, working in a domain it's never encountered, and likely get good results, provided I have the knowledge and experience to guide the model. The reality is that this "they are only good at things they have 'seen before'" talking point that often gets parroted is vaguely defined and largely based in opinion. Obviously, models perform worse when the input or expected output are out of distribution. But this was much more true a couple years ago than it is today; the gap has closed considerably, and those who are learning to think deeply with these tools will be better positioned than those who arrogantly think that their process cannot be augmented by the most intelligent systems ever created.
- my-next-account 2mo ago
  This is HN, most of it is vibes. Lol, conjecture, touch grass man. I do PL research, it's not that good at that stuff, do you think I'm not trying using it?
  - soulofmischief 2mo ago
    You're claiming that one cannot understand the details of well-documented and well-written code which they did not write themselves, and I'm pushing back asking for proof. We can move on from this though, I'm more interested in where you currently feel they fall short doing PL research. I think the state of frontier models today in this area is a lot better than it was even six months ago and I think there's still room for improvement.

soulofmischief 2mo ago

I welcome you to prove this conjecture, otherwise it's just vibes. > A lot of the time, I'm writing code that no LLM has seen before I hear this tired point over and over from people who cannot fathom that others who use LLMs successfully could possibly also be working in a specialized domain. Frontier models are excelling at difficult, long-horizon tasks now. I write all sorts of esoteric stuff, and I can confidently hand a frontier model specification for a language it's never even seen before, working in a domain it's never encountered, and likely get good results, provided I have the knowledge and experience to guide the model. The reality is that this "they are only good at things they have 'seen before'" talking point that often gets parroted is vaguely defined and largely based in opinion. Obviously, models perform worse when the input or expected output are out of distribution. But this was much more true a couple years ago than it is today; the gap has closed considerably, and those who are learning to think deeply with these tools will be better positioned than those who arrogantly think that their process cannot be augmented by the most intelligent systems ever created.

my-next-account 2mo ago
This is HN, most of it is vibes. Lol, conjecture, touch grass man. I do PL research, it's not that good at that stuff, do you think I'm not trying using it?
- soulofmischief 2mo ago
  You're claiming that one cannot understand the details of well-documented and well-written code which they did not write themselves, and I'm pushing back asking for proof. We can move on from this though, I'm more interested in where you currently feel they fall short doing PL research. I think the state of frontier models today in this area is a lot better than it was even six months ago and I think there's still room for improvement.