3 ms·

> LLMs make it very difficult to make an immediate snap judgement about the quality [...] That's the core of the issue. It's time to say goodbye to heuristics

by lxgr 1y ago

> LLMs make it very difficult to make an immediate snap judgement about the quality [...] That's the core of the issue. It's time to say goodbye to heuristics like "the blog post is written in eloquent, grammatical English, hence the point its author is trying to make must be true" or "the code is idiomatic and following all code styles, hence it must be modeling the world with high fidelity". Maybe that's not the worst thing in the world. I feel like it often made people complacent.

acedTrex 1y ago
> Maybe that's not the worst thing in the world. I feel like it often made people complacent. For sure, in some ways perhaps reverting to a low trust environment might improve quality in that it now forces harsher/more in depth reviews. That however doesn't make the requirement less exhausting for people previously relying heavily on those markers to speed things up. Will be very interesting to see how the industry standardizes around this. Right now it's a bit of the wild west. Maybe people in ten years will look back at this post and think "what do you mean you judged people based on the code itself that's ridiculous"
furyofantares 1y ago
I think you're unfair to the heuristics people use in your framing here. You said "hence the point its author is trying to make must be true" and "hence it must be modeling the world with high fidelity". But it's more like "hence the author is likely competent and likely put in a reasonable effort." When those assumptions hold, putting in a very deep review is less likely to pay off. Maybe you are right that people have been too complacent to begin with, I don't know, but I don't think you've framed it fairly.
- lxgr 1y ago
  > But it's more like "hence the author is likely competent and likely put in a reasonable effort." And isn't dyslexic, and is a native speaker etc. Some will gain from this shift, some will lose.
  - furyofantares 1y ago
    Yes! This is part of why I bristle at such reductive takes, we can use more nuance thinking about what we are gaining and what we are losing and how to deal with it.
tempodox 1y ago
Anyway, “following all code styles” is just a fancy way of saying “adheres to fashion”. What meaningful conclusions can you draw from that?
- rurp 1y ago
  It's not about fashion, it's about diligence and consideration. Code formatting is totally different from say clothing fashion. Social fashions are often about being novel or surprising which is the opposite of how good code is written. Code should be as standard, clear and unsurprising as is reasonably possible. If someone is writing code in a way that's deliberately unconventional or overly fancy that's a strong signal that it isn't very good. When someone follows standard conventions it means that they A) have a baseline level of knowledge to know about them, and B) care to write the code in a clear and approachable way for others.
  - tempodox 1y ago
    > If someone is writing code in a way that's deliberately unconventional or overly fancy that's a strong signal that it isn't very good. “unconventional” or “fancy” is in the eye of the beholder. Whose conventions are we talking about? Code is bad when it doesn't look the way you want it to? How convenient. I may find code hard to read because it's formatted “conventionally”, but I wouldn't be so entitled as to call it bad just because of that.
    - kiitos 1y ago
      > “unconventional” or “fancy” is in the eye of the beholder. Literally not: a language defines its own conventions, they're not defined in terms of individual users/readers/maintainers subjective opinions. > Whose conventions are we talking about? The conventions defined by the language. > Code is bad when it doesn't look the way you want it to? No -- when it doesn't satisfy the conventions established by the language. > I may find code hard to read because it's formatted “conventionally”, If you did this then you'd be wrong, and that'd be a problem with your personal evaluation process/criteria, that you would need to fix.
o11c 1y ago
That's not how heuristics work. The heuristic is "this submission doesn't even follow the basic laws of grammar, therefore I can safely assume incompetence and ignore it entirely." You still have to do verification for what passes the heuristic, but it keeps 90% of the crap away.
deleted 1y ago
[deleted]

by lxgr 1y ago

acedTrex 1y ago

> Maybe that's not the worst thing in the world. I feel like it often made people complacent. For sure, in some ways perhaps reverting to a low trust environment might improve quality in that it now forces harsher/more in depth reviews. That however doesn't make the requirement less exhausting for people previously relying heavily on those markers to speed things up. Will be very interesting to see how the industry standardizes around this. Right now it's a bit of the wild west. Maybe people in ten years will look back at this post and think "what do you mean you judged people based on the code itself that's ridiculous"

furyofantares 1y ago

I think you're unfair to the heuristics people use in your framing here. You said "hence the point its author is trying to make must be true" and "hence it must be modeling the world with high fidelity". But it's more like "hence the author is likely competent and likely put in a reasonable effort." When those assumptions hold, putting in a very deep review is less likely to pay off. Maybe you are right that people have been too complacent to begin with, I don't know, but I don't think you've framed it fairly.

lxgr 1y ago
> But it's more like "hence the author is likely competent and likely put in a reasonable effort." And isn't dyslexic, and is a native speaker etc. Some will gain from this shift, some will lose.
- furyofantares 1y ago
  Yes! This is part of why I bristle at such reductive takes, we can use more nuance thinking about what we are gaining and what we are losing and how to deal with it.

tempodox 1y ago

Anyway, “following all code styles” is just a fancy way of saying “adheres to fashion”. What meaningful conclusions can you draw from that?

rurp 1y ago
It's not about fashion, it's about diligence and consideration. Code formatting is totally different from say clothing fashion. Social fashions are often about being novel or surprising which is the opposite of how good code is written. Code should be as standard, clear and unsurprising as is reasonably possible. If someone is writing code in a way that's deliberately unconventional or overly fancy that's a strong signal that it isn't very good. When someone follows standard conventions it means that they A) have a baseline level of knowledge to know about them, and B) care to write the code in a clear and approachable way for others.
- tempodox 1y ago
  > If someone is writing code in a way that's deliberately unconventional or overly fancy that's a strong signal that it isn't very good. “unconventional” or “fancy” is in the eye of the beholder. Whose conventions are we talking about? Code is bad when it doesn't look the way you want it to? How convenient. I may find code hard to read because it's formatted “conventionally”, but I wouldn't be so entitled as to call it bad just because of that.
  - kiitos 1y ago
    > “unconventional” or “fancy” is in the eye of the beholder. Literally not: a language defines its own conventions, they're not defined in terms of individual users/readers/maintainers subjective opinions. > Whose conventions are we talking about? The conventions defined by the language. > Code is bad when it doesn't look the way you want it to? No -- when it doesn't satisfy the conventions established by the language. > I may find code hard to read because it's formatted “conventionally”, If you did this then you'd be wrong, and that'd be a problem with your personal evaluation process/criteria, that you would need to fix.

o11c 1y ago

That's not how heuristics work. The heuristic is "this submission doesn't even follow the basic laws of grammar, therefore I can safely assume incompetence and ignore it entirely." You still have to do verification for what passes the heuristic, but it keeps 90% of the crap away.

deleted 1y ago

[deleted]