4 ms·

In principle I don't see a problem with bots looking out for grammar mistakes and typos that could confuse readers (so long as it's less intrusive than this bot

In principle I don't see a problem with bots looking out for grammar mistakes and typos that could confuse readers (so long as it's less intrusive than this bot), but in this case the bot is just incorrect. "Good to merge? Test suite passes locally?" is perfectly valid English. You need to make sure that the bot is configured to not insist on arbitrary prescriptivist style guides that nobody cares about.

samrus 1y ago
The problem is these bots dont understand the concept of "goals" or "achieving something". So they cant judge for themselves when they are actually helping (like in correcting grammar where its so bad its hindering communications with other contributors) or being pedantic (like this) because they wont think to consider the difference unless wxplicitly told to do so And NGI like a human would be given this task and consider the spirit of the law, the goal we want to acheive, and enforce it to reach that. The next token AI doesnt model that. It just predicts the next token, and understanding the spirit of the law does not seem to be in the emergent capabilities of that
- Asraelite 1y ago
  I often see comments on Github issues where poor wording makes it difficult to understand what is actually meant. Things like "I reproduced the bug on Linux, then I tried Windows. I can't reproduce it now." Does that mean it's just not reproducible on Windows, or not reproducible at all anymore? Ambiguities like that are especially annoying when it's someone posting a solution to a problem. Sometimes it's because of grammatical errors, sometimes not. I think LLMs are actually great for catching things like this, and you generally don't need some higher-level understanding about the goals involved to notice the ambiguity. My point wasn't that bots shouldn't be used like this, just that they need to be given the right instructions.
  - samrus 1y ago
    > just that they need to be given the right instructions. this is the part im talking about. i also think LLMs are very capable at detecting different types and levels or grammar, but they cant decide which ones should be filtered out to meet a certain goal. they need detailed instructions for that, and thats somewhat inefficient and causes issues like this right here. we have done this song and dance many times with AI. its the bitter lesson: you need a system that learns these things, you cant just give it a rule based engine to patch the things it cant learn. that works in the short term, but leads to a dead end. we need something that has the "common sense" to see when grammar is fine versus hindering communication, and this just isnt there yet. so it needs to be given detailed instructions to do so. which may or may not be sustainable
    - Asraelite 1y ago
      I agree with you but I think we're talking about different things haha. It sounds like you mean that curating messages purely to conform to a particular style guide requires context-dependent information and can never be accomplished reliably with some unmodified generic prompt across many different projects. I'm saying that while this is true, if you ignore grammar guidelines and just look for cases of ambiguous and confusing wording specifically, then this can actually be accomplished reliably with a generic prompt, if you get that prompt right. Not 100% accurately of course, but good enough that it would be beneficial overall in the long run.
antonvs 1y ago
Yeah, no-one wants Grammarly on pull requests.
- samrus 1y ago
  Well the maintainers might. The both in (badly) enforcing guidelines they set out. So they might want to police grammar a little bit. And i could see their point if the grammar is so bad it hinders communication. Bit uncomfortably, it was also be a reasonable predictor of those spam PRs put in from people just trying to pad resumes. But yeah the bot needs to loosen the boundary a bit there

samrus 1y ago

The problem is these bots dont understand the concept of "goals" or "achieving something". So they cant judge for themselves when they are actually helping (like in correcting grammar where its so bad its hindering communications with other contributors) or being pedantic (like this) because they wont think to consider the difference unless wxplicitly told to do so And NGI like a human would be given this task and consider the spirit of the law, the goal we want to acheive, and enforce it to reach that. The next token AI doesnt model that. It just predicts the next token, and understanding the spirit of the law does not seem to be in the emergent capabilities of that

Asraelite 1y ago
I often see comments on Github issues where poor wording makes it difficult to understand what is actually meant. Things like "I reproduced the bug on Linux, then I tried Windows. I can't reproduce it now." Does that mean it's just not reproducible on Windows, or not reproducible at all anymore? Ambiguities like that are especially annoying when it's someone posting a solution to a problem. Sometimes it's because of grammatical errors, sometimes not. I think LLMs are actually great for catching things like this, and you generally don't need some higher-level understanding about the goals involved to notice the ambiguity. My point wasn't that bots shouldn't be used like this, just that they need to be given the right instructions.
- samrus 1y ago
  > just that they need to be given the right instructions. this is the part im talking about. i also think LLMs are very capable at detecting different types and levels or grammar, but they cant decide which ones should be filtered out to meet a certain goal. they need detailed instructions for that, and thats somewhat inefficient and causes issues like this right here. we have done this song and dance many times with AI. its the bitter lesson: you need a system that learns these things, you cant just give it a rule based engine to patch the things it cant learn. that works in the short term, but leads to a dead end. we need something that has the "common sense" to see when grammar is fine versus hindering communication, and this just isnt there yet. so it needs to be given detailed instructions to do so. which may or may not be sustainable
  - Asraelite 1y ago
    I agree with you but I think we're talking about different things haha. It sounds like you mean that curating messages purely to conform to a particular style guide requires context-dependent information and can never be accomplished reliably with some unmodified generic prompt across many different projects. I'm saying that while this is true, if you ignore grammar guidelines and just look for cases of ambiguous and confusing wording specifically, then this can actually be accomplished reliably with a generic prompt, if you get that prompt right. Not 100% accurately of course, but good enough that it would be beneficial overall in the long run.

antonvs 1y ago

Yeah, no-one wants Grammarly on pull requests.

samrus 1y ago
Well the maintainers might. The both in (badly) enforcing guidelines they set out. So they might want to police grammar a little bit. And i could see their point if the grammar is so bad it hinders communication. Bit uncomfortably, it was also be a reasonable predictor of those spam PRs put in from people just trying to pad resumes. But yeah the bot needs to loosen the boundary a bit there