4 ms·

> but they will not cover the user perspective or edge cases unless those are already documented somewhere Are you suggesting that LLM's can't test for people

> but they will not cover the user perspective or edge cases unless those are already documented somewhere Are you suggesting that LLM's can't test for people who use screen readers? Keyboard only users? Slow network requests? You're acting like the issues an app faces are so bespoke to the actual app itself (and have absolutely no relation to existing problems in this space) that an LLM couldn't possibly cover it. And it's just patently wrong.

thunderfork 6mo ago
>Are you suggesting that LLM's can't test for people who use screen readers? Keyboard only users? Slow network requests? I don't think it's feasible to fully simulate the full depth of actual usage, given that (especially in the case of screen readers and the like) there's a great deal of combinatorial depth and context to the problem. Which screen readers, on which operating systems, and which users thereof?
tovej 6mo ago
I'm not talking about keyboards or screen readers or any sort of input testing, I'm talking about how the software is used in practice. If you disagree with that, I think the onus is on you to show me that an LLM could simulate the full context in which a user interfaces with software. That's a ridiculous claim. Feel free to show literally any evidence for this claim.
- mexicocitinluez 6mo ago
  I'm disagreeing with the saying it's impossible across the board, I'm not saying it's universally possible. lol And you made the claim, not me. The proof is on YOU.
  - tovej 6mo ago
    No, that's not how burden of proof works. The status quo is that this capability does not exist. Whoever makes a claim contradicting the status quo has the burden of proof. I can't prove a negative. And even with your logic, I did not make the original claim, it was made by simon. Your statement now also makes little sense. For any nontrivial software project, the usage patterns and interactions with other systems are complex enough that the code itself does not contain enough context to understand how it is used, or what the invariants are. There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare.
    - mexicocitinluez 6mo ago
      > There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare. Its not rare. I've built 2 dozen line-of-business apps in it last handful of years that were glorified CRUD apps. Every environment I've been in has had a mix of the 2. And even then, that's at odds with your absolute above. On top of being in a field that changes daily.
      - tovej 6mo ago
        You are interpreting a general statement as a categorical one. I wasn't really going for an exact, formal statement, but I can give you a formal interpretation of what I said above, if you want to be pedantic. In general, you can't expect an LLM to produce thorough documentation or robust tests for nontrivial software, because the use of those software (i.e. how their interfaces are expected to behave) contains assumptions from the context in which they are used, and that information will not be encoded in the source. If the above was somehow ambiguous, this should be clear and uncontroversial.
        mexicocitinluez 6mo ago
        > You are interpreting a general statement as a categorical one. That is in fact what I did and if you meant otherwise, then yes I agree that currently there are plenty of cases in which those tools fall short and will never replace a human.
fibonacci_man 6mo ago
I can’t tell if you’re being sarcastic or not
- mexicocitinluez 6mo ago
  You're saying that every app on this planet has bespoke usages that can't be derived from the app itself? That's your claim or am I getting this wrong?

thunderfork 6mo ago

>Are you suggesting that LLM's can't test for people who use screen readers? Keyboard only users? Slow network requests? I don't think it's feasible to fully simulate the full depth of actual usage, given that (especially in the case of screen readers and the like) there's a great deal of combinatorial depth and context to the problem. Which screen readers, on which operating systems, and which users thereof?

tovej 6mo ago

I'm not talking about keyboards or screen readers or any sort of input testing, I'm talking about how the software is used in practice. If you disagree with that, I think the onus is on you to show me that an LLM could simulate the full context in which a user interfaces with software. That's a ridiculous claim. Feel free to show literally any evidence for this claim.

mexicocitinluez 6mo ago
I'm disagreeing with the saying it's impossible across the board, I'm not saying it's universally possible. lol And you made the claim, not me. The proof is on YOU.
- tovej 6mo ago
  No, that's not how burden of proof works. The status quo is that this capability does not exist. Whoever makes a claim contradicting the status quo has the burden of proof. I can't prove a negative. And even with your logic, I did not make the original claim, it was made by simon. Your statement now also makes little sense. For any nontrivial software project, the usage patterns and interactions with other systems are complex enough that the code itself does not contain enough context to understand how it is used, or what the invariants are. There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare.
  - mexicocitinluez 6mo ago
    > There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare. Its not rare. I've built 2 dozen line-of-business apps in it last handful of years that were glorified CRUD apps. Every environment I've been in has had a mix of the 2. And even then, that's at odds with your absolute above. On top of being in a field that changes daily.
    - tovej 6mo ago
      You are interpreting a general statement as a categorical one. I wasn't really going for an exact, formal statement, but I can give you a formal interpretation of what I said above, if you want to be pedantic. In general, you can't expect an LLM to produce thorough documentation or robust tests for nontrivial software, because the use of those software (i.e. how their interfaces are expected to behave) contains assumptions from the context in which they are used, and that information will not be encoded in the source. If the above was somehow ambiguous, this should be clear and uncontroversial.
      - mexicocitinluez 6mo ago
        > You are interpreting a general statement as a categorical one. That is in fact what I did and if you meant otherwise, then yes I agree that currently there are plenty of cases in which those tools fall short and will never replace a human.

fibonacci_man 6mo ago

I can’t tell if you’re being sarcastic or not

mexicocitinluez 6mo ago
You're saying that every app on this planet has bespoke usages that can't be derived from the app itself? That's your claim or am I getting this wrong?