3 ms·

Code coverage is useful as a metric as long as it's not turned into a target. But it's primary usefulness is to tell a developer if they've given enough thought

by zaphar 2y ago

Code coverage is useful as a metric as long as it's not turned into a target. But it's primary usefulness is to tell a developer if they've given enough thought to how to test their code is correct or not. A robot that back-fills coverage in tests seems... counterproductive to me.

federiconafria 2y ago
I was thinking the same, one usage could be to discover cases that don't make functional sense. Assuming these tests get reviewed.
malfist 2y ago
I agree, this seems to start from the assumption that a new unit test should pass, but that isn't really the case. A unit test should assert that the code functions accordingly to it's planned design. The AI writing unit tests can't know it's planned design, so it can't write test cases that prove the code is wrong
kevincox 2y ago
I think it may be useful if you are carefully reviewing the tests generated. Basically the robot is saying "Did you know that your code had this behaviour? If so let's lock it in" In many ways this feels similar to mutation testing, but sort of the opposite. To approximate behaviour: Mutation testing changes or simplifies code that is "dead" to the tests. That is the change doesn't affect the result of the tests. This adds tests that lock-in behaviour of previously "dead" branches. (Or tests that expose the incorrect behaviour of those branches.) In another way it is like snapshot testing. It recording the current behaviour of the code and that behaviour can be approved/rejected by the human reviewer.
callc 2y ago
Robot is giving too much credit. It’s rather a fancy dumb autocorrect emulating some corpus. The only place LLMs should be trusted is how we all probably originally tried them out: “Write me a poem about a space traveling teddy bear in the style of Eminem.” Trying to use LLMs for factual, numeric, logical purposes is a fundamentally flawed endeavor. Especially from our community, I’m disappointed to see this amount of blind trust and willful ignorance and lack of real engineering discipline in treating LLMs as anywhere near trustworthy. LLMs != AGI. Hot take, even if we did have AGI, it shouldn’t be given access / free rein over code and computing systems. I trust humans, with all our flaws, because we are limited in our time and energy, and speed.
- keybored 2y ago
  I haven’t had the energy to inform myself on all of this but I have a similar instinct. > with all our flaws, because we are limited in our time and energy, and speed. Something that might be relevant also is that we have skin in the game. External motivation can lead us to create intentionally correct or malicious programs. A robot’s fate is irrelevant since it has no capacity to care. I’m sure that algorithms try to emulate this skin-in-the-game factor. But it will always just be a simulation.
- tivert 2y ago
  > Especially from our community, I’m disappointed to see this amount of blind trust and willful ignorance and lack of real engineering discipline in treating LLMs as anywhere near trustworthy. I'm not disappointed anymore, because I've lowered my expectations for software engineers and the tech community in general. VC-adjacent hype and seeing the world through sci-fi fantasies trumps "engineering discipline" every time, at least in its discourse (you've got both for LLMs). There's a lot of unacknowledged ignorance of areas outside the tech bubble, and also a lot of contempt for it. Also, there's a pathological inability to think through to consequences to temper a compulsion to mindlessly "build the next thing," (especially acute with AGI, which for our sake I hope is infeasible or at least impractical). I really strongly believe someone needs to "take the keys away" from the tech community. And I say that as a member of it. > LLMs != AGI. Hot take, even if we did have AGI, it shouldn’t be given access / free rein over code and computing systems. I trust humans, with all our flaws, because we are limited in our time and energy, and speed. Also, more importantly, we are humans. It's literally some of the scariest, most hopeless things to imagine our world being dominated by something else, especially something else that unreachably exceeds our capabilities (I'd count a billionaire controlling an army of AGI drones to be essentially inhuman).
- zaphar 2y ago
  The term robot in no way indicates AGI. It's a term meaning automation. The robot's in most factories are not intelligent. The linter I use for javascript is also not intelligent. I refer to both of them as robots.

by zaphar 2y ago

federiconafria 2y ago

I was thinking the same, one usage could be to discover cases that don't make functional sense. Assuming these tests get reviewed.

malfist 2y ago

I agree, this seems to start from the assumption that a new unit test should pass, but that isn't really the case. A unit test should assert that the code functions accordingly to it's planned design. The AI writing unit tests can't know it's planned design, so it can't write test cases that prove the code is wrong

kevincox 2y ago

I think it may be useful if you are carefully reviewing the tests generated. Basically the robot is saying "Did you know that your code had this behaviour? If so let's lock it in" In many ways this feels similar to mutation testing, but sort of the opposite. To approximate behaviour: Mutation testing changes or simplifies code that is "dead" to the tests. That is the change doesn't affect the result of the tests. This adds tests that lock-in behaviour of previously "dead" branches. (Or tests that expose the incorrect behaviour of those branches.) In another way it is like snapshot testing. It recording the current behaviour of the code and that behaviour can be approved/rejected by the human reviewer.

callc 2y ago

Robot is giving too much credit. It’s rather a fancy dumb autocorrect emulating some corpus. The only place LLMs should be trusted is how we all probably originally tried them out: “Write me a poem about a space traveling teddy bear in the style of Eminem.” Trying to use LLMs for factual, numeric, logical purposes is a fundamentally flawed endeavor. Especially from our community, I’m disappointed to see this amount of blind trust and willful ignorance and lack of real engineering discipline in treating LLMs as anywhere near trustworthy. LLMs != AGI. Hot take, even if we did have AGI, it shouldn’t be given access / free rein over code and computing systems. I trust humans, with all our flaws, because we are limited in our time and energy, and speed.

keybored 2y ago
I haven’t had the energy to inform myself on all of this but I have a similar instinct. > with all our flaws, because we are limited in our time and energy, and speed. Something that might be relevant also is that we have skin in the game. External motivation can lead us to create intentionally correct or malicious programs. A robot’s fate is irrelevant since it has no capacity to care. I’m sure that algorithms try to emulate this skin-in-the-game factor. But it will always just be a simulation.
tivert 2y ago
> Especially from our community, I’m disappointed to see this amount of blind trust and willful ignorance and lack of real engineering discipline in treating LLMs as anywhere near trustworthy. I'm not disappointed anymore, because I've lowered my expectations for software engineers and the tech community in general. VC-adjacent hype and seeing the world through sci-fi fantasies trumps "engineering discipline" every time, at least in its discourse (you've got both for LLMs). There's a lot of unacknowledged ignorance of areas outside the tech bubble, and also a lot of contempt for it. Also, there's a pathological inability to think through to consequences to temper a compulsion to mindlessly "build the next thing," (especially acute with AGI, which for our sake I hope is infeasible or at least impractical). I really strongly believe someone needs to "take the keys away" from the tech community. And I say that as a member of it. > LLMs != AGI. Hot take, even if we did have AGI, it shouldn’t be given access / free rein over code and computing systems. I trust humans, with all our flaws, because we are limited in our time and energy, and speed. Also, more importantly, we are humans. It's literally some of the scariest, most hopeless things to imagine our world being dominated by something else, especially something else that unreachably exceeds our capabilities (I'd count a billionaire controlling an army of AGI drones to be essentially inhuman).
zaphar 2y ago
The term robot in no way indicates AGI. It's a term meaning automation. The robot's in most factories are not intelligent. The linter I use for javascript is also not intelligent. I refer to both of them as robots.