2 ms·
This is from the actual Slack blog post: > We examined the conversion rates of approximately 2,300 individual test cases spread out within 338 files. Among the
by Aurornis 2y ago
This is from the actual Slack blog post:
> We examined the conversion rates of approximately 2,300 individual test cases spread out within 338 files. Among these, approximately 500 test cases were successfully converted, executed, and passed. This highlights how effective AI can be, leading to a significant saving of 22% of developer time. It’s important to note that this 22% time saving represents only the documented cases where the test case passed.
So the blog post says they converted 22% of tests, which they claim as saving 22% of developer time, which InfoQ interpreted as converting 80% of tests automatically?
Am I missing something? Or is this InfoQ article just completely misinterpreting the blog post it’s supposed to be reporting on?
The topic itself is interesting, but between all of the statistics games and editorializing of the already editorialized blog post, it feels like I’m doing heavy work just to figure out what’s going on.
- deleted 2y ago[deleted]
- nl 2y agoMy reading of this is that the examination was a subset of the full set they manually examined. From the source: > It’s so compelling that we at Slack decided to convert more than 15,000 of our frontend unit and integration Enzyme tests to RTL, as part of the update to React 18. and > Our benchmark for quality was set by the standards achieved by the frontend developers based on our quality rubric that covers imports, rendering methods, JavaScript/TypeScript logic, and Jest assertions. We aimed to match their level of quality. The evaluation revealed that 80% of the content within these files was accurately converted, while the remaining 20% required manual intervention. There is a diagram that mentions 22% of the subset of manually inspected files that were 100% converted. But Slack is manually checking all converted test cases anyway so they don't seem to consider this the success rate. https://slack.engineering/balancing-old-tricks-with-new-feats-ai-powered-conversion-from-enzyme-to-react-testing-library-at-slack/ https://slack.engineering/balancing-old-tricks-with-new-feat...
- dboreham 2y agoHaving a success criteria "the test passed" is a huge red flag. So the test can be : print("Passed") (or some more subtle variation on that) and we succeeded.
- deleted 2y ago[deleted]