5 ms·

Misleading title. Maybe try this one? "Slack uses ASTs to convert test code from Enzyme to React with 22% success rate" This article is a poor summary of the

Misleading title. Maybe try this one? "Slack uses ASTs to convert test code from Enzyme to React with 22% success rate" This article is a poor summary of the actual article, which is at least linked to Slack's engineering blog [0]. [0] https://slack.engineering/balancing-old-tricks-with-new-feats-ai-powered-conversion-from-enzyme-to-react-testing-library-at-slack/ https://slack.engineering/balancing-old-tricks-with-new-feat... [updated]

doubloon 2y ago
Reminds me of Hitchhikers Guide … they had to figure out the right question to ask.
- dvh 2y ago
  Is it true that they give out free BMWs in Moscow? Yes, it is true! But it's not Moscow but St. Petersburg. And it's not BMWs but Ladas. And they don't give them out, they steal them.
  - Joker_vD 2y ago
    So all in all, still a mostly accurate news reporting, as they go.
  - dloranc 2y ago
    This is a variation of famous jokes about Radio Yerevan, very popular in former soviet states. I live in Poland and I know this version: Is it true that they give away cars on Red Square? Radio Yerevan answers: not cars, only bicycles, not on Red Square, but near the Warsaw station, and they don't give them away, they steal them.
zer00eyz 2y ago
The slack blog is for engineers. It's PR to hire in talent. The INFOQ article is for your C types. It's the spin on what buzz words they should be using with their peers and to make a splash in the market. NFT, Crypto, Cloud, Microservice, SAS, Podcast (the first time when there wasnt video), Web 2.0, The Ad (double-click, pre google) market, The Dot Com Bubble... Im sure I missed a few hype cycles in there. Both articles show how deep we are in the stink of this one. Im tired of swimming the bullshit, It keeps getting in my mouth.
- ben_jones 2y ago
  Thus the conclusion should be heavily scrutinize future infoq .com articles and perhaps future articles by the same author Eran Stiller. We shouldn’t detach responsibility from the publisher and author.
  - nl 2y ago
    Except the article turns out to be accurate (see https://news.ycombinator.com/item?id=40728179 https://news.ycombinator.com/item?id=40728179) So I guess the publisher and author should get credit. I'll leave others to discuss the misleading comment...
    - phillipcarter 2y ago
      Hah. A lot of tech folks like to trash journalists (it's fine, I get it, there's legitimate reasons) ... but then misread source content that the journalist interpreted better/correctly.
deleted 2y ago
[deleted]
nl 2y ago
Actually the Infoq article is more correct than this comment! The comment: ""Slack uses ASTs to convert test code from Enzyme to React with 22% success rate"" To quote[1], this 22% comes from this part: > We examined the conversion rates of approximately 2,300 individual test cases spread out within 338 files. Among these, approximately 500 test cases were successfully converted, executed, and passed. This highlights how effective AI can be, leading to a significant saving of 22% of developer time. It’s important to note that this 22% time saving represents only the documented cases where the test case passed. So that 22% rate is 22% saving of developer time, measured on a sample. No reasonable reading of that makes it a "22% success rate". Over the whole set of tests: > This strategic pivot, and the integration of both AST and AI technologies, helped us achieve the remarkable 80% conversion success rate, based on selected files, demonstrating the complementary nature of these approaches and their combined efficacy in addressing the challenges we faced. and > Our benchmark for quality was set by the standards achieved by the frontend developers based on our quality rubric that covers imports, rendering methods, JavaScript/TypeScript logic, and Jest assertions. We aimed to match their level of quality. The evaluation revealed that 80% of the content within these files was accurately converted, while the remaining 20% required manual intervention. (So I guess the "80% conversion success rate" is this percentage of files?) The Infoq title "Slack Combines ASTs with Large Language Models to Automatically Convert 80% of 15,000 Unit Tests" certainly more accurately reflects the underlying article than this comment. Edit: they do have a diagram that talks about 22% of the subset of manually inspected files being 100% complete. This doesn't appear to be what Slack considers their success rate because they manually inspect files anyway. [1] https://slack.engineering/balancing-old-tricks-with-new-feats-ai-powered-conversion-from-enzyme-to-react-testing-library-at-slack/ https://slack.engineering/balancing-old-tricks-with-new-feat...
- moefh 2y ago
  > No reasonable reading of that makes it a "22% success rate". Well, 500/2300 is 22%, so calling it 22% seems pretty reasonable. From what I get from the rest, the 78% remaining tests (the ones that failed to convert) were "80% accurately converted", I guess they had some metric for measuring that. So it looks like it depends on how you interpret "automatically converted 80%". If it's taken to mean "80% could be used without manual intervention", then it's clearly false. If you take it to mean "it required manual intervention on just 20% of the contents to be usable", then it's reasonable.
dang 2y ago
We changed the URL to that from https://www.infoq.com/news/2024/06/slack-automatic-test-conversion/ https://www.infoq.com/news/2024/06/slack-automatic-test-conv.... Thanks!

doubloon 2y ago

Reminds me of Hitchhikers Guide … they had to figure out the right question to ask.

dvh 2y ago
Is it true that they give out free BMWs in Moscow? Yes, it is true! But it's not Moscow but St. Petersburg. And it's not BMWs but Ladas. And they don't give them out, they steal them.
- Joker_vD 2y ago
  So all in all, still a mostly accurate news reporting, as they go.
- dloranc 2y ago
  This is a variation of famous jokes about Radio Yerevan, very popular in former soviet states. I live in Poland and I know this version: Is it true that they give away cars on Red Square? Radio Yerevan answers: not cars, only bicycles, not on Red Square, but near the Warsaw station, and they don't give them away, they steal them.

zer00eyz 2y ago

The slack blog is for engineers. It's PR to hire in talent. The INFOQ article is for your C types. It's the spin on what buzz words they should be using with their peers and to make a splash in the market. NFT, Crypto, Cloud, Microservice, SAS, Podcast (the first time when there wasnt video), Web 2.0, The Ad (double-click, pre google) market, The Dot Com Bubble... Im sure I missed a few hype cycles in there. Both articles show how deep we are in the stink of this one. Im tired of swimming the bullshit, It keeps getting in my mouth.

ben_jones 2y ago
Thus the conclusion should be heavily scrutinize future infoq .com articles and perhaps future articles by the same author Eran Stiller. We shouldn’t detach responsibility from the publisher and author.
- nl 2y ago
  Except the article turns out to be accurate (see https://news.ycombinator.com/item?id=40728179 https://news.ycombinator.com/item?id=40728179) So I guess the publisher and author should get credit. I'll leave others to discuss the misleading comment...
  - phillipcarter 2y ago
    Hah. A lot of tech folks like to trash journalists (it's fine, I get it, there's legitimate reasons) ... but then misread source content that the journalist interpreted better/correctly.

deleted 2y ago

[deleted]

nl 2y ago

Actually the Infoq article is more correct than this comment! The comment: ""Slack uses ASTs to convert test code from Enzyme to React with 22% success rate"" To quote[1], this 22% comes from this part: > We examined the conversion rates of approximately 2,300 individual test cases spread out within 338 files. Among these, approximately 500 test cases were successfully converted, executed, and passed. This highlights how effective AI can be, leading to a significant saving of 22% of developer time. It’s important to note that this 22% time saving represents only the documented cases where the test case passed. So that 22% rate is 22% saving of developer time, measured on a sample. No reasonable reading of that makes it a "22% success rate". Over the whole set of tests: > This strategic pivot, and the integration of both AST and AI technologies, helped us achieve the remarkable 80% conversion success rate, based on selected files, demonstrating the complementary nature of these approaches and their combined efficacy in addressing the challenges we faced. and > Our benchmark for quality was set by the standards achieved by the frontend developers based on our quality rubric that covers imports, rendering methods, JavaScript/TypeScript logic, and Jest assertions. We aimed to match their level of quality. The evaluation revealed that 80% of the content within these files was accurately converted, while the remaining 20% required manual intervention. (So I guess the "80% conversion success rate" is this percentage of files?) The Infoq title "Slack Combines ASTs with Large Language Models to Automatically Convert 80% of 15,000 Unit Tests" certainly more accurately reflects the underlying article than this comment. Edit: they do have a diagram that talks about 22% of the subset of manually inspected files being 100% complete. This doesn't appear to be what Slack considers their success rate because they manually inspect files anyway. [1] https://slack.engineering/balancing-old-tricks-with-new-feats-ai-powered-conversion-from-enzyme-to-react-testing-library-at-slack/ https://slack.engineering/balancing-old-tricks-with-new-feat...

moefh 2y ago
> No reasonable reading of that makes it a "22% success rate". Well, 500/2300 is 22%, so calling it 22% seems pretty reasonable. From what I get from the rest, the 78% remaining tests (the ones that failed to convert) were "80% accurately converted", I guess they had some metric for measuring that. So it looks like it depends on how you interpret "automatically converted 80%". If it's taken to mean "80% could be used without manual intervention", then it's clearly false. If you take it to mean "it required manual intervention on just 20% of the contents to be usable", then it's reasonable.

dang 2y ago

We changed the URL to that from https://www.infoq.com/news/2024/06/slack-automatic-test-conversion/ https://www.infoq.com/news/2024/06/slack-automatic-test-conv.... Thanks!