3 ms·

An attempt to measure say 5% difference in a tests would likely produce a flaky test which frequent failures would waste team's time. I'm not even starting that

An attempt to measure say 5% difference in a tests would likely produce a flaky test which frequent failures would waste team's time. I'm not even starting that to measure this difference in a test one would have to maintain two implementations - slow and faster one and run in a test suite both to have two times to compare. Looks like unnecessary work to me. Enough to run benchmark manually when a decision is being made.

still_grokking 4y ago
Exactly. I wonder people don't recognize that upfront. I'm getting the impression, like it was stated by some people here, that this whole "no comments" / "everything should be a unit test" idea comes form inexperienced but idealistic developers mostly. People with more experience seem to know already OTOH that good comments can save your ass in the middle of the night.
- randomdata 4y ago
  > this whole "no comments" / "everything should be a unit test" idea Nothing in this thread says "no comments" or "everything should be a unit test". But the reality is that most code doesn't matter. It is just a means to an end and if it no longer serves that means it's expendable. Explaining why something was chosen doesn't tell you anything meaningful most of the time. In fact, I expect the "why" 99% of the time is "I felt like it and it worked". When a specific approach is essential in meeting certain application requirements, absolutely you need to comment that. But you also need tests, else how are you going to ensure that you stay within those requirements? Of an application of any meaningful size, you're going to have a hard time manually ensuring that some externality didn't break your assumption on every deploy. Even if it weren't hard, why put in the man hours when a machine can do it for you?
  - still_grokking 4y ago
    > In fact, I expect the "why" 99% of the time is "I felt like it and it worked". Erm, we're talking about professional software development here. What someone is tinkering together in his hobby cellar is of no interest to me. Code created by the principle "I felt like that" is garbage by definition. It makes no sense to even discuss such stuff. > When a specific approach is essential in meeting certain application requirements, absolutely you need to comment that. But you also need tests, else how are you going to ensure that you stay within those requirements? Of an application of any meaningful size, you're going to have a hard time manually ensuring that some externality didn't break your assumption on every deploy. Even if it weren't hard, why put in the man hours when a machine can do it for you? Here I actually agree fully. What can be tested by the machine should be. That's for example a reason to use statically languages, as in contrast to such thing as Ruby, the machine can give much better correctness guaranties with static type checking. But the point is: Not everything can be expressed as meaningful test. For example you proposed something that would result in a flaky test case… Some other things can't be expressed as test either. And the experience shows that's this things are almost always related to the why question. In the end you put exactly this question front while discussing some imaginary implementation of something, which shows my point nicely. :-)
    - wpietri 4y ago
      > Code created by the principle "I felt like that" is garbage by definition. It makes no sense to even discuss such stuff. I think you are vastly overestimating not only how much people do work rationally, but how much it's even possible. E.g., System 2 thinking is just much more expensive. And programmers usually work in environments that are pretty data-poor with respect to important factors. And that's before we even get to both project-level and individual path-dependence.
- wpietri 4y ago
  One, I didn't say that. And two, I write under my real name. It's pretty easy to find out how much experience I have. As a person with "more" experience, I went through a phase of writing lots of comments. Obviously, given how much I comment here, I like writing. But as I said elsewhere in this discussion, comments have become for me a last resort. I'd rather put the information I'm trying to convey almost anywhere else. Variable names, method names, improved interfaces, better object relationships, doc strings, test code, test names, commit comments, or my colleagues' heads.
randomdata 4y ago
If you can't consistently measure a 5% difference in a tightly controlled environment, how are your real world users going to notice? If your users don't notice, who cares? Furthermore, if that 5% optimization is truly critical to the application (despite not being measurable?), how are you going to ensure that some future change doesn't break that? Something as simple as a compiler update could change some optimization that breaks your assumptions. What do you do in the absence of tests to ensure that doesn't slip through?
- citrin_ru 4y ago
  Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about (but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests). > 5% difference in a tightly controlled environment, A test suite usually doesn't run in tightly controlled environment. To get meaningful performance testing results you'll need to run it on dedicated hardware (no noisy neighbors) and don't run different tests in parallel (so one test will not be a noisy neighbor for another test). It is something not hard to do once in a while manually but would be quite expensive to do as part of a test suite which typically is running on each change.
  - randomdata 4y ago
    > Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about Users won't notice, but they will notice...? > A test suite usually doesn't run in tightly controlled environment. What don't you have control over? > but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests Why would you avoid comments? Why would your tests have to take longer to write than your comments? Days to write a test or two to clarify your intent?
wpietri 4y ago
If the difference is so small it's hard to measure, then that sounds like something that's not worth testing to me. Meaning that it's also ok if somebody changes the implementation later on. On the other hand, if it's performance critical code in a way that's important to the project, then running a manual benchmark once and hoping that nobody accidentally breaks it strikes me as poor practice.

still_grokking 4y ago

Exactly. I wonder people don't recognize that upfront. I'm getting the impression, like it was stated by some people here, that this whole "no comments" / "everything should be a unit test" idea comes form inexperienced but idealistic developers mostly. People with more experience seem to know already OTOH that good comments can save your ass in the middle of the night.

randomdata 4y ago
> this whole "no comments" / "everything should be a unit test" idea Nothing in this thread says "no comments" or "everything should be a unit test". But the reality is that most code doesn't matter. It is just a means to an end and if it no longer serves that means it's expendable. Explaining why something was chosen doesn't tell you anything meaningful most of the time. In fact, I expect the "why" 99% of the time is "I felt like it and it worked". When a specific approach is essential in meeting certain application requirements, absolutely you need to comment that. But you also need tests, else how are you going to ensure that you stay within those requirements? Of an application of any meaningful size, you're going to have a hard time manually ensuring that some externality didn't break your assumption on every deploy. Even if it weren't hard, why put in the man hours when a machine can do it for you?
- still_grokking 4y ago
  > In fact, I expect the "why" 99% of the time is "I felt like it and it worked". Erm, we're talking about professional software development here. What someone is tinkering together in his hobby cellar is of no interest to me. Code created by the principle "I felt like that" is garbage by definition. It makes no sense to even discuss such stuff. > When a specific approach is essential in meeting certain application requirements, absolutely you need to comment that. But you also need tests, else how are you going to ensure that you stay within those requirements? Of an application of any meaningful size, you're going to have a hard time manually ensuring that some externality didn't break your assumption on every deploy. Even if it weren't hard, why put in the man hours when a machine can do it for you? Here I actually agree fully. What can be tested by the machine should be. That's for example a reason to use statically languages, as in contrast to such thing as Ruby, the machine can give much better correctness guaranties with static type checking. But the point is: Not everything can be expressed as meaningful test. For example you proposed something that would result in a flaky test case… Some other things can't be expressed as test either. And the experience shows that's this things are almost always related to the why question. In the end you put exactly this question front while discussing some imaginary implementation of something, which shows my point nicely. :-)
  - wpietri 4y ago
    > Code created by the principle "I felt like that" is garbage by definition. It makes no sense to even discuss such stuff. I think you are vastly overestimating not only how much people do work rationally, but how much it's even possible. E.g., System 2 thinking is just much more expensive. And programmers usually work in environments that are pretty data-poor with respect to important factors. And that's before we even get to both project-level and individual path-dependence.
wpietri 4y ago
One, I didn't say that. And two, I write under my real name. It's pretty easy to find out how much experience I have. As a person with "more" experience, I went through a phase of writing lots of comments. Obviously, given how much I comment here, I like writing. But as I said elsewhere in this discussion, comments have become for me a last resort. I'd rather put the information I'm trying to convey almost anywhere else. Variable names, method names, improved interfaces, better object relationships, doc strings, test code, test names, commit comments, or my colleagues' heads.

randomdata 4y ago

If you can't consistently measure a 5% difference in a tightly controlled environment, how are your real world users going to notice? If your users don't notice, who cares? Furthermore, if that 5% optimization is truly critical to the application (despite not being measurable?), how are you going to ensure that some future change doesn't break that? Something as simple as a compiler update could change some optimization that breaks your assumptions. What do you do in the absence of tests to ensure that doesn't slip through?

citrin_ru 4y ago
Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about (but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests). > 5% difference in a tightly controlled environment, A test suite usually doesn't run in tightly controlled environment. To get meaningful performance testing results you'll need to run it on dedicated hardware (no noisy neighbors) and don't run different tests in parallel (so one test will not be a noisy neighbor for another test). It is something not hard to do once in a while manually but would be quite expensive to do as part of a test suite which typically is running on each change.
- randomdata 4y ago
  > Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about Users won't notice, but they will notice...? > A test suite usually doesn't run in tightly controlled environment. What don't you have control over? > but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests Why would you avoid comments? Why would your tests have to take longer to write than your comments? Days to write a test or two to clarify your intent?

wpietri 4y ago

If the difference is so small it's hard to measure, then that sounds like something that's not worth testing to me. Meaning that it's also ok if somebody changes the implementation later on. On the other hand, if it's performance critical code in a way that's important to the project, then running a manual benchmark once and hoping that nobody accidentally breaks it strikes me as poor practice.