5 ms·

First, users don't use string.indexOf(string). Users use software that uses string.indexOf(string). Second, I would write a test that takes generates a variety

by mempko 2y ago

First, users don't use string.indexOf(string). Users use software that uses string.indexOf(string). Second, I would write a test that takes generates a variety of inputs to cover the domain of string.indexOf(string) and have it call the old version and the new version. I would then collect stats around the performance of each call and make sure the new implementation is faster by the threshold I meant for it to be. Since the PURPOSE OF THE REFACTOR WAS PERFORMANCE. Then I would delete the test after I'm satisfied it worked well on the target hardware, etc. Let's look at the chrome unit tests. https://chromium.googlesource.com/v8/v8/+/3.0.12.1/test/mjsunit/string-indexof-2.js https://chromium.googlesource.com/v8/v8/+/3.0.12.1/test/mjsu... This test checks one case and then does a loop over to make sure various sizes work. This is a POOR test because it doesn't cover the domain of the function and exceptional cases. This test is pointless to have and run every build. Why keep it, it does nothing useful. My point is that if you are refactoring a function, you are going to have to write a NEW TEST anyway because the purpose of a refactor is different each time. The old tests are going to be useless. This is why contracts are so useful because they will ALERT YOU when the software does something you didn't assume. Unit tests don't do this. From an information theory perspective, the old tests would not provide you with new information. If you want to make sure your refactor works like the old function, then compare the outputs of your new function with the old one over the domain of the function. Then if it's good, delete the tests because why keep them. Experience shows that users will do things to your program you did NOT expect and therefore your tests will not cover. And more likely than not, refactoring requires rewriting tests anyway so the old tests are usually always useless. Now SYSTEM tests ARE useful. Tests that wire up many components and test them are useful. But a UNIT test, which tests one function is often just a waste of time to keep.

dataflow 2y ago
Honestly... I can see it's going to be genuinely exhausting for me to rebut point-by-point here, so I'll just let others read this and continue if they're interested. I really appreciate the response though, at least it explains your thought process well. I have one question for you that I feel I have to ask: have you actually practiced commercial software development on a team (say, 5+ developers on the same codebase) in this manner that you describe for a significant period of time (say, 2+ years)? and if so, do you feel the resulting software has been robust and successful?
- mempko 2y ago
  I've been doing this style of software development for 16 years (out of 28 years programming) across various projects. If you want a small project that I wrote to look at, see henhouse https://github.com/mempko/henhouse https://github.com/mempko/henhouse. I wrote an article talking about design by contract and why it's better than TDD here https://mempko.wordpress.com/2016/08/23/the-fox-in-the-henhouse/ https://mempko.wordpress.com/2016/08/23/the-fox-in-the-henho... I've built a computer vision system that processed petabytes of data with only one or two bugs in production for any given year. At any given time we kept a bug list of zero. For the last five years I built a trading system using this same process. Again, we don't keep a bug list because if there were bugs, the system wouldn't even run. And if there is a bug we have to fix it immediately. We do have tests, but they are system tests. The worst bug we had that took too long to catch was in a part of the system that DIDN'T use contracts. Design by Contract is a secret weapon people don't seem to know about. Also see https://github.com/mempko/firestr https://github.com/mempko/firestr, which I used Design by Contract extensively. It's a GUI program so the user can do crazy things. Learn Design by Contract. Do system testing. Do exploratory testing. Do fuzz testing. Keep a bug list of zero. Don't waste time on unit testing. If you are looking for popular software built using Design by Contract to various degree. See SQLite (uses assertions heavily), .NET framework from Microsoft, Many of the C++ Boost libraries, Parts of Rust, Parts of Qt. The Vala programming language, Ada GNAT... and many others. Here is a research paper from Microsoft that shows the advantage of contracts and code quality. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2006-54.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
  - dataflow 2y ago
    I feel there's something missing in what you describe, and I'm trying to pinpoint what it is... I can see it working if you have very few developers touching each piece of code, or if you get to exert control over the final application. But I don't see how it can work for large codebases or teams over long periods of time (read: large businesses)... especially not for library development, where your team is only responsible for providing library functionality (like string.indexOf(string) in my example, or matrix multiplication, or regexes, or whatever libraries do) and you don't necessarily even know whom the users are. There is no "system" or "integration" at that point, you're just developing one layer of code there, which is the library -- a piece of code responsible for doing just one thing. How the heck do you make sure arbitrary team members touching code don't end up introducing silly bugs over time, if not with unit tests? Have you built any commercial libraries in this manner, rather than applications? i.e. where everyone on your team is jointly responsible for your library's development (like the implementation of string.indexOf(string) in my example), and other folks (whether inside or outside the company) are the ones who piece together the libraries to create their final application(s)?

by mempko 2y ago

dataflow 2y ago

Honestly... I can see it's going to be genuinely exhausting for me to rebut point-by-point here, so I'll just let others read this and continue if they're interested. I really appreciate the response though, at least it explains your thought process well. I have one question for you that I feel I have to ask: have you actually practiced commercial software development on a team (say, 5+ developers on the same codebase) in this manner that you describe for a significant period of time (say, 2+ years)? and if so, do you feel the resulting software has been robust and successful?

mempko 2y ago
I've been doing this style of software development for 16 years (out of 28 years programming) across various projects. If you want a small project that I wrote to look at, see henhouse https://github.com/mempko/henhouse https://github.com/mempko/henhouse. I wrote an article talking about design by contract and why it's better than TDD here https://mempko.wordpress.com/2016/08/23/the-fox-in-the-henhouse/ https://mempko.wordpress.com/2016/08/23/the-fox-in-the-henho... I've built a computer vision system that processed petabytes of data with only one or two bugs in production for any given year. At any given time we kept a bug list of zero. For the last five years I built a trading system using this same process. Again, we don't keep a bug list because if there were bugs, the system wouldn't even run. And if there is a bug we have to fix it immediately. We do have tests, but they are system tests. The worst bug we had that took too long to catch was in a part of the system that DIDN'T use contracts. Design by Contract is a secret weapon people don't seem to know about. Also see https://github.com/mempko/firestr https://github.com/mempko/firestr, which I used Design by Contract extensively. It's a GUI program so the user can do crazy things. Learn Design by Contract. Do system testing. Do exploratory testing. Do fuzz testing. Keep a bug list of zero. Don't waste time on unit testing. If you are looking for popular software built using Design by Contract to various degree. See SQLite (uses assertions heavily), .NET framework from Microsoft, Many of the C++ Boost libraries, Parts of Rust, Parts of Qt. The Vala programming language, Ada GNAT... and many others. Here is a research paper from Microsoft that shows the advantage of contracts and code quality. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2006-54.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
- dataflow 2y ago
  I feel there's something missing in what you describe, and I'm trying to pinpoint what it is... I can see it working if you have very few developers touching each piece of code, or if you get to exert control over the final application. But I don't see how it can work for large codebases or teams over long periods of time (read: large businesses)... especially not for library development, where your team is only responsible for providing library functionality (like string.indexOf(string) in my example, or matrix multiplication, or regexes, or whatever libraries do) and you don't necessarily even know whom the users are. There is no "system" or "integration" at that point, you're just developing one layer of code there, which is the library -- a piece of code responsible for doing just one thing. How the heck do you make sure arbitrary team members touching code don't end up introducing silly bugs over time, if not with unit tests? Have you built any commercial libraries in this manner, rather than applications? i.e. where everyone on your team is jointly responsible for your library's development (like the implementation of string.indexOf(string) in my example), and other folks (whether inside or outside the company) are the ones who piece together the libraries to create their final application(s)?