3 ms·

I strongly disagree. If you keep things tidy coming in then you have a known good base.

by egwor 6y ago

mehrdadn 6y ago
Don't you always have a known good base? It's just the latest build that worked correctly, right?
- tom_mellior 6y ago
  > It's just the latest build that worked correctly, right? But how would you tell which one that is? I guess even within the current system one could set up a bot that monitors master, and whenever the latest commit builds and passes tests, it updates a "known working" branch. But this doesn't seem to be the case. And if such a system is set up, it would make a lot more sense for contributors to push to a "testing" branch instead of master and then only updating master for working builds, rather than the other way round.
  - jstimpfle 6y ago
    How do you even tell whether something is right? That's not always a simple question to answer. Sure, one way is to have an automatic test suite and an integration system that moves commits that passed it from one branch to another. If you can actually make a test suite that has no false negatives and finds enough negatives to be useful, that's great. In my (limited) experience test suites are not close to that ideal. I've worked with systems where there was an automatic build test at least, and sometimes that catches a bad commit. But I think for many fast-moving projects there are few useful errors these systems can find, and sometimes they have annoying false negatives. The thing that I can think of that would most improve my productivity right now is a check that nobody added tab characters in the source code. (I realize my comment might not apply to a situation like LLVM, where there could be a large test suite of programs to compile and run for example).
    - tom_mellior 6y ago
      I think your comments apply more to other situations. Compilers are very testable: The input is usually human-readable text that can be stored in a file (source code), the final output is usually human-readable text that can be stored in a file (assembly code), intermediate states can usually be dumped to human-readable text that can be stored in a file (in the case of LLVM, bitcode). There is no nondeterminism, no network, no database state, no particular hardware or system requirements for most tests, etc., to consider during testing. Compilers are also often very modular. LLVM in particular makes it very easy to take a bitcode file, apply a certain well-defined set of transformations, dump the resulting bitcode file, and check whether it fulfills the expected properties, e.g., "this computation has been optimized to this other computation". LLVM has a tool called lit that does this kind of test, and there is an extensive test suite that uses it. Of course none of this guarantees that other programs will behave differently, but in my experience these test suites really are very good and useful. The only missing piece in LLVM is consistent enforcement of a policy that tests must pass in order for the repository to reach certain well-defined states.

I strongly disagree. If you keep things tidy coming in then you have a known good base.

by egwor 6y ago

I strongly disagree. If you keep things tidy coming in then you have a known good base.

mehrdadn 6y ago

Don't you always have a known good base? It's just the latest build that worked correctly, right?

tom_mellior 6y ago
> It's just the latest build that worked correctly, right? But how would you tell which one that is? I guess even within the current system one could set up a bot that monitors master, and whenever the latest commit builds and passes tests, it updates a "known working" branch. But this doesn't seem to be the case. And if such a system is set up, it would make a lot more sense for contributors to push to a "testing" branch instead of master and then only updating master for working builds, rather than the other way round.
- jstimpfle 6y ago
  How do you even tell whether something is right? That's not always a simple question to answer. Sure, one way is to have an automatic test suite and an integration system that moves commits that passed it from one branch to another. If you can actually make a test suite that has no false negatives and finds enough negatives to be useful, that's great. In my (limited) experience test suites are not close to that ideal. I've worked with systems where there was an automatic build test at least, and sometimes that catches a bad commit. But I think for many fast-moving projects there are few useful errors these systems can find, and sometimes they have annoying false negatives. The thing that I can think of that would most improve my productivity right now is a check that nobody added tab characters in the source code. (I realize my comment might not apply to a situation like LLVM, where there could be a large test suite of programs to compile and run for example).
  - tom_mellior 6y ago
    I think your comments apply more to other situations. Compilers are very testable: The input is usually human-readable text that can be stored in a file (source code), the final output is usually human-readable text that can be stored in a file (assembly code), intermediate states can usually be dumped to human-readable text that can be stored in a file (in the case of LLVM, bitcode). There is no nondeterminism, no network, no database state, no particular hardware or system requirements for most tests, etc., to consider during testing. Compilers are also often very modular. LLVM in particular makes it very easy to take a bitcode file, apply a certain well-defined set of transformations, dump the resulting bitcode file, and check whether it fulfills the expected properties, e.g., "this computation has been optimized to this other computation". LLVM has a tool called lit that does this kind of test, and there is an extensive test suite that uses it. Of course none of this guarantees that other programs will behave differently, but in my experience these test suites really are very good and useful. The only missing piece in LLVM is consistent enforcement of a policy that tests must pass in order for the repository to reach certain well-defined states.