4 ms·

I actually wouldn't be surprised if the total number of tests run in the Julia ecosystem wasn't too different (thousands of packages with typically hundreds to

I actually wouldn't be surprised if the total number of tests run in the Julia ecosystem wasn't too different (thousands of packages with typically hundreds to thousands of unit tests, run on every commit and PR) -- virtually every Julia package has CI set up (at least standalone unit tests, though many packages could use more integration tests). Of course, in neither Matlab nor Julia do tests guarantee correctness.

Sporktacular 4y ago
Is that tests for the purpose of verifying correctness or tests of applications that will flag problems incidentally? I'm not too familiar, but like the idea of dedicating resources to that specifically. Guarantees aside, does MATLAB have an issue with this to the same extent as Julia?
- cbkeller 4y ago
  Personally I'd probably categorize most unit tests as verifying correctness (but only for the scenarios tested); integration tests may be more useful for finding incidental issues that you wouldn't have thought to test for directly. I'm for sure on board with dedicating more resources to testing -- and in my case as an academic, this is something I only have really been exposed to as a result of interacting with the Julia community. Matlab is pretty mature at this point, but I'm sure it's had its share of bugs over the years as well (especially if you also counted the file exchange, which is probably the closest thing they have to an open source package ecosystem); it would be interesting to compare the two at a similar level of maturity / development person-hours if quantitative data could be found.
adgjlsfhk1 4y ago
sample size of 1, but I've run 1 billion tests today in Julia (floating point power for Float16, Float32 and Float64)
- Sporktacular 4y ago
  For correctness? What was the result?
  - mbauman 4y ago
    Knowing adgjlsfhk1's work, yes, this would be for correctness — specifically measuring error in ULPs. Most frequently, adgjlsfhk1 pushes Julia's numeric routines to errors below 0.5 ULPs — that is, perfect correctly rounded behavior.
    - adgjlsfhk1 4y ago
      I actually am not a believer in perfect rounding. It tends to have a high performance cost, and IMO isn't that useful.
  - adgjlsfhk1 4y ago
    Up until a few days ago, we were testing that x^y was accurate to 1.3 ULPs for Float16, Float32, and Float64. However, for Float16 and Float32, we were actually accurate to (at least) .51 ULP, and Float64 was accurate to 1 ULP so I made the tests stricter there. There are 2 exceptions to this: x^3 and x^-2. because people from a math background often write code with literal powers, and expect it to be fast, for small integer powers (-2, -1, 0, 1, 2, and 3) that are constant, Julia will replace the call to pow with a call to (for example) xxx for x^3. As such, the accuracy bound for x^3 is 1.5 ULP and the bound for x^-2 is 2 ULP for all data types. This fixed a rare test failure on CI (since for ^3 and ^-2 the bounds were too tight the previous test would fail roughly 1/1000 runs), and will prevent regressions in accuracy if I ever come back to try to make the implementation faster.

Sporktacular 4y ago

Is that tests for the purpose of verifying correctness or tests of applications that will flag problems incidentally? I'm not too familiar, but like the idea of dedicating resources to that specifically. Guarantees aside, does MATLAB have an issue with this to the same extent as Julia?

cbkeller 4y ago
Personally I'd probably categorize most unit tests as verifying correctness (but only for the scenarios tested); integration tests may be more useful for finding incidental issues that you wouldn't have thought to test for directly. I'm for sure on board with dedicating more resources to testing -- and in my case as an academic, this is something I only have really been exposed to as a result of interacting with the Julia community. Matlab is pretty mature at this point, but I'm sure it's had its share of bugs over the years as well (especially if you also counted the file exchange, which is probably the closest thing they have to an open source package ecosystem); it would be interesting to compare the two at a similar level of maturity / development person-hours if quantitative data could be found.

adgjlsfhk1 4y ago

sample size of 1, but I've run 1 billion tests today in Julia (floating point power for Float16, Float32 and Float64)

Sporktacular 4y ago
For correctness? What was the result?
- mbauman 4y ago
  Knowing adgjlsfhk1's work, yes, this would be for correctness — specifically measuring error in ULPs. Most frequently, adgjlsfhk1 pushes Julia's numeric routines to errors below 0.5 ULPs — that is, perfect correctly rounded behavior.
  - adgjlsfhk1 4y ago
    I actually am not a believer in perfect rounding. It tends to have a high performance cost, and IMO isn't that useful.
- adgjlsfhk1 4y ago
  Up until a few days ago, we were testing that x^y was accurate to 1.3 ULPs for Float16, Float32, and Float64. However, for Float16 and Float32, we were actually accurate to (at least) .51 ULP, and Float64 was accurate to 1 ULP so I made the tests stricter there. There are 2 exceptions to this: x^3 and x^-2. because people from a math background often write code with literal powers, and expect it to be fast, for small integer powers (-2, -1, 0, 1, 2, and 3) that are constant, Julia will replace the call to pow with a call to (for example) xxx for x^3. As such, the accuracy bound for x^3 is 1.5 ULP and the bound for x^-2 is 2 ULP for all data types. This fixed a rare test failure on CI (since for ^3 and ^-2 the bounds were too tight the previous test would fail roughly 1/1000 runs), and will prevent regressions in accuracy if I ever come back to try to make the implementation faster.