6 ms·

Diagnosing a Linux-only unit test failure

70 points by curtis 9y ago

deleted 9y ago
[deleted]
MPSimmons 9y ago
Read this, thought it was well written and thought out, look up at the headline... "Oh man, Jon Skeet!" Fun problem.
magnat 9y ago
Not directly related to the article, but string interpolation in code samples in step 6 looks somewhat unreadable without proper syntax coloring: Console.WriteLine($"lhs=({lhs.x}, {lhs.y}); rhs=({rhs.x}, {rhs.y})"); Console.WriteLine($"{lhs.x == rhs.x}, {lhs.y == rhs.y}");
- masklinn 9y ago
  They look fine to me, but I guess it helps that both Python and Rust use the same interpolation style.
  - gpm 9y ago
    Err... do either python or rust allow this style, in both languages I've always used ("lhs=({}, {}); rhs=({}, {})", lhs.x, lhs.y, rhs.x, rhs.y)
    - cyphar 9y ago
      Python added "f strings" recently (read: in 2015)[1], which look like f"{some_var}" which evaluates "some_var" in the local scope. Sort of like Ruby's "#{...}" (if I'm remembering that syntax correctly). Rust supports named parameters in format strings[2] but that's not quite the same. [1]: https://www.python.org/dev/peps/pep-0498/ https://www.python.org/dev/peps/pep-0498/ [2]: https://doc.rust-lang.org/std/fmt/#named-parameters https://doc.rust-lang.org/std/fmt/#named-parameters
      - vog 9y ago
        Note that you can do this in old version of Python, too, using the "format-locals" idiom: some_var = ... "{some_var}".format(**locals()) Although this does have some limitations regarding variables to strictly being in the local scope, this is a fine idiom if you need to be compatible with older Python versions.
        cyphar 9y ago
        That is a bit of a hack though ;). You could also do it with %-style formatting as well: some_var = ... "%(some_var)s" % locals() If you want to do both locals and globals you can do: some_var = ... "%(some_var)s" % dict(globals(), **locals()) New Python versions (>=3.5) have this syntax: "%(some_var)s" % {**globals(), **locals()} But those versions also have f-strings so that's sort of moot.
    - antsar 9y ago
      In addition to f-strings, Python supports this (slight) improvement on your version. "lhs=({l.x}, {l.y}); rhs=({r.x}, {r.y})".format(l=lhs, r=rhs)
    - masklinn 9y ago
      I was mostly talking about the placeholders (hence style), but > ("lhs=({}, {}); rhs=({}, {})", lhs.x, lhs.y, rhs.x, rhs.y) Python 3.6 added "natively interpolated" f-strings, just replace C#'s `$` prefix by `f`, and even without f-strings you could write e.g. "lhs=({lhs.x}, {lhs.y}); rhs=({rhs.x}, {rhx.y})".format(lhs=lhs, rhs=rhs) Rust supports named parameters but IIRC not attribute access or method calls.
      - steveklabnik 9y ago
        Yes, you can't do something like this in Rust, you'd be doing what's in the quote above; named params are it. You have to do the attribute access/method call when binding the parameter, not applying it to a parameter somehow.
lloeki 9y ago
> Tests that only fail in CI are really annoying. I’ve had this a couple of times, and it’s always taken hours to sort out, because the “try something, get results, think about what they mean” cycle is so slow. To address this, CircleCI has an interesting feature where you can enable SSH access to the build, so one can investigate locally (relative to the build) instead of repeatedly throwing code over the wall and waiting for something remotely meaningful to come back in logs.
- wwwigham 9y ago
  Travis actually has this[1], too, if you ask their support to have the feature beta enabled for your repo. A while back, I was having a CI only test failure that was a massive pain to try and debug, and this helped enormously as I could explore the build state and try alternatives without waiting on the typical build time. [1]https://docs.travis-ci.com/user/running-build-in-debug-mode/ https://docs.travis-ci.com/user/running-build-in-debug-mode/
alexfoo 9y ago
Comparing doubles with == ?
- half-kh-hacker 9y ago
  I'm not versed in C#, what should you use otherwise?
  - philh 9y ago
    It's not a C# thing, it's a floating-point thing. Two numbers which mathematically "should" be equal, might not be due to rounding errors. (In python, 0.15 + 0.15 != 0.1 + 0.2.) So it's risky to compare them with ==, sometimes you'll get the result you expect (and sometimes you can be confident of that) but sometimes you won't. If you can't be sure, you should compare that they're the same within some threshold, but that has pitfalls as well. (I feel a little silly saying "it's not a language thing" and then "in python", but there may (or may not) be various implementation details which other languages might use to get a different result.)
    - smitherfield 9y ago
      `0.1 + 0.2 != 0.3` is a property of all (base-2) floating-point arithmetic, no matter the precision or programming language.
      - estomagordo 9y ago
        Well yeah, but this was 0.1 + 0.2 != 0.15 + 0.15, and there's no obvious reason why 0.15 + 0.15 = 0.3 imo.
        philh 9y ago
        I interpreted their comment as "here's an example that will always work", and I appreciated it in that light.
- xioxox 9y ago
dunham 9y ago
I ran into a similar issue years ago. Our tests, written on macos, were failing on linux, because pdftotext was producing different output. I traced down the logic, added some debugging print statements, and it magically fixed itself. Eventually, I figured out the issue that they were comparing floats with == and intel processors have a few extra internal bits that disappear when stored. Macos, which uses floats in its UI layer, defaults to adding -ffloat-store to the compile flags to get around similar issues. I ended up just adding -ffloat-store to our Linux build of pdftotext. (I briefly thought about fixing the comparisons, but it turned out the code was full of them.)
without_words 9y ago
> My next step is definitely to try to find some history of this issue Well, you know exactly how to reproduce the issue, so git bisect it to find the exact commit that fixed it and how. Granted, it may take a very long time, given the size of .Net Core.

70 points by curtis 9y ago

deleted 9y ago

[deleted]

MPSimmons 9y ago

Read this, thought it was well written and thought out, look up at the headline... "Oh man, Jon Skeet!" Fun problem.

magnat 9y ago

Not directly related to the article, but string interpolation in code samples in step 6 looks somewhat unreadable without proper syntax coloring: Console.WriteLine($"lhs=({lhs.x}, {lhs.y}); rhs=({rhs.x}, {rhs.y})"); Console.WriteLine($"{lhs.x == rhs.x}, {lhs.y == rhs.y}");

masklinn 9y ago
They look fine to me, but I guess it helps that both Python and Rust use the same interpolation style.
- gpm 9y ago
  Err... do either python or rust allow this style, in both languages I've always used ("lhs=({}, {}); rhs=({}, {})", lhs.x, lhs.y, rhs.x, rhs.y)
  - cyphar 9y ago
    Python added "f strings" recently (read: in 2015)[1], which look like f"{some_var}" which evaluates "some_var" in the local scope. Sort of like Ruby's "#{...}" (if I'm remembering that syntax correctly). Rust supports named parameters in format strings[2] but that's not quite the same. [1]: https://www.python.org/dev/peps/pep-0498/ https://www.python.org/dev/peps/pep-0498/ [2]: https://doc.rust-lang.org/std/fmt/#named-parameters https://doc.rust-lang.org/std/fmt/#named-parameters
    - vog 9y ago
      Note that you can do this in old version of Python, too, using the "format-locals" idiom: some_var = ... "{some_var}".format(**locals()) Although this does have some limitations regarding variables to strictly being in the local scope, this is a fine idiom if you need to be compatible with older Python versions.
      - cyphar 9y ago
        That is a bit of a hack though ;). You could also do it with %-style formatting as well: some_var = ... "%(some_var)s" % locals() If you want to do both locals and globals you can do: some_var = ... "%(some_var)s" % dict(globals(), **locals()) New Python versions (>=3.5) have this syntax: "%(some_var)s" % {**globals(), **locals()} But those versions also have f-strings so that's sort of moot.
  - antsar 9y ago
    In addition to f-strings, Python supports this (slight) improvement on your version. "lhs=({l.x}, {l.y}); rhs=({r.x}, {r.y})".format(l=lhs, r=rhs)
  - masklinn 9y ago
    I was mostly talking about the placeholders (hence style), but > ("lhs=({}, {}); rhs=({}, {})", lhs.x, lhs.y, rhs.x, rhs.y) Python 3.6 added "natively interpolated" f-strings, just replace C#'s `$` prefix by `f`, and even without f-strings you could write e.g. "lhs=({lhs.x}, {lhs.y}); rhs=({rhs.x}, {rhx.y})".format(lhs=lhs, rhs=rhs) Rust supports named parameters but IIRC not attribute access or method calls.
    - steveklabnik 9y ago
      Yes, you can't do something like this in Rust, you'd be doing what's in the quote above; named params are it. You have to do the attribute access/method call when binding the parameter, not applying it to a parameter somehow.

lloeki 9y ago

> Tests that only fail in CI are really annoying. I’ve had this a couple of times, and it’s always taken hours to sort out, because the “try something, get results, think about what they mean” cycle is so slow. To address this, CircleCI has an interesting feature where you can enable SSH access to the build, so one can investigate locally (relative to the build) instead of repeatedly throwing code over the wall and waiting for something remotely meaningful to come back in logs.

wwwigham 9y ago
Travis actually has this[1], too, if you ask their support to have the feature beta enabled for your repo. A while back, I was having a CI only test failure that was a massive pain to try and debug, and this helped enormously as I could explore the build state and try alternatives without waiting on the typical build time. [1]https://docs.travis-ci.com/user/running-build-in-debug-mode/ https://docs.travis-ci.com/user/running-build-in-debug-mode/

alexfoo 9y ago

Comparing doubles with == ?

half-kh-hacker 9y ago
I'm not versed in C#, what should you use otherwise?
- philh 9y ago
  It's not a C# thing, it's a floating-point thing. Two numbers which mathematically "should" be equal, might not be due to rounding errors. (In python, 0.15 + 0.15 != 0.1 + 0.2.) So it's risky to compare them with ==, sometimes you'll get the result you expect (and sometimes you can be confident of that) but sometimes you won't. If you can't be sure, you should compare that they're the same within some threshold, but that has pitfalls as well. (I feel a little silly saying "it's not a language thing" and then "in python", but there may (or may not) be various implementation details which other languages might use to get a different result.)
  - smitherfield 9y ago
    `0.1 + 0.2 != 0.3` is a property of all (base-2) floating-point arithmetic, no matter the precision or programming language.
    - estomagordo 9y ago
      Well yeah, but this was 0.1 + 0.2 != 0.15 + 0.15, and there's no obvious reason why 0.15 + 0.15 = 0.3 imo.
      - philh 9y ago
        I interpreted their comment as "here's an example that will always work", and I appreciated it in that light.
xioxox 9y ago

dunham 9y ago

I ran into a similar issue years ago. Our tests, written on macos, were failing on linux, because pdftotext was producing different output. I traced down the logic, added some debugging print statements, and it magically fixed itself. Eventually, I figured out the issue that they were comparing floats with == and intel processors have a few extra internal bits that disappear when stored. Macos, which uses floats in its UI layer, defaults to adding -ffloat-store to the compile flags to get around similar issues. I ended up just adding -ffloat-store to our Linux build of pdftotext. (I briefly thought about fixing the comparisons, but it turned out the code was full of them.)

without_words 9y ago

> My next step is definitely to try to find some history of this issue Well, you know exactly how to reproduce the issue, so git bisect it to find the exact commit that fixed it and how. Granted, it may take a very long time, given the size of .Net Core.