3 ms·

One real-world example: All the textbook algorithms for triangulation by ear-clipping (i.e. turning a convex shape into a bunch of triangles) are broken if exec

One real-world example: All the textbook algorithms for triangulation by ear-clipping (i.e. turning a convex shape into a bunch of triangles) are broken if executed using IEEE floats, some formulations of them are worse than others. Typically they loop forever. If you take an implementation that appears to work on x86 and compile it for powerpc, it might break. In general you can be taking textbook accurate math based on rock solid proofs and then when you turn it into code it's busted, so nsan seems like it could be really valuable for flushing out some of those problems.

ggm 6y ago
I honestly thought the numerical algorithm group NAG library had fixed this three decades ago. I'm not being sarcastic, I thought the numerical analysis libraries tested the living bejesus out of their code. I guess a lot more intensive FP code exists now, outside of highly tested library code, or calls made from the calls, or whatever.
- yablak 6y ago
  You'd be surprised how many of the old, painstakingly tested fp algorithms, have been reimplemented in DL frameworks. I ported many algorithms from cephes to Eigen for use in TensorFlow. Some I was able to make simd. Then the XLA folks reimplemented some in HLO using other approximations that are easier to parallelize. Then consider that some linear algebra and simple functions like softmax and softplus have to be implemented for fast execution in float16 and bfloat16 supporting hardware. You can iterate over the entire fp16 space quite quickly but the initial version isn't usually so thoroughly tested.. performance is king anyway in NNs. Then a couple months later someone ports an old model to V100 and it turns out your softplus underflows. Guess how many weeks it takes to figure out why your model quality went to s**?
  - ggm 6y ago
    I sat next to the NAG maintenance crew at lunch sometimes, back in the 80s in Leeds university who held a contract (I think it moved around, possibly by design). They were fierce, collectivist correctness freaks. Outside code was reviewed mercilessly. Nothing anyone said about their experiences coding mattered as much as what they did to that library. If they stuffed it up, bridges fell down during construction or rockets blew up, and they knew it. I remain in awe. All those jokes about real programmers are wrong. Real programmers take great care.
- kevingadd 6y ago
  I'm sure those libraries are good, you just tend not to encounter stuff like that in programs written casually or by people who think performance really matters. Most people doing triangulation by ear clipping are trying to rasterize shapes. In the past, cairo (the 2d graphics library) had a bug that would cause crashes/hangs in Firefox because a VPN driver was changing the floating point state on random threads... which would cause one of those geometric algorithms to break.
  - gpderetta 6y ago
    That must have been extremely painful to debug. Let me guess, on Windows it was resetting the x87 fpu precision from 64 bits to 80?

ggm 6y ago

I honestly thought the numerical algorithm group NAG library had fixed this three decades ago. I'm not being sarcastic, I thought the numerical analysis libraries tested the living bejesus out of their code. I guess a lot more intensive FP code exists now, outside of highly tested library code, or calls made from the calls, or whatever.

yablak 6y ago
You'd be surprised how many of the old, painstakingly tested fp algorithms, have been reimplemented in DL frameworks. I ported many algorithms from cephes to Eigen for use in TensorFlow. Some I was able to make simd. Then the XLA folks reimplemented some in HLO using other approximations that are easier to parallelize. Then consider that some linear algebra and simple functions like softmax and softplus have to be implemented for fast execution in float16 and bfloat16 supporting hardware. You can iterate over the entire fp16 space quite quickly but the initial version isn't usually so thoroughly tested.. performance is king anyway in NNs. Then a couple months later someone ports an old model to V100 and it turns out your softplus underflows. Guess how many weeks it takes to figure out why your model quality went to s**?
- ggm 6y ago
  I sat next to the NAG maintenance crew at lunch sometimes, back in the 80s in Leeds university who held a contract (I think it moved around, possibly by design). They were fierce, collectivist correctness freaks. Outside code was reviewed mercilessly. Nothing anyone said about their experiences coding mattered as much as what they did to that library. If they stuffed it up, bridges fell down during construction or rockets blew up, and they knew it. I remain in awe. All those jokes about real programmers are wrong. Real programmers take great care.
kevingadd 6y ago
I'm sure those libraries are good, you just tend not to encounter stuff like that in programs written casually or by people who think performance really matters. Most people doing triangulation by ear clipping are trying to rasterize shapes. In the past, cairo (the 2d graphics library) had a bug that would cause crashes/hangs in Firefox because a VPN driver was changing the floating point state on random threads... which would cause one of those geometric algorithms to break.
- gpderetta 6y ago
  That must have been extremely painful to debug. Let me guess, on Windows it was resetting the x87 fpu precision from 64 bits to 80?