4 ms·

While I tend to agree that modeling and approximation error are usually greater than floating point error, floating point error can be more devastating in pract

While I tend to agree that modeling and approximation error are usually greater than floating point error, floating point error can be more devastating in practice because once you realize you have a floating point problem, it's much more difficult to work around. When we develop models or approximation schemes, we're usually cognizant of the fact that we're fudging some things - we accept the limitations of the model, or we have an epsilon parameter or five in the approximation scheme that we can tune if we need better accuracy. Since we need to build all the machinery in code to solve the problem at all, it's pretty easy to tune it when we find that machinery lacking. But we usually have little choice other than to hope that floating point math of some sort will be "good enough", since we're reliant on the hardware for performance there. If doubles don't cut it, it can be a pretty substantial project to retrofit a mathy algorithm so that it has higher numerical precision, and it can be all but impossible if we're already compute-heavy and don't have many more cycles to waste.

mjb 15y ago
> While I tend to agree that modeling and approximation error are usually greater than floating point error, floating point error can be more devastating in practice because once you realize you have a floating point problem, it's much more difficult to work around. There are many well-understood approaches to reducing many sources of floating point error that don't require throwing more bits at the problem. > When we develop models or approximation schemes, we're usually cognizant of the fact that we're fudging some things I think that may be true in some cases, but false in general. To pick one model as an example, it takes a bit of effort to really understand the point where Newtonian mechanics is no longer a valid model of a physical system. In many cases, this is hand-waved away, with no attempt to quantify the contributions of the known model error (Newtonian physics is a flawed model of reality) against the contributions of the calculation error (floating point is a flawed model of reality). > it's much more difficult to work around. Floating point error may be more difficult to work around, but in most interesting cases model error is more difficult to detect. This is especially true if we have an incomplete understanding of reality.
- bermanoid 15y ago
  Agreed on all these points; you're right, I've seriously played down the difficulty of addressing model error, which in reality usually overwhelm the other two sources of error anyways. I understand this all too well - I once blew more than six months (and also burnt myself out, ending up in a long period of career-related depression, but that's another story) on a fixed-bid contract that I had quoted based on a three week estimate. Why? Because the last .5% of error to reach the acceptance threshold ended up requiring a model that was several times the complexity (over 5x as much code, seriously mathy shit, too, and I'm saying that as someone with a background in mathematical finance...) of the one that got me to that plateau, which I had assumed would be enough to do the job. The "lesson learned" there was that it's seriously foolish to submit fixed bids for modeling problems, unless you've previously fully solved the exact same problem (within epsilon) to the required accuracy. I don't care if you have a reasonable expectation that you can hit the target based on domain knowledge, if you haven't seen the data before, you have no freaking clue what you're dealing with - you might need to blow two weeks sanitizing the data before you can even think about modeling (I did), and you might discover that the "3 million" data points that the client promised are mostly irrelevant to the behavior that you're being asked to model, so you spend the next week trying to track down other data sources, and more time negotiating those transactions. Given all that, you're likely to end up doing long term open-ended research, which is fine if someone wants to pay you for your time and is willing to accept a reasonable chance of failure, but those sorts of tasks are most definitely not typical development projects: my rule of thumb these days is, if I would be able to publish an academic paper in a legit peer reviewed journal if I did the project by myself instead of for a company, then it's not a deliverable suitable for a fixed bid. Another word-to-the-wise: you're doubly stupid if you agree, as I did, to both an accuracy target and a model performance one, triply so if you also neglect to specify in the agreement what hardware they'll be evaluating performance with respect to... Better yet, just don't do this type of stuff as a consultant unless and until you really know the game, it's way too easy to go wrong and end up screwing over both yourself and your clients.

mjb 15y ago

> While I tend to agree that modeling and approximation error are usually greater than floating point error, floating point error can be more devastating in practice because once you realize you have a floating point problem, it's much more difficult to work around. There are many well-understood approaches to reducing many sources of floating point error that don't require throwing more bits at the problem. > When we develop models or approximation schemes, we're usually cognizant of the fact that we're fudging some things I think that may be true in some cases, but false in general. To pick one model as an example, it takes a bit of effort to really understand the point where Newtonian mechanics is no longer a valid model of a physical system. In many cases, this is hand-waved away, with no attempt to quantify the contributions of the known model error (Newtonian physics is a flawed model of reality) against the contributions of the calculation error (floating point is a flawed model of reality). > it's much more difficult to work around. Floating point error may be more difficult to work around, but in most interesting cases model error is more difficult to detect. This is especially true if we have an incomplete understanding of reality.

bermanoid 15y ago
Agreed on all these points; you're right, I've seriously played down the difficulty of addressing model error, which in reality usually overwhelm the other two sources of error anyways. I understand this all too well - I once blew more than six months (and also burnt myself out, ending up in a long period of career-related depression, but that's another story) on a fixed-bid contract that I had quoted based on a three week estimate. Why? Because the last .5% of error to reach the acceptance threshold ended up requiring a model that was several times the complexity (over 5x as much code, seriously mathy shit, too, and I'm saying that as someone with a background in mathematical finance...) of the one that got me to that plateau, which I had assumed would be enough to do the job. The "lesson learned" there was that it's seriously foolish to submit fixed bids for modeling problems, unless you've previously fully solved the exact same problem (within epsilon) to the required accuracy. I don't care if you have a reasonable expectation that you can hit the target based on domain knowledge, if you haven't seen the data before, you have no freaking clue what you're dealing with - you might need to blow two weeks sanitizing the data before you can even think about modeling (I did), and you might discover that the "3 million" data points that the client promised are mostly irrelevant to the behavior that you're being asked to model, so you spend the next week trying to track down other data sources, and more time negotiating those transactions. Given all that, you're likely to end up doing long term open-ended research, which is fine if someone wants to pay you for your time and is willing to accept a reasonable chance of failure, but those sorts of tasks are most definitely not typical development projects: my rule of thumb these days is, if I would be able to publish an academic paper in a legit peer reviewed journal if I did the project by myself instead of for a company, then it's not a deliverable suitable for a fixed bid. Another word-to-the-wise: you're doubly stupid if you agree, as I did, to both an accuracy target and a model performance one, triply so if you also neglect to specify in the agreement what hardware they'll be evaluating performance with respect to... Better yet, just don't do this type of stuff as a consultant unless and until you really know the game, it's way too easy to go wrong and end up screwing over both yourself and your clients.