4 ms·
>I don't think you can ensure that with gradients though: if f and f' are computed in machine arithmetic, cancellation errors might pile up. Yes, you can. You
by constantcrying 1y ago
>I don't think you can ensure that with gradients though: if f and f' are computed in machine arithmetic, cancellation errors might pile up.
Yes, you can. You need to do automatic differentiation with interval arithmetic. This gives you a mathematically correct result for the derivative.
Always keep in mind with interval arithmetic that (-inf, inf) is also a mathematically correct result. Keeping intervals small is the challenge with interval arithmetic and means that every algorithm not specifically developed for it is likely going to lead to useless results.
- helltone 1y agoI think perhaps this could be done in other ways that don't require interval arithmetic for autodiff, only that the gradient is conservatively computed, in other words carrying the numerical error from f into f'
- deleted 1y ago[deleted]
- fph 1y agoBut the only way we know how to make "conservative computations" is via interval arithmetic.
- helltone 1y agoNo? Or maybe I'm missing something. If the goal is to be able to bound the computation of f, you can: 1) compute f with interval arithmetic 2) compute f normally and f' with interval arithmetic 3) compute f rounding towards zero, compute f' from f rounded towards infinity, and round f' up (if f positive) or round f' down (if f negative). In all 3 cases you can use what you computed to figure out bounds on f, (1) is direct, the other two need extra work.
- kragen 1y agoThis doesn't help if your computation for f is numerically unstable, unless you compute f with interval arithmetic too.
- fph 1y ago> You need to do automatic differentiation with interval arithmetic. But that kinda defeats the point of replacing interval arithmetic with gradients, though.