3 ms·

Another good approach can be to start by prototyping something in MATLAB, numpy/scipy, or Julia, and then consult their docs when whatever you're doing is falli

Another good approach can be to start by prototyping something in MATLAB, numpy/scipy, or Julia, and then consult their docs when whatever you're doing is falling short in some way. All of these computing environments have extensive references. Look up the function you're using in their docs and you'll find links to papers and books with more context. Being able to "help yourself" in numerical methods basically requires that you're comfortable in the literature. This assumes that you have a very sophisticated and precise need which is not satisfied by what existing libraries offer. If you have some quick one-off need, copying something from Numerical Recipes is a bad idea. Bugs in numerical methods can be very subtle and difficult to diagnose in a way that normal discrete algorithms aren't. If you've never heard of the asymptotic order of convergence of a numerical algorithm, how are you possibly going to detect that you have a bug there, let alone fix it? In my experience, most people who have one-off needs don't have the requisite background in numerical analysis to get themselves out of trouble when things go wrong. A better approach is to just use a library. There are lots of options for simple stuff. If you do have a sophisticated need, you will outgrow what Numerical Recipes offers very quickly. There are some good books in the list you linked, but that list is also very outdated. Notably absent: any of Trefethen's books (ATAP in particular), Nocedal and Wright's "Numerical Optimization", Stoer & Bulirsch, anything about convex optimization, anything about the finite element method, anything about geometry or CAD, anything about Monte Carlo, anything about special functions, and on and on...

leoc 2y ago
Would you have the time and inclination to have a go at doing your own list sometime, perhaps? I'm sure it might be very widely appreciated.
- sfpotter 2y ago
  A general pitfall in numerical methods is settling on a bad old method without understanding what all your options are. It's easy to get stuck on suboptimal methods, especially if you don't look at more recent references and the literature. Not all new methods are good; lots of old methods are great. There are a dozen different ways to solve any given problem. Usually, each method will be ~equally as hard for you the implementer to understand, but they can have wildly different performance characteristics. This is the main reason to avoid looking at something like Numerical Recipes---it will get you stuck in a local minimum. I'll try to give books that are good starting points and provide a useful and correct orientation for each topic. For each of these topics, there are many, many other books. I don't know all the books---I'm sure I'm missing other great references. General numerical analysis reference: Suli & Mayers, Stoer & Bulirsch, Ridgway Scott's book. There are many others---these are the ones I like. Suli & Mayers is truly excellent. Stoer & Bulirsch is comprehensive. Linear algebra, etc: Trefethen & Bau, "Matrix Computations". Yousef Saad's books. Greenbaum's book. There are other key topics not covered much by these, namely sparse direct solvers. Tim Davis has a good intro book. Martinsson has a new book on fast direct solvers---unclear how useful such solvers are in general. Approximation theory: "Approximation Theory and Approximation Practice" by Trefethen. Extremely important book. Many numerical methods for solving 1D problems (rootfinding, approximation, optimization, solving differential equations, nonlinear boundary value problems, representing special functions, etc) are made completely obsolete by using Chebyshev approximation, at least when the function you're working with is analytic (and even when not, you can get very far by using a "Chebyshev spline"). If all you're doing is calculus on a computer, this is probably the only book you need to read. Optimization: Nocedal & Wright. Boyd & Vandenberghe. Bertsekas has great books. Interval arithmetic: Moore, Kearfott & Cloud. This topic doesn't get as much play in the academic applied math/scientific computing/numerical analysis community, which is a real shame, IMO. Gives you the basic tools to get into verified numerics, where you can write algorithms which work provably correctly, allowing you to avoid epsilon hacking to some extent. E.g., "do X if |small number| < eps, otherwise do Y". Without verified numerics it's impossible to build correct algorithms on these kinds of predicates with floating point arithmetic). This covers what I would consider the "basic" topics. At least, if you're doing something else that's higher level (like solving PDEs), you need stuff from this toolbox.
rramadass 2y ago
The point was not to dismiss NR but use it as a good start to know about various algorithms (ignore the idiotic software licensing terms). Agreed that for usage itself, a well known library is to be preferred (my first goto is GSL) unless one knows (and needs) better. The list of alternatives itself is from 2001 and hence does not have later published books. A google search brings up other discussions on alternatives.

leoc 2y ago

Would you have the time and inclination to have a go at doing your own list sometime, perhaps? I'm sure it might be very widely appreciated.

sfpotter 2y ago
A general pitfall in numerical methods is settling on a bad old method without understanding what all your options are. It's easy to get stuck on suboptimal methods, especially if you don't look at more recent references and the literature. Not all new methods are good; lots of old methods are great. There are a dozen different ways to solve any given problem. Usually, each method will be ~equally as hard for you the implementer to understand, but they can have wildly different performance characteristics. This is the main reason to avoid looking at something like Numerical Recipes---it will get you stuck in a local minimum. I'll try to give books that are good starting points and provide a useful and correct orientation for each topic. For each of these topics, there are many, many other books. I don't know all the books---I'm sure I'm missing other great references. General numerical analysis reference: Suli & Mayers, Stoer & Bulirsch, Ridgway Scott's book. There are many others---these are the ones I like. Suli & Mayers is truly excellent. Stoer & Bulirsch is comprehensive. Linear algebra, etc: Trefethen & Bau, "Matrix Computations". Yousef Saad's books. Greenbaum's book. There are other key topics not covered much by these, namely sparse direct solvers. Tim Davis has a good intro book. Martinsson has a new book on fast direct solvers---unclear how useful such solvers are in general. Approximation theory: "Approximation Theory and Approximation Practice" by Trefethen. Extremely important book. Many numerical methods for solving 1D problems (rootfinding, approximation, optimization, solving differential equations, nonlinear boundary value problems, representing special functions, etc) are made completely obsolete by using Chebyshev approximation, at least when the function you're working with is analytic (and even when not, you can get very far by using a "Chebyshev spline"). If all you're doing is calculus on a computer, this is probably the only book you need to read. Optimization: Nocedal & Wright. Boyd & Vandenberghe. Bertsekas has great books. Interval arithmetic: Moore, Kearfott & Cloud. This topic doesn't get as much play in the academic applied math/scientific computing/numerical analysis community, which is a real shame, IMO. Gives you the basic tools to get into verified numerics, where you can write algorithms which work provably correctly, allowing you to avoid epsilon hacking to some extent. E.g., "do X if |small number| < eps, otherwise do Y". Without verified numerics it's impossible to build correct algorithms on these kinds of predicates with floating point arithmetic). This covers what I would consider the "basic" topics. At least, if you're doing something else that's higher level (like solving PDEs), you need stuff from this toolbox.

rramadass 2y ago

The point was not to dismiss NR but use it as a good start to know about various algorithms (ignore the idiotic software licensing terms). Agreed that for usage itself, a well known library is to be preferred (my first goto is GSL) unless one knows (and needs) better. The list of alternatives itself is from 2001 and hence does not have later published books. A google search brings up other discussions on alternatives.