4 ms·

This is the argument made against all software stack advancements. Nothing to do with industry. But when the benefits outweigh the the hurdles, change happens.

by davidu 6y ago

This is the argument made against all software stack advancements. Nothing to do with industry. But when the benefits outweigh the the hurdles, change happens. And if I was starting a new insurance company (which I've considered) I'd be doing our work in code not xls, and probably python. Having RCS, Numpy, unlimited compute, unlimited storage, all gives me an advantage over my competition. :-) As to the memoization, that is not hard to manage in Python.

yold__ 6y ago
"As to the memoization, that is not hard to manage in Python." Yes it is. Recursive calls for financial calculations easily go hundreds of thousands of calls deep. This is why high-end actuarial modeling software either decomposes it into a dependency graph and unrolls function calls where possible, or just "brute-forces" it by being a thin wrapper over c++, i.e. using operator overloading on ::operator(). I've seen ill-fated efforts of capable software developers attempting to unroll the recursive function calls, and ending up with 2000 line functions that are impossible to maintain.
- solresol 6y ago
  Why doesn't annotating these functions with @functools.lru_cache(10000000) work?
  - pmart123 6y ago
    My guess is the numeric inputs would be changing significantly each call?
  - yold__ 6y ago
    First of all, let me say that I've tried it :) Your recursion needs to "bottom-out" in order for that to work. If you don't get a stack overflow / out of memory error, you're good. But bear in mind that there will be thousands of stack frames. Before you get to time=0 (the recursive base case) in a long-term liability actuarial calc. The recursion isn't simple like the Fibonacci sequence . It's more like: f(t+1) = if t > 0 (f(t) + g(t)) * h(t) else initial_constant g(t) = f(t) + q(t) - d(t) q(t) = .... d(t) = ....
    - andi999 6y ago
      Although at fist glance this formula is written recursively, one doesn't have to (and shouldn't) implement using recursion, does one? Just making f, g, q, d arrays and then loop over t should be good, or is there more to this formula?
      - yold__ 6y ago
        Appreciate the curiosity. In this small trivial case, yes that works. But what happens when something in the logic changes? You wind up needing to know the order of calculations since things are no longer lazily evaluated via recursion. This is a problem when you have dozens of "columns" (i.e. recursive functions or arrays as you are suggesting). Often times, the value in the array is NULL (or worse, leftover from a previous calculation). You are left to manually try and re-order the calculations, which is not trivial when there are hundreds of functions. Excel takes care of these details for you automatically. Users program functionally and recursively (fill-down) without even thinking about it. Excel reactively updates when dependent values change (re-evaluates as necessary). If power, speed, and scale are necessary, there are purpose-built systems (with Domain Specific Languages) which specifically solve this problem in the insurance domain (e.g. FIS Prophet, Risk Agility, AXIS, etc).
        BuuQu9hu2 6y ago
        It is common knowledge that all recursive functions can be re-written using iteration (e.g. loops). See “ Recursion versus iteration” here https://en.m.wikipedia.org/wiki/Recursion_(computer_science) https://en.m.wikipedia.org/wiki/Recursion_(computer_science). The assumption that only trivial calculations can occur using iteration, or that recursion alone allows for supportable code, I believe are very flawed assumptions.
    - BuuQu9hu2 6y ago
      Um, ever heard of a “for loop”? An obvious alternative to recursion.
  - Der_Einzige 6y ago
    Not sure why this is being downvoted since I don't think the OP has done a good job of showing evidence that this doesn't work for recursion. Your computer almost certainly WILL have enough space for all the stack frames necessary.
    - deleted 6y ago
      [deleted]
    - deleted 6y ago
      [deleted]
- edmundsauto 6y ago
  I can't visualize what you mean by deep recursive calls. What are the calculations that mean you can't just use fairly bog standard python for? I didn't realize there was "big data" in accounting.
klelatti 6y ago
Agree 100% with this. Better analytics can be a key competitive advantage for insurers and modern tools / cloud offer potential to be much better than Excel. I've been working on a product that turns JupyterLab into an IDE for life insurance calculations - Python API wrapped around an optimised C / GPU computation layer underneath, all integrated with key open source libraries.

by davidu 6y ago

yold__ 6y ago

"As to the memoization, that is not hard to manage in Python." Yes it is. Recursive calls for financial calculations easily go hundreds of thousands of calls deep. This is why high-end actuarial modeling software either decomposes it into a dependency graph and unrolls function calls where possible, or just "brute-forces" it by being a thin wrapper over c++, i.e. using operator overloading on ::operator(). I've seen ill-fated efforts of capable software developers attempting to unroll the recursive function calls, and ending up with 2000 line functions that are impossible to maintain.

solresol 6y ago
Why doesn't annotating these functions with @functools.lru_cache(10000000) work?
- pmart123 6y ago
  My guess is the numeric inputs would be changing significantly each call?
- yold__ 6y ago
  First of all, let me say that I've tried it :) Your recursion needs to "bottom-out" in order for that to work. If you don't get a stack overflow / out of memory error, you're good. But bear in mind that there will be thousands of stack frames. Before you get to time=0 (the recursive base case) in a long-term liability actuarial calc. The recursion isn't simple like the Fibonacci sequence . It's more like: f(t+1) = if t > 0 (f(t) + g(t)) * h(t) else initial_constant g(t) = f(t) + q(t) - d(t) q(t) = .... d(t) = ....
  - andi999 6y ago
    Although at fist glance this formula is written recursively, one doesn't have to (and shouldn't) implement using recursion, does one? Just making f, g, q, d arrays and then loop over t should be good, or is there more to this formula?
    - yold__ 6y ago
      Appreciate the curiosity. In this small trivial case, yes that works. But what happens when something in the logic changes? You wind up needing to know the order of calculations since things are no longer lazily evaluated via recursion. This is a problem when you have dozens of "columns" (i.e. recursive functions or arrays as you are suggesting). Often times, the value in the array is NULL (or worse, leftover from a previous calculation). You are left to manually try and re-order the calculations, which is not trivial when there are hundreds of functions. Excel takes care of these details for you automatically. Users program functionally and recursively (fill-down) without even thinking about it. Excel reactively updates when dependent values change (re-evaluates as necessary). If power, speed, and scale are necessary, there are purpose-built systems (with Domain Specific Languages) which specifically solve this problem in the insurance domain (e.g. FIS Prophet, Risk Agility, AXIS, etc).
      - BuuQu9hu2 6y ago
        It is common knowledge that all recursive functions can be re-written using iteration (e.g. loops). See “ Recursion versus iteration” here https://en.m.wikipedia.org/wiki/Recursion_(computer_science) https://en.m.wikipedia.org/wiki/Recursion_(computer_science). The assumption that only trivial calculations can occur using iteration, or that recursion alone allows for supportable code, I believe are very flawed assumptions.
  - BuuQu9hu2 6y ago
    Um, ever heard of a “for loop”? An obvious alternative to recursion.
- Der_Einzige 6y ago
  Not sure why this is being downvoted since I don't think the OP has done a good job of showing evidence that this doesn't work for recursion. Your computer almost certainly WILL have enough space for all the stack frames necessary.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
edmundsauto 6y ago
I can't visualize what you mean by deep recursive calls. What are the calculations that mean you can't just use fairly bog standard python for? I didn't realize there was "big data" in accounting.

klelatti 6y ago

Agree 100% with this. Better analytics can be a key competitive advantage for insurers and modern tools / cloud offer potential to be much better than Excel. I've been working on a product that turns JupyterLab into an IDE for life insurance calculations - Python API wrapped around an optimised C / GPU computation layer underneath, all integrated with key open source libraries.