5 ms·

Hi folks! Lukas (Lukas1994) and I built this. If you have a spreadsheet model, it lets you answer questions like "Which variable in my model is the most import

Hi folks! Lukas (Lukas1994) and I built this. If you have a spreadsheet model, it lets you answer questions like "Which variable in my model is the most important?" and "If I underestimate X by 10%, what's the effect on Y?". Most models are built on assumptions and estimates — if you run a company, your financial model will probably include an assumption around your customer growth rate in the future. If you use this model to make decisions — "How many people can we hire in Q4?" — then it's important to understand how sensitive these decisions are to your assumptions. It may turn out that overestimating your growth rate by 10% means you can only hire half as many people, in which case you'll probably want to hire a bit more conservatively. This analysis is pretty cumbersome to do in spreadsheets. More broadly, we think there should be a better way than Excel to crunch numbers and do modelling, and we're trying to build it (https://causal.app https://causal.app). Would love to chat if this sounds interesting: taimur@causal.app :)

sheikheddy 7y ago
I see you're using a variance based approach. How does it handle a dependent structure? To my knowledge, Sobol indices assume that the input variables are independent.
- refrigerator 7y ago
  Yep — we assume that the input variables are independent. This is more acceptable in some settings than others, so as always, it's good to understand the limitations of the model. Thanks for pointing this one out :)
  - sheikheddy 7y ago
    There has been some exploration into variance based global sensitivity analysis for models with dependent inputs. [1][2][3] [1]https://www.sciencedirect.com/science/article/pii/S0951832016301065 https://www.sciencedirect.com/science/article/pii/S095183201... [2]https://www.sciencedirect.com/science/article/pii/S0010465511004085 https://www.sciencedirect.com/science/article/pii/S001046551... [3]https://www.sciencedirect.com/science/article/pii/S0951832011001724 https://www.sciencedirect.com/science/article/pii/S095183201... I am not aware of any software package that implements these algorithms, although I am working towards creating or contributing to a python library that does. Ideally I'd publish a paper in a few months showing how the results (e.g, ranking of importance) differs from the independent case on particular datasets. Edit: It turns out SAlib recently added some of this functionality. I'll check it out!
  - 6gvONxR4sf7o 7y ago
    I'm late to the party. Hoping you can answer a question. On the one hand, you're lowering the bar to do useful statistical work, which is awesome. A new set of people will be able to do better analysis. On the other hand, you're baking in assumptions which this new set of people won't understand, so you aren't necessarily lowering the bar to do correct statistical work. That seems challenging, and I'm curious how you think about it.
- Lukas1994 7y ago
  Yes, right now we assume independence. SALib recently added a few algorithms that can handle correlation: https://github.com/SALib/SALib/issues/193#issuecomment-551955158 https://github.com/SALib/SALib/issues/193#issuecomment-55195... I'm planning to try them out soon :)
sansnomme 7y ago
Are there any benefits to using backpropagation for sensitivity analysis on larger datasets?
- Lukas1994 7y ago
  Can you elaborate on that?
  - sansnomme 7y ago
    The process of backpropagation is figuring out which input has the greatest effect on output and adjusting the weights on the input to fit the output curve. In other words, assuming a trained single layer neural network, the weights alone will tell you which inputs have the greatest effect on output.
    - Lukas1994 7y ago
      Sorry for the late reply, just saw your comment. I think what you're describing is training a model and then using feature importance. This is also a valid method for calculating sensitivities and it's used if the function from inputs to outputs is unknown. However, in our case, this function is known so there's no need to train a predictive model.

sheikheddy 7y ago

I see you're using a variance based approach. How does it handle a dependent structure? To my knowledge, Sobol indices assume that the input variables are independent.

refrigerator 7y ago
Yep — we assume that the input variables are independent. This is more acceptable in some settings than others, so as always, it's good to understand the limitations of the model. Thanks for pointing this one out :)
- sheikheddy 7y ago
  There has been some exploration into variance based global sensitivity analysis for models with dependent inputs. [1][2][3] [1]https://www.sciencedirect.com/science/article/pii/S0951832016301065 https://www.sciencedirect.com/science/article/pii/S095183201... [2]https://www.sciencedirect.com/science/article/pii/S0010465511004085 https://www.sciencedirect.com/science/article/pii/S001046551... [3]https://www.sciencedirect.com/science/article/pii/S0951832011001724 https://www.sciencedirect.com/science/article/pii/S095183201... I am not aware of any software package that implements these algorithms, although I am working towards creating or contributing to a python library that does. Ideally I'd publish a paper in a few months showing how the results (e.g, ranking of importance) differs from the independent case on particular datasets. Edit: It turns out SAlib recently added some of this functionality. I'll check it out!
- 6gvONxR4sf7o 7y ago
  I'm late to the party. Hoping you can answer a question. On the one hand, you're lowering the bar to do useful statistical work, which is awesome. A new set of people will be able to do better analysis. On the other hand, you're baking in assumptions which this new set of people won't understand, so you aren't necessarily lowering the bar to do correct statistical work. That seems challenging, and I'm curious how you think about it.
Lukas1994 7y ago
Yes, right now we assume independence. SALib recently added a few algorithms that can handle correlation: https://github.com/SALib/SALib/issues/193#issuecomment-551955158 https://github.com/SALib/SALib/issues/193#issuecomment-55195... I'm planning to try them out soon :)

sansnomme 7y ago

Are there any benefits to using backpropagation for sensitivity analysis on larger datasets?

Lukas1994 7y ago
Can you elaborate on that?
- sansnomme 7y ago
  The process of backpropagation is figuring out which input has the greatest effect on output and adjusting the weights on the input to fit the output curve. In other words, assuming a trained single layer neural network, the weights alone will tell you which inputs have the greatest effect on output.
  - Lukas1994 7y ago
    Sorry for the late reply, just saw your comment. I think what you're describing is training a model and then using feature importance. This is also a valid method for calculating sensitivities and it's used if the function from inputs to outputs is unknown. However, in our case, this function is known so there's no need to train a predictive model.