4 ms·

I have worked as data scientist on a lot of finance domain problems - forecasting default, fraud, conversion probability ect. Lightgbm library has consistently

I have worked as data scientist on a lot of finance domain problems - forecasting default, fraud, conversion probability ect. Lightgbm library has consistently performed well. I've been interested in how many colleagues instantly jump to neural nets when in my experience this often doesn't beat lightgbm on medium sized datasets not related to text/images.

elmalto 7y ago
Our team has a similar experience using xgboost and catboost. Models perform great and beat (our) neural nets
abhgh 7y ago
LightGBM occupies a sweet spot between speed and accuracy, and is a library I've grown to love. Nowadays, this is my primary choice for quick impactful results. NB: if your data has categorical features, you might easily beat xgboost in training time, since LightGBM explicitly supports them, and for xgboost you would need to use one hot encoding, increasing the size of the data that the library needs to work with.
suresk 7y ago
I think this is pretty common. For tabular data, lightgbm/xgboost/catboost usually give better results and require a lot less work (less pre-processing, for example) than neural nets. One area where I wonder if neural nets would be a more useful option is using something like an LSTM to predict defaults based on a sequence of data? I've tried this a handful of times and doing a bit of feature engineering to aggregate data in a handful of fixed buckets has usually been better and easier, but I'm far from an expert in that area. I know Jeremy Howard has shown decent results with fastai/pytorch for tabular data and I've seen some Kaggle teams do well with neural nets for tabular data. I've also had decent results with gbdt/nn ensembles. But I think in most situations where you just have tabular data, you'll get better results with less effort if you use lightgbm or the like.
jimfleming 7y ago
More anecdata: we consistently outperform lightgbm, xgboost, random forests, linear models, etc. using neural networks even on smaller datasets. This applies whether we implemented the other algorithms ourselves or simply compared to someone else’s results with them. In my experience it really comes down to how many “tricks” you know for each algorithm and how well can you apply and combine these “tricks”. The difference is that neural networks have many more of these tricks and a broader coverage of research detailing the interactions between them. I call them “tricks” but really they’re just design decisions based on what current research indicates about certain problems. This is largely where the “art” part of neural networks comes from that many people refer. The search space is simply too big to try everything and hope for the best. Therefore, how a problem is approached and how solutions are narrowed and applied really matter. Even simple things like which optimizer you use, how you leverage learning rate schedules, how the loss function is formulated, how weights are updated, feature engineering (often neglected in neural networks), and architectural priors make a big difference on both sample efficiency and overall performance. Most people, if they’re not just fine-tuning an existing model, simply load up a neural network framework, stack some layers together and throw data at it expecting better results than other approaches. But there’s a huge spectrum from that naive approach to architecting a custom model. This is why neural networks are so powerful and why we tend to favor it (though not for every problem). It’s much easier to design a model from the ground up with neural networks than it is for e.g. xgboost because not only are the components more easily composable thanks to the available frameworks but there’s a ton more research on the specific interactions between those components. That doesn’t mean than every problem is appropriate for neural networks. I completely agree with you that no matter what the problem is you should never jump to an approach just because its popular. Neural networks are a tool and for many problems you need to be comfortable with every one of those decision points to get the best results and even if you’re comfortable it can take time and that isn’t always appropriate for every problem. My other point is that I wouldn’t draw too many conclusions about a particular algorithm being better or worse than another. I’m not saying that was the intention with your comment but I know many people in the ML industry tend to take a similar position. It really depends on current experience with the applied algorithms, not just experience with ML in general.
- suresk 7y ago
hobofromabroad 7y ago
The problem with these models is that you have to be careful that your are not modeling something that incorporates a trend. And if you have any kind of seasonality you a dataset with a large enough timeframe. (At least more than a year.) Nonetheless, LightGBM and xgboost are also commonly used in the insurance sector. They are still somewhat problematic for conversion rates in a highly dynamic market though.

elmalto 7y ago

Our team has a similar experience using xgboost and catboost. Models perform great and beat (our) neural nets

abhgh 7y ago

LightGBM occupies a sweet spot between speed and accuracy, and is a library I've grown to love. Nowadays, this is my primary choice for quick impactful results. NB: if your data has categorical features, you might easily beat xgboost in training time, since LightGBM explicitly supports them, and for xgboost you would need to use one hot encoding, increasing the size of the data that the library needs to work with.

suresk 7y ago

I think this is pretty common. For tabular data, lightgbm/xgboost/catboost usually give better results and require a lot less work (less pre-processing, for example) than neural nets. One area where I wonder if neural nets would be a more useful option is using something like an LSTM to predict defaults based on a sequence of data? I've tried this a handful of times and doing a bit of feature engineering to aggregate data in a handful of fixed buckets has usually been better and easier, but I'm far from an expert in that area. I know Jeremy Howard has shown decent results with fastai/pytorch for tabular data and I've seen some Kaggle teams do well with neural nets for tabular data. I've also had decent results with gbdt/nn ensembles. But I think in most situations where you just have tabular data, you'll get better results with less effort if you use lightgbm or the like.

jimfleming 7y ago

More anecdata: we consistently outperform lightgbm, xgboost, random forests, linear models, etc. using neural networks even on smaller datasets. This applies whether we implemented the other algorithms ourselves or simply compared to someone else’s results with them. In my experience it really comes down to how many “tricks” you know for each algorithm and how well can you apply and combine these “tricks”. The difference is that neural networks have many more of these tricks and a broader coverage of research detailing the interactions between them. I call them “tricks” but really they’re just design decisions based on what current research indicates about certain problems. This is largely where the “art” part of neural networks comes from that many people refer. The search space is simply too big to try everything and hope for the best. Therefore, how a problem is approached and how solutions are narrowed and applied really matter. Even simple things like which optimizer you use, how you leverage learning rate schedules, how the loss function is formulated, how weights are updated, feature engineering (often neglected in neural networks), and architectural priors make a big difference on both sample efficiency and overall performance. Most people, if they’re not just fine-tuning an existing model, simply load up a neural network framework, stack some layers together and throw data at it expecting better results than other approaches. But there’s a huge spectrum from that naive approach to architecting a custom model. This is why neural networks are so powerful and why we tend to favor it (though not for every problem). It’s much easier to design a model from the ground up with neural networks than it is for e.g. xgboost because not only are the components more easily composable thanks to the available frameworks but there’s a ton more research on the specific interactions between those components. That doesn’t mean than every problem is appropriate for neural networks. I completely agree with you that no matter what the problem is you should never jump to an approach just because its popular. Neural networks are a tool and for many problems you need to be comfortable with every one of those decision points to get the best results and even if you’re comfortable it can take time and that isn’t always appropriate for every problem. My other point is that I wouldn’t draw too many conclusions about a particular algorithm being better or worse than another. I’m not saying that was the intention with your comment but I know many people in the ML industry tend to take a similar position. It really depends on current experience with the applied algorithms, not just experience with ML in general.

suresk 7y ago

hobofromabroad 7y ago

The problem with these models is that you have to be careful that your are not modeling something that incorporates a trend. And if you have any kind of seasonality you a dataset with a large enough timeframe. (At least more than a year.) Nonetheless, LightGBM and xgboost are also commonly used in the insurance sector. They are still somewhat problematic for conversion rates in a highly dynamic market though.