4 ms·

20% better than prophet sounds great. But then, I've read a couple of posts discussing in depth how prophet itself was not always the go-to model [1], so perha

20% better than prophet sounds great. But then, I've read a couple of posts discussing in depth how prophet itself was not always the go-to model [1], so perhaps some caution is in order. [1] https://www.microprediction.com/blog/prophet https://www.microprediction.com/blog/prophet [edit] And this one, linked in [1]: https://kourentzes.com/forecasting/2017/07/29/benchmarking-facebooks-prophet/ https://kourentzes.com/forecasting/2017/07/29/benchmarking-f...

confounded 5y ago
If this is on the M5 data set as a benchmark, Prophet wouldn't be a great model choice at all. It's fantastic for dense, long, univariate time series with strong seasonality. M5 is very very sparse and intermittent. The M5 competition was won with lgbm, which isn't a time-series model at all. (Edit: looks like this defaults to something like lgbm, so, unsurprising results.)
quantperson 5y ago
It's not hard to beat Prophet by >20%. As the paper itself states, it's designed for rapidly producing good enough forecasts at scale by domain experts with no stats background. If you have a small team consisting of domain experts, engineers and data science/statisticians, you should be able to build something which handily beats Prophet. If you're careful, you should be able to design a pipeline which can do the model tuning and selection in something resembling an "automated" manner (but it requires extreme care, and Prophet can't save you from the dangers herein either). Prophet has a relatively narrow use case where it makes sense; not coincidentally, it makes sense at a place like Facebook where the sheer variety of problems to be tackled cannot scale to such a foregoing team for each such problem. Prophet has received a lot of backlash for workflows it never intended, nor claimed, to be able to tackle. That's undeserved in my opinion. Zillow didn't lose money because it used Prophet, for example. But it still really shouldn't be used as a benchmark like this.
fedegr 5y ago
Thanks for your comments. We agree that in most cases prophet is not a good benchmark; however, we wanted to use it because it is one of the most used libraries in forecasting. For that reason, we also tested the solution against AWS Forecast obtaining better results. Besides the better performance and scalability, the pipeline we created considering all the stages of time series forecasting: preprocessing (e.g. missing value imputation), creation of static and dynamic features, forecast generation, and finally evaluation using data sets of important competencies. (https://github.com/Nixtla/tsfeatures https://github.com/Nixtla/tsfeatures) On the deployment side, the entire pipeline can be quickly deployed in the user's cloud using terraform. This allows for less development time. (https://github.com/Nixtla/nixtla https://github.com/Nixtla/nixtla)

confounded 5y ago

If this is on the M5 data set as a benchmark, Prophet wouldn't be a great model choice at all. It's fantastic for dense, long, univariate time series with strong seasonality. M5 is very very sparse and intermittent. The M5 competition was won with lgbm, which isn't a time-series model at all. (Edit: looks like this defaults to something like lgbm, so, unsurprising results.)

quantperson 5y ago

It's not hard to beat Prophet by >20%. As the paper itself states, it's designed for rapidly producing good enough forecasts at scale by domain experts with no stats background. If you have a small team consisting of domain experts, engineers and data science/statisticians, you should be able to build something which handily beats Prophet. If you're careful, you should be able to design a pipeline which can do the model tuning and selection in something resembling an "automated" manner (but it requires extreme care, and Prophet can't save you from the dangers herein either). Prophet has a relatively narrow use case where it makes sense; not coincidentally, it makes sense at a place like Facebook where the sheer variety of problems to be tackled cannot scale to such a foregoing team for each such problem. Prophet has received a lot of backlash for workflows it never intended, nor claimed, to be able to tackle. That's undeserved in my opinion. Zillow didn't lose money because it used Prophet, for example. But it still really shouldn't be used as a benchmark like this.

fedegr 5y ago

Thanks for your comments. We agree that in most cases prophet is not a good benchmark; however, we wanted to use it because it is one of the most used libraries in forecasting. For that reason, we also tested the solution against AWS Forecast obtaining better results. Besides the better performance and scalability, the pipeline we created considering all the stages of time series forecasting: preprocessing (e.g. missing value imputation), creation of static and dynamic features, forecast generation, and finally evaluation using data sets of important competencies. (https://github.com/Nixtla/tsfeatures https://github.com/Nixtla/tsfeatures) On the deployment side, the entire pipeline can be quickly deployed in the user's cloud using terraform. This allows for less development time. (https://github.com/Nixtla/nixtla https://github.com/Nixtla/nixtla)