3 ms·

In the original Prophet paper (https://peerj.com/preprints/3190.pdf https://peerj.com/preprints/3190.pdf) they claim that Prophet outperforms ETS (see Figure 7,

by Imnimo 4y ago

In the original Prophet paper (https://peerj.com/preprints/3190.pdf https://peerj.com/preprints/3190.pdf) they claim that Prophet outperforms ETS (see Figure 7, for example). And in the NeuralProphet paper, they claim that it outperforms Prophet (but do not, as far as I can see, compare directly to ETS). Here we see ETS outperforms NeuralProphet. Presumably this apparent non-transitivity is because of differences in each evaluation. If we fix the evaluation to the method used here, is it still the case that NeuralProphet outperforms Prophet (and therefore the claim that Prophet outperforms ETS is not correct)? Or is it that NeuralProphet does not outperform Prophet, but Prophet does outperform ETS?

fedegr 4y ago
I think the problem arises from the datasets used to evaluate the performance of the models. In the case of Prophet's paper, only one time series is used (The number of events created on Facebook). We can conclude from the results comparing AutoARIMA vs. Prophet (https://github.com/Nixtla/statsforecast/tree/main/experiments/arima_prophet_adapter https://github.com/Nixtla/statsforecast/tree/main/experiment..., using the same datasets as in the ETS vs. NeuralProphet experiment) that ETS is also better than Prophet. Regarding NeuralProphet vs. Prophet, the results are not conclusive for these datasets.
maxmc 4y ago
Figure 7 of the mentioned paper evaluates FB-Prophet in a extremely convenient environment of long horizon h in {30,60,90,120,150,180}. It is known that ETS and ARIMA models concatenate errors and degrade in performance with longer forecasting horizons. We have explored and offered solutions to these issues with the N-HiTS model specialized in long-horizon (https://arxiv.org/abs/2201.12886 https://arxiv.org/abs/2201.12886). In recent years, FB-Prophet has gained a reputation for the poor quality of its predictions in many practical scenarios (short/medium term horizon) and its slow performance on bigger data sets; our ARIMA/ETS work and NeuralProphet confirmed those suspicions. The mentioned 92 percent improvements of the paper are restricted to h in {1,3,15,60} (https://arxiv.org/pdf/2111.15397.pdf https://arxiv.org/pdf/2111.15397.pdf). This post's results are rather for short-horizon tasks, the same as NeuralProphet experiments. But we are confident that specialized tools like N-HiTS would outperform Prophet in long-horizon settings.
- disgruntledphd2 4y ago
  Prophets original use case was for forecasting extremely long time series rather automatically. If that's not your use case, it may not be the right fit.

by Imnimo 4y ago

fedegr 4y ago

I think the problem arises from the datasets used to evaluate the performance of the models. In the case of Prophet's paper, only one time series is used (The number of events created on Facebook). We can conclude from the results comparing AutoARIMA vs. Prophet (https://github.com/Nixtla/statsforecast/tree/main/experiments/arima_prophet_adapter https://github.com/Nixtla/statsforecast/tree/main/experiment..., using the same datasets as in the ETS vs. NeuralProphet experiment) that ETS is also better than Prophet. Regarding NeuralProphet vs. Prophet, the results are not conclusive for these datasets.

maxmc 4y ago

Figure 7 of the mentioned paper evaluates FB-Prophet in a extremely convenient environment of long horizon h in {30,60,90,120,150,180}. It is known that ETS and ARIMA models concatenate errors and degrade in performance with longer forecasting horizons. We have explored and offered solutions to these issues with the N-HiTS model specialized in long-horizon (https://arxiv.org/abs/2201.12886 https://arxiv.org/abs/2201.12886). In recent years, FB-Prophet has gained a reputation for the poor quality of its predictions in many practical scenarios (short/medium term horizon) and its slow performance on bigger data sets; our ARIMA/ETS work and NeuralProphet confirmed those suspicions. The mentioned 92 percent improvements of the paper are restricted to h in {1,3,15,60} (https://arxiv.org/pdf/2111.15397.pdf https://arxiv.org/pdf/2111.15397.pdf). This post's results are rather for short-horizon tasks, the same as NeuralProphet experiments. But we are confident that specialized tools like N-HiTS would outperform Prophet in long-horizon settings.

disgruntledphd2 4y ago
Prophets original use case was for forecasting extremely long time series rather automatically. If that's not your use case, it may not be the right fit.