5 ms·
This wouldn't pass peer-review of it were a paper. Major issues: - No fair hyperparametrization for Neural Prophet. They mention multiple times they used defau
by gillesjacobs 4y ago
This wouldn't pass peer-review of it were a paper. Major issues:
- No fair hyperparametrization for Neural Prophet. They mention multiple times they used default hyperparams or ad-hoc example hyperparams.
- 3/4 benchmark datasets (one they didn't finish training) where ETS outperforms is not strong evidence of all-round robustness. Benchmarks like SuperGlue for NLP combine 10 completely different tasks with more subtasks to assess language model performance. And even SuperGlue is not uncontroversial.
- gillesjacobs 4y agoWhile the results don't prove the superiority convincingly, it does seem that ETS is a good candidate as a first go-to in practical applications. "In practice, practice and theory are the same. In theory, they are not.”
- variaga 4y agoIn theory practice and theory are the same. In practice they are not.
- gillesjacobs 4y agoHN pedantry ruins the fun of wordplay yet again.
- anon_123g987 4y agoIn theory, his version is right. In practice, yours.
- lr1970 4y agoThe difference between theory and practice is greater in theory than in practice.
- timy2shoes 4y agoIn theory there is no difference between theory and practice. In practice there is. - Yogi Berra
- DoctorOetker 4y ago"Sometimes malpractice is just malpractice" - Sigmund Freud
- maxmc 4y agoThis comparison was not intended as an academic paper. However, we are confident that predicting and evaluating performance over more than 55k series from the M international forecasting competitions (standard benchmarks in the field) is a good start. After corresponding with the author, we included additional data sets from the electricity domain like Ercot and ETTM2. The dataset that did not finish training shows that Neuralprophet simply does not scale. (We let the training run for over 73 hours with 96 CPUs -as NeuralProphet implementation is restricted to CPU-, and we canceled it after investing 288 USD in it) Suggestions to strengthen our experiments (without spending hundreds of dollars) are highly welcomed. Regarding the hyperparameter selection, we went beyond the original NeuralProphet paper, tunning and actively trying to help improve its performance. We even made a PR fixing NP bugs in the process.
- deleted 4y ago[deleted]
- predict_addict 4y agoThis is ironic, because the Neural Prophet paper wasn’t reviewed either and didn’t benchmark it on anything other than Prophet.