4 ms·

Has anyone else struggled with Prophet? I've experimented with it on a few real world datasets and I've had very inconsistent results.

jgalt212 3y ago
maybe because time series forecasting, for any time series of interest, is pretty much not possible.
ttt333 3y ago
Yes. I've tried using it for pretty straightforward time series forecasts, and I struggled to make it into something useful in a business context. I'll disclaim that I'm just a finance dude and not a data scientist or programmer. But the documentation leads me to believe that I am in the target audience. I felt like I could grasp the basic mechanics after reading the paper, but I wish the documentation could help someone like me be more intelligent with the 'tuning' of the model. I could never get accuracy below 15% average error, which is too large for my use case. Probably user ignorance, but that's my experience.
- SpaceManNabs 3y ago
  You are the primary audience. Time series forecasting with deep learning is fraught with inconsistency. Someone on r/ML went pretty hard on detailing a survey and the stuff that was SOTA 10 years ago still is. Wish I saved that thread. The dude was well published. edit: found it https://www.reddit.com/r/MachineLearning/comments/pe1lst/r_irrational_exuberance_why_we_should_not_believe/ https://www.reddit.com/r/MachineLearning/comments/pe1lst/r_i... Turns out it was about time series anomaly detection, but if you can detect, you can forecast if your model is generative
  - Pandabob 3y ago
    When was this? I might go chasing this lead down, but even a fuzzy estimation of when would help. Will come link it here if I find it.
    - SpaceManNabs 3y ago
      I updated my comment!
  - isaacfung 3y ago
    These may be related. https://www.reddit.com/r/datascience/comments/11vzjhi/statistical_vs_machine_learning_vs_deep_learning/ https://www.reddit.com/r/datascience/comments/11vzjhi/statis... https://www.reddit.com/user/eamonnkeogh/submitted/?sort=top https://www.reddit.com/user/eamonnkeogh/submitted/?sort=top
    - SpaceManNabs 3y ago
      I updated my comment with the thread but it was actually about time series anomaly detection. Turns out it was the same dude in your second link, and your comment includes forecasting in the first link as well. Thank you!
  - genewitch 3y ago
    aaaaand i just spent 3 hours watching that, trying to remember some parts of calculus, and reading all of the wikipedia articles and "also see" that were grey on white in the video. Then i fell asleep, but i wanted to thank you, as i also thanked the Prof. that made that video (on reddit).
- eep_social 3y ago
  This looks to me like something they’d be using for internal capacity planning. If so, they’d be asking it questions like, “how much capacity do we build out for the upcoming holiday rush?” I wouldn't be surprised if financial datasets are very noisy compared to service capacity metrics. I didn’t read the paper though, maybe this is addressed and maybe I’m wrong about the use case! But stuff like the below from the docs reads like capacity planning tool to me: > As an example, let’s look at a time series of the log daily page views for the Wikipedia page for Peyton Manning. We scraped this data using the Wikipediatrend package in R. Peyton Manning provides a nice example because it illustrates some of Prophet’s features, like multiple seasonality, changing growth rates, and the ability to model special days (such as Manning’s playoff and superbowl appearances).
  - philjohn 3y ago
    Also perhaps anomaly detection in a metric.
- hackernewds 3y ago
  I'm sad to see no one has responded with a solution to your problem. You are absolutely the target audience, and in my experience, Prophet is "as good as it gets" to generalized forecasting.
HWR_14 3y ago
Based just on the documentation, it seems there are some assumptions they expect the data to adhere to, and if they don't apply then it would not produce good results.
sbohacek 3y ago
I have not been able to get good results either, but I have not tried it in the past year. I also tried many of the architectures in Darts. I have found that fairly straightforward architectures work well. That is, I can iterate on my own design for my own specific data (with all its specific covariates) and get better results than I could with Darts or Prophet.
neoclassic2077 3y ago
While using Prophet for purely "forecasting" setup might not guarantee consistent high-quality results out of the box, especially for noisy and complicated time series data, at VictoriaMetrics we found it practically useful for anomaly detection task: In our vmanomaly product, Prophet is one of the go-to models for anomaly detection in metrics data and it usually requires little tuning to achieve considerable results. The main purpose for the use of Prophet or similar forecasting models is to reformulate the task of anomaly detection: - given fitted model M, ground truth Y_i for particular data point X_i, we produce forecast Yhat_i and its uncertainty estimate [Yhat_lb, Yhat_ub] - if ground truth Y_i falls beyond the range of [Yhat_lb, Yhat_ub], we consider this point an anomaly - the further Y_i is from the range, the higher the anomaly score would be. In our particular implementation for easier alerting purposes, anomaly_score > 1 means "anomaly" here's a small visual example: https://docs.victoriametrics.com/vmanomaly.html#examples https://docs.victoriametrics.com/vmanomaly.html#examples

jgalt212 3y ago

maybe because time series forecasting, for any time series of interest, is pretty much not possible.

ttt333 3y ago

Yes. I've tried using it for pretty straightforward time series forecasts, and I struggled to make it into something useful in a business context. I'll disclaim that I'm just a finance dude and not a data scientist or programmer. But the documentation leads me to believe that I am in the target audience. I felt like I could grasp the basic mechanics after reading the paper, but I wish the documentation could help someone like me be more intelligent with the 'tuning' of the model. I could never get accuracy below 15% average error, which is too large for my use case. Probably user ignorance, but that's my experience.

SpaceManNabs 3y ago
You are the primary audience. Time series forecasting with deep learning is fraught with inconsistency. Someone on r/ML went pretty hard on detailing a survey and the stuff that was SOTA 10 years ago still is. Wish I saved that thread. The dude was well published. edit: found it https://www.reddit.com/r/MachineLearning/comments/pe1lst/r_irrational_exuberance_why_we_should_not_believe/ https://www.reddit.com/r/MachineLearning/comments/pe1lst/r_i... Turns out it was about time series anomaly detection, but if you can detect, you can forecast if your model is generative
- Pandabob 3y ago
  When was this? I might go chasing this lead down, but even a fuzzy estimation of when would help. Will come link it here if I find it.
  - SpaceManNabs 3y ago
    I updated my comment!
- isaacfung 3y ago
  These may be related. https://www.reddit.com/r/datascience/comments/11vzjhi/statistical_vs_machine_learning_vs_deep_learning/ https://www.reddit.com/r/datascience/comments/11vzjhi/statis... https://www.reddit.com/user/eamonnkeogh/submitted/?sort=top https://www.reddit.com/user/eamonnkeogh/submitted/?sort=top
  - SpaceManNabs 3y ago
    I updated my comment with the thread but it was actually about time series anomaly detection. Turns out it was the same dude in your second link, and your comment includes forecasting in the first link as well. Thank you!
- genewitch 3y ago
  aaaaand i just spent 3 hours watching that, trying to remember some parts of calculus, and reading all of the wikipedia articles and "also see" that were grey on white in the video. Then i fell asleep, but i wanted to thank you, as i also thanked the Prof. that made that video (on reddit).
eep_social 3y ago
This looks to me like something they’d be using for internal capacity planning. If so, they’d be asking it questions like, “how much capacity do we build out for the upcoming holiday rush?” I wouldn't be surprised if financial datasets are very noisy compared to service capacity metrics. I didn’t read the paper though, maybe this is addressed and maybe I’m wrong about the use case! But stuff like the below from the docs reads like capacity planning tool to me: > As an example, let’s look at a time series of the log daily page views for the Wikipedia page for Peyton Manning. We scraped this data using the Wikipediatrend package in R. Peyton Manning provides a nice example because it illustrates some of Prophet’s features, like multiple seasonality, changing growth rates, and the ability to model special days (such as Manning’s playoff and superbowl appearances).
- philjohn 3y ago
  Also perhaps anomaly detection in a metric.
hackernewds 3y ago
I'm sad to see no one has responded with a solution to your problem. You are absolutely the target audience, and in my experience, Prophet is "as good as it gets" to generalized forecasting.

HWR_14 3y ago

Based just on the documentation, it seems there are some assumptions they expect the data to adhere to, and if they don't apply then it would not produce good results.

sbohacek 3y ago

I have not been able to get good results either, but I have not tried it in the past year. I also tried many of the architectures in Darts. I have found that fairly straightforward architectures work well. That is, I can iterate on my own design for my own specific data (with all its specific covariates) and get better results than I could with Darts or Prophet.

neoclassic2077 3y ago

While using Prophet for purely "forecasting" setup might not guarantee consistent high-quality results out of the box, especially for noisy and complicated time series data, at VictoriaMetrics we found it practically useful for anomaly detection task: In our vmanomaly product, Prophet is one of the go-to models for anomaly detection in metrics data and it usually requires little tuning to achieve considerable results. The main purpose for the use of Prophet or similar forecasting models is to reformulate the task of anomaly detection: - given fitted model M, ground truth Y_i for particular data point X_i, we produce forecast Yhat_i and its uncertainty estimate [Yhat_lb, Yhat_ub] - if ground truth Y_i falls beyond the range of [Yhat_lb, Yhat_ub], we consider this point an anomaly - the further Y_i is from the range, the higher the anomaly score would be. In our particular implementation for easier alerting purposes, anomaly_score > 1 means "anomaly" here's a small visual example: https://docs.victoriametrics.com/vmanomaly.html#examples https://docs.victoriametrics.com/vmanomaly.html#examples