11 ms·
Prophet: forecasting at scale
- asafira 10y agoSo...How much will this do at forecasting stock prices? =) Very cool though --- I would be interested to dive into the methods they've implemented sometime in the near future!
- blazespin 10y agoProbably just help verify that the stock market is a random walk with a meager trend upwards that doesn't beat inflation + trading costs.
- matheweis 10y ago> the stock market is a random walk with a meager trend upwards that doesn't beat inflation + trading costs. That assumes that the efficient-market hypothesis holds true, but it has yet to be thoroughly proven or disproven... (and funds like Medallion would strongly suggest otherwise for the medium term: https://www.bloomberg.com/news/articles/2016-11-21/how-renaissance-s-medallion-fund-became-finance-s-blackest-box https://www.bloomberg.com/news/articles/2016-11-21/how-renai...)
- dandermotj 10y agoIt doesn't assume the Effiecient Market hypothesis - empirical studies of returns support random returns without the imposing a model (non-parametric tests). That's not to say returns are actually random, but in any given time range, it appears to be.
- matheweis 10y agoEMH and random walk theory are intrinsically linked; you can't have one without the other... Or are you saying that movements aren't actually random, and only appear to be?
- dandermotj 10y agoThe latter
- deleted 10y ago[deleted]
- etjossem 10y ago> Probably just help verify that the stock market is a random walk with a meager trend upwards that doesn't beat inflation + trading costs. That doesn't sound right. Let me clear that up for you. Since 1950: S&P 500 Annual Price Change: 7.2% S&P 500 Annual Div Dist: 3.6% S&P 500 Annual Total Return: 11.0% Annual Inflation: 3.8% Annual Real Price Change: 3.3% Annual Real Total Return: 7.0 % Buying the straight S&P 500 beats inflation by seven percent, on average, every year. You're welcome!
- justonepost 10y agoYou don't need prophet for that.
- etjossem 10y ago:)
- T-A 10y agoBuying the S&P 500 in 1950 and holding 67 years does. One sample tells you nothing about randomness. What if you buy in August 1929? What if you hold for a more realistic 20 or 30 years from peak earning years to retirement?
- etjossem 10y agoBought way back in August 1929: Annual Total Return: 9.1% Annual Real Total Return: 5.9% Bought in January 1987, held for a realistic 30 years: Annual Total Return: 9.8% Annual Real Total Return: 7.0% There's always going to be some deviation, but over any given multi-decade holding period, you will generally end up with a predictable 5-9% annualized (inflation-adjusted) return. That is more than zero. My point stands: long-term investment in the S&P 500 can be reasonably expected to gain value faster than inflation. If you're interested, here's a simulator that looks at historic market data. You'll note that even the lowest possible percentile of 30-year holding periods will still yield a 3.43% inflation-adjusted total return: https://dqydj.com/sp-500-historical-return-calculator-popout/ https://dqydj.com/sp-500-historical-return-calculator-popout...
- ainiriand 10y agoSome people are making pretty penny for being so random.
- icebraining 10y agoThe same could be said about the lottery.
- matheweis 10y ago> So...How much will this do at forecasting stock prices? =) Probably quite poorly (due to stocks appearing "random" at scale), especially for indexes, which are a sum of their parts. On the other hand, this would probably be quite useful for things that have non-random trends (like the Global Energy Forecasting Competition: http://www.drhongtao.com/gefcom http://www.drhongtao.com/gefcom)
- syntaxing 10y agoIt would probably perform pretty poorly as other has suggested. This is mainly due to the fact that stock prices by itself is a pretty non-stationary dataset/measurement. Most of these probabilistic models are poorly equipped to make accurate predictions for non-stationary data since it's trends are seemingly similar to noise.
- curuinor 10y agoFaced with phenomena I view as self-affine, other students take an extremely different tack. Most economists, scientists and engineers from diverse fields begin by subdividing time into alternating periods of quiescence and activity. Examples are provided by the following contrasts: between turbulent flow and its laminar inserts, between error-prone periods in communication and error-free periods, and between periods of orderly and agitated ("quiet" and "turbulent") Stock Market activity. Such subdivisions must be natural to human thinking, since they are widely accepted with no obvious mutual consultation. Rene Descartes endorsed them by recommending that every difficulty be decomposed into parts to be handled separately. Such subdivisions were very successful in the past, but this does not guarantee their continuing success. Past investigations only tackled variability and randomness that are mild, hence, local. In every field where variability / randomness is wild, my view is that such subdivisions are powerless. They can only hide the important facts, and cannot provide understanding. My alternative is to move to the above-mentioned apparatus centered on scaling. -Mandelbrot, in the foreward to Multifractals and 1/f Noise. it's worth saying that Mandelbrot was apparently a large influence to E Fama, who proposed the efficient market hypothesis in the first place.
- cardosof 10y agoThat's very cool, congrats and thank you to the Facebook guys! A few days ago I was asked to do some forecasting with a daily revenue series for a client. Due to her business' nature the series was really tricky with weekdays and months/semesters having some specific effects on the data. I as many use Hyndman's forecast package, but I threw this data at prophet and it delivered a nice plot with the (correct) overall trend and seasonalities. Very cool and easy to do something.
- agounaris 10y agoHow different this framework is from statsmodels?
- adw 10y agoStatsmodels is a grab-bag of various statistical models from linear regression upwards. This is an opinionated library for (some relevant parts of) econometrics.
- jl6 10y agoI wonder what Sungard/FIS think of the name, which is the same as their commercial financial modelling/forecasting tool.
- vinw 10y agoFIS Prophet is targeted at actuaries, and really no-one else so I don't know if anyone will care. They have had the name a lot longer than Facebook though!
- schlarpc 10y agoModerately relevant short story: https://www.facebook.com/notes/robin-sloan/julie-rubicon/985697811525170 https://www.facebook.com/notes/robin-sloan/julie-rubicon/985...
- JoshTriplett 10y agoThat was the first thing I thought of when I saw the title.
- hnarayanan 10y agoIs there a way to extend these models to handle spatial variation (e.g. weather forecasting, property price estimation etc.) as well?
- rodionos 10y agoThis would be non-trivial. Consider this paper on marijuana usage where the researchers had to group statistics by adjacent counties in Oregon and Washington in order to control the tests. https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2841267 https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2841267
- hnarayanan 10y agoThank you for the pointer, will read the article. All my attempts thus far have pointed me to something called Gaussian Proceeses that I am still working through grokking.
- confounded 10y agoWorth noting Prophet is R/Python wrappers to some models with reasonable defaults, written in and fit by Stan, a probabilistic programming language, and Bayesian estimation framework. Stan is amazing in that you can fit pretty much any model you can describe in an equation (given enough time and compute, of course)! More on Stan here: http://mc-stan.org/ http://mc-stan.org/
- dragandj 10y ago... and if you like Clojure, you might try Bayadera, which has its own engine running the analysis on the GPU. http://github.com/uncomplicate/bayadera http://github.com/uncomplicate/bayadera
- bpicolo 10y agoReadme has neither useful docs, nor any link to docs. =/
- mej10 10y agoThis looks like it could be awesome but it has almost no information about what its purpose is or how to use it.
- dragandj 10y agoYou are right. The docs have been due to be written for many months now, and that is the main reason the library has not been released yet. On the other hand, the test folder contains many tests, among them full examples from many chapters from the book Doing Bayesian Dara Analysis, recommended above.
- mej10 10y agoOh I see. That's cool! I do want to check it out but will probably wait for its release. And I wait with bated breath.
- diab0lic 10y ago
- minimaxir 10y agoInteresting definition of "scale" in this context, as it does not imply "big data" like every other usage of the word scale in data science. The tool works on, and is optimized, for day-to-day, mundane data. See also the R vignette, which shows that the data is returned per-column which gives it a lot of flexibility if you only want certain values: https://cran.r-project.org/web/packages/prophet/vignettes/quick_start.html https://cran.r-project.org/web/packages/prophet/vignettes/qu...
- fagnerbrack 10y agoFacebook...
- pacifika 10y agoThe more facebook grows the more tools it aligns tooling with intelligence services.
- rodionos 10y agoI didn't know wikipedia page view counters are available for public usage. The wikipediatrend R package relies on http://stats.grok.se/ http://stats.grok.se/, which in turn relies on https://dumps.wikimedia.org/other/pagecounts-raw/ https://dumps.wikimedia.org/other/pagecounts-raw/ which has been deprecated. The new dump is located at https://dumps.wikimedia.org/other/pageviews/ https://dumps.wikimedia.org/other/pageviews/ Data is available in hourly intervals. * pageviews-20170227-050000 en Peyton_Manning 58 0 [edit] There is a wikipedia-hosted OSS viewer for these logs, e.g. Swedish crime stats: https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&range=latest-90&pages=Crime_in_Sweden https://tools.wmflabs.org/pageviews/?project=en.wikipedia.or...
- fpvracing 10y agoCool! I wonder what spiked the views for artificial intelligence on 10/11/2016? https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&start=2015-07-01&end=2017-01-31&pages=Artificial_intelligence https://tools.wmflabs.org/pageviews/?project=en.wikipedia.or...
- T-A 10y agoI think that spike peaks on October 12, which is when this was released: https://obamawhitehouse.archives.gov/blog/2016/10/12/administrations-report-future-artificial-intelligence https://obamawhitehouse.archives.gov/blog/2016/10/12/adminis...
- JoelSanchez 10y agohttps://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&start=2015-07-01&end=2017-01-31&pages=Clojure|PHP|JavaScript|Java_(programming_language)|Elixir_(programming_language)|Haskell_(programming_language)|C_(programming_language)|Ruby_(programming_language)|Python_(programming_language) https://tools.wmflabs.org/pageviews/?project=en.wikipedia.or... What's up with Java? (Set "logarithmic scale" to improve the visualization)
- 10y ago
- yoghurtio 10y agoWe at https://yoghurt.io/ https://yoghurt.io/ have been working towards similar forecasting solution. So far the feedback has been that automated solutions can also bring good results at a far lesser cost compared to hiring an expert analyst. Its a completely managed solution. No need to setup anything yourself.Just upload the data and predict next week's data, today itself. There is a free trial and if anyone here is looking for an extended trial, they can reach out to me.
- redindian75 10y agoyour website is very sparse on details - any examples/demos?
- yoghurtio 10y agoExample: Like you want to predict the app downloads of your website coming week. Just upload the data in time series format against the date and app downloads from last 30 weeks. It will return the next 7 days predicted app downloads along with the analytical confidence. It can predict any KPI like visitors, app downloads, conversion etc. Just signup and start predicting.
- ainiriand 10y agoYour website is not working for me. The upload never completes. Tried Chrome 56 and firefox 51.
- yoghurtio 10y agoCan you please try uploading XLS or XLSX format. Normally, it should show error message in this case.We are going to fix it soon. CSV and other formats support would be coming soon.
- Steeeve 10y agoThis actually looks incredibly useful and pretty simple to learn. Between this and Stan I think my free time for the next week is gone.
- nodesocket 10y agoAre there any startups/services where you pass it a series and it returns forecast models? That's something I'd be willing to pay for.
- yoghurtio 10y agoYou can try https://yoghurt.io/ https://yoghurt.io/. Its fully managed platform and no need to setup anything yourself. Example: Like you want to predict the app downloads of your website coming week. Just upload the data in time series format against the date and app downloads from last 30 weeks. It will return the next 7 days predicted app downloads along with the analytical confidence. It can predict any KPI like visitors, app downloads, conversion etc. Just signup and start predicting.
- nodesocket 10y agoIs it possible for example to send you monthly revenue numbers for my startup for the last two years (24 data points) and have yoghurt predict the next two years of monthly revenue?
- throwaway_374 10y agoIf the model is autoregressive you can only forecast N steps ahead. Any further forecasting will be based on these generated near-future forecasts. In English, no. See https://www.youtube.com/watch?v=tJ-O3hk1vRw#t=01h16m https://www.youtube.com/watch?v=tJ-O3hk1vRw#t=01h16m
- nodesocket 10y agoThanks for posting this talk by Jeffrey Yau. I am 9 minutes into it and can't stop watching. He explains things very easily and clearly.
- yoghurtio 10y agoIts very simple to use Yoghurt, just upload the data and rest it does automatically. 24 data points is less to make any accurate prediction. You need more data points. However, Yoghurt currently supports 1 week prediction only and very soon we will be adding prediction upto 1 Month and plus.
- anacleto 10y agoThis is so great! I've been using CasualImpact by Google [0] for months. This seems pretty straightforward. [0] https://google.github.io/CausalImpact/CausalImpact.html https://google.github.io/CausalImpact/CausalImpact.html
- recurser 10y agoVery cool. Could this be re-purposed for detecting anomalies/outliers in time series data?
- fgpwd 10y agoMy guess would be yes. I'm thinking this could be used to find out how effective a particular marketing campaign was. Just compare the forecast with actuals and the difference would be the number of sales/clicks you got from that campaign.
- techno_modus 10y ago>Could this be re-purposed for detecting anomalies/outliers in time series data? If you define anomaly as something unexpected then yes. In this case, if the reality differs significantly from the forecast (=expectation) then it is an anomaly (according to our definition). In numeric univariate case, there could be positive anomalies where you get more than expected, and negative anomaly where you get less than expected.
- poppingtonic 10y agoThis is very interesting. Forecasters who participate in the Good Judgment Project, such as myself, will find this useful.
- techno_modus 10y agoIt seems that they have developed a model for only univariate forecasts and only numeric regular time series which is a classical use case in statistics. Yet, most data sources have many dimensions (for example, energy consumption, temperature, humidity etc.) as well as categorical data like current state (On, Off). The situation is even more difficult if the data is not a regular time series but is more like asynchronous event stream. It would be interesting to find a good forecasting model for some of these use cases. In particular, it is interesting if this Prophet model can be generalized and applied to multivariate data.
- unoti 10y ago> most data sources have many dimensions (for example, energy consumption, temperature, humidity etc.) as well as categorical data like current state (On, Off). The situation is even more difficult if the data is not a regular time series but is more like asynchronous event stream. It would be interesting to find a good forecasting model for some of these use cases. I'm guessing you already know about this based on the way you described the situation, but the Hyndman Forecasting book [1] discusses various models at length for doing multivariate forecasting models. It's loaded with code and samples in R. 1. https://www.otexts.org/fpp https://www.otexts.org/fpp
- nickfzx 10y agoThis looks amazing, congratulations. We're planning to add forecasting to our SaaS analytics product (https://chartmogul.com https://chartmogul.com) later this year, I'm going to look and see if we can use this in our product now.
- tommynicholas 10y agoI was trying to sort out whether adding this to an existing charting/analytics product makes sense but it looks like you've checked it out and think it does. I couldn't tell only because it seems to be built to do the charting/plotting itself, but I guess you can just use the data/API to get the forecasts then plot them yourself yes? I may do a test implementation into Airbnb Superset actually to see how it flies.
- ayayecocojambo 10y agoCan we use other features (like temperatue?), or it has to be only time-based?
- alexpetralia 10y agoSlightly inconvenient that the main image <figure> needs to be replaced by an <img> tag just to have the image appear in print outs.
- dmichulke 10y agoI have been working for a few years on a similar project using evolutionary algorithms on top of other models (linear / ann). It works quite well (e.g., for equidistant energy demand / supply forecasts) but there's still lots of stuff to do. It's major benefit is that it figures out relationship to the target time series by itself, so you can just throw in all time series and see what comes out. Language is Clojure, 20kloc, incanter, encog. If anyone is interested in working for/with it, let me know. I currently develop a Rest Api for it and plan to release it as open source once the major code smells are dealt with.
- eternalban 10y ago/please ignore: Oracle & Prophet. Oracle sifts through signs but Prophet has a line to the larger picture. I suppose the next 'product' will be called Messiah to complete the picture.
- deleted 10y ago[deleted]
- hubot 10y agocan someone explain what's the meaning of this line > df['y'] = np.log(df['y'])
- slashcom 10y agodf is a dataframe, which is like a spreadsheet. This line takes the logarithm of the column named 'y' and updates it in place.
- hubot 10y agothanks. that part i can understand but why do that?
- llimllib 10y agoI have not read the code, but assuming df is a pandas dataframe, it sets the 'y' column to the log of what was previously the 'y' column. https://gist.github.com/llimllib/385230f38c3f9b70c3e46158e6029f2a https://gist.github.com/llimllib/385230f38c3f9b70c3e46158e60...
- zebrafish 10y agoSo.... I don't understand how this is better or worse than using forecast. You talk about having to choose the best algorithm but it seems like Prophet is just another algorithm to choose from. Is there some kind of built in grid-search or are you just stating that results from your AM have been more accurate than ARIMA?
- saosebastiao 10y agoThis is an interesting project, and in one of the areas where almost all businesses could do better. Anecdotally, there is a ton of money left on the table by established businesses that do it poorly, which also leaves lots of room for resume-padding technical experience. So anything that claims to improve the state of the art of automated forecasting is definitely worth watching. That being said this claim in point #1 baffles me: > Prophet makes it much more straightforward to create a reasonable, accurate forecast. The forecast package includes many different forecasting techniques (ARIMA, exponential smoothing, etc), each with their own strengths, weaknesses, and tuning parameters. We have found that choosing the wrong model or parameters can often yield poor results, and it is unlikely that even experienced analysts can choose the correct model and parameters efficiently given this array of choices. The forecast package contains an auto.arima function which does full parameter optimization using AIC which is just as hands free as is claimed of Prophet. I have been using it commercially and successfully for years now. Maybe prophet produces better models (I'll definitely take a look myself), but to claim that it's not possible to get good results without experience seems a bit disingenuous. As an aside, anybody interested in a great introductory book on time series forecasting should check out Rob Hyndman's book which is freely available online. https://www.otexts.org/fpp https://www.otexts.org/fpp
- dxbydt 10y ago> Anecdotally, there is a ton of money left on the table by established businesses... True. fwiw, I worked on the same project at Twitter 4 years back - the Facebook folks call it capacity planning at scale, we called it capacity utilization modeling. The goal was the same - there are all these "jobs" - 10s of 1000s of programs running on distributed clusters, hogging CPU, memory and disk. Can we look at a snapshot in time of the jobs usage, and then predict/forecast what the next quarter jobs usage would be ? If you get these forecasts right ( within reasonable error bounds ), the folks making purchasing decisions ( how many machines to lease for the next quarter for the datacenters) can save a bundle. From an engineering pov, every job would need to log it's p95 and p99 CPU usage, memory stats, disk stats...Since Twitter was running some 50k programs back then (2013ish) on these Mesos clusters, the underlying C++ API had hooks to obtain CPU and memory stats, even though the actual programs running were all coded up in Scala (mostly), or python/Ruby (bigger minority), or C/Java/R/perl ( smaller minority ). There's an interesting Quora discussion on why Mesos was in C++ while rest of Twitter is Scalaland...mostly because you can't do these sort of CPU/memory/disk profiling in the jvmland as well as you can in C++. OK, so you now have all these CPU stats. What do you do with them ? Before you get to that, you have the usual engineering hassles - how often should you obtain the CPU stats ? Where would you store them ? So at Twitter we got these stats every minute ( serious overkill :) and stored them in a monstrous JSON ( horrible idea given 50000 programs * number of minutes in day * all the different stats you were storing :)) So every day I'd get a gigantic 20gb JSON from infra, then I'd have to do the modeling. In those days, you couldn't find a single Scala JSON parser that would load up that gigantic JSON without choking. We tried them all. Finally we settled on GSON - Google's JSON parser written in Java, that handled these gigantic jsons with no hiccups. Before you get to the math, you would have to parse the JSON and build a data structure that would store these (x,t) tuples in memory. You had 50k programs, so each program would get a model, each model originated from a shitton of (x,t) tuples, the t being minutely and the fact that some of these programs had been running for years, meant you had very large datasets. The math was relatively straightforward...I used so called "LAD" - least absolute deviation from mean, as opposed to simple OLS, because least squares wasn't quite predictive for that use case. Building the LAD modeling thing in Scala was somewhat interesting...Most of the work was done by the commons math Apache libraries, I mostly had to ensure the edge cases wouldn't throw you off, because LAD admits multiple solutions to the same dataset - it's not like OLS where you give it a dataset and it finds a unique best fit line. Here you'd have many lines sitting in an array, depending on how long you let the Simplex solver run. Then came the problem of visualizing these 50,000 piecewise line models using javascript heh heh. The front end guys had a ball with the models I spit out. If someone's doing this from scratch these days, NNs would be your best bet. Regime changes are a big part of that.
- paulvs 10y agoFor a corporate credit analyst working at a bank, what are some good introduction material for getting into forecasting using tools like these? I see this being applicable to analysts when deciding on on a company's credit worthiness.
- zebrafish 10y agoThere are some models out there which could be used but i'm not sure that forecasting is actually what you would use. I would think if you're already assigning credit ratings, you can set that as your dependent variable and use things like company revenue, number of employees, age of company, etc. as your independent variables. You can use a number of different models to assess credit worthiness based on this data. Evaluate several to determine the most accurate.
- syntaxing 10y agoThe fact that Prophet follows the "sklearn model API" and that it's very well integrated with pandas makes it super appealing and usable!
- SmellTheGlove 10y agoFor us insurance/financial services folks, I would like to simply clarify that this is not the Sungard/FIS risk management platform that is also called Prophet! :D I got really excited for a second. Actually, I'm still pretty excited about this even if it was something else entirely.
- hn_username 10y agoThis is a nice piece of work - thanks for sharing with the community! Some feedback: it'd be nice to see you actually quantify how accurate Prophet's forecasts are on the landing page for the project. In the Wikipedia page view example, you go as far as showing a Prophet forecast, but it'd be nice to have you take it one step further and quantify its performance. Maybe withhold some of the data you use to fit the model and see how it performs on that out of sample data. It's nice that you show qualitatively that it captures seasonality, but you make bold claims about its accuracy and the data to back those claims up is conspicuously absent. Related, it might be worth benchmarking its performance against existing automated forecasting tools. I'll definitely be checking it out!
- Helmet 10y agojust wanted to point out to potential windows users - this will only run on python 3.5 due to dependencies (pystan only works on python 3.5 for windows)
- elwell 10y agoWhy do we need Prophet when we already have Temple OS (http://www.templeos.org/ http://www.templeos.org/)?
- monkeydust 10y agoVery cool, got loads of sensor data around my house over a years worth so curious to throw it at Prophet. Has anyone managed to get this working on windows with Juypter (Anaconda build) struggling with Pystan errors. Any guidance welcomed.