3 ms·

The catch it the “and looking at our data” part. ML is basically a collection of thorough ways to look at your data, understand the patterns and infer what that

The catch it the “and looking at our data” part. ML is basically a collection of thorough ways to look at your data, understand the patterns and infer what that means for the future. In your example, you should absolutely start by cleaning your data up and run some basic SQL aggregations and plotting volume over time. So you look that that and notice (1) volume is increasing over time, (2) some holidays bump a few days ahead but drive very low volume day of (3) weekends are higher, but the effect isn’t pronounced the whole year and (4) summer is better for you than winter except for the Christmas season. Now: it’s two days before Halloween, what’s our anticipated sales volume? If you baked all those observations into an ARIMA model, it’s trivial to crank out a forecast with quantifiable accuracy. If you just have lines on a graph, it’s hard to pin down all the independent effects and recombine them for arbitrary scenarios.

philliphaydon 9y ago
Haha, I wanted to have a bar graph which were by day of week: Bar 1: Last years deliveries. Bar 2: Predicted this years deliveries (based on % of increase from other markets if no previous years, or percentage of growth from last 2 years) Bar 3: Actual deliveries. Then another graph: Bar 1: Average processing time from previous year. Bar 2: Average processing time for current year. I'm terrible with graphs and such tho, I can get all the data, I really suck at displaying that data tho. (and also show the holidays in that region, India has a lot!)
- edraferi 9y ago
  You're absolutely on the right track. There are well-established statistics tools that formalize your intuition and carry it forward to its logical conclusion. > "by day of week" This means you have a time series with daily resolution (one observation per day) and you expect Weekly Seasonality to matter. Model this as 7-period lag in your daily series. > based on % of increase from other markets if no previous years There are multiple markets, each with their own time series? Congratulations, you have Panel Data [0]. Do some regions have similar trends? Need to account for that correlation, maybe try a Mixed Model [1] > percentage of growth from last 2 years So there's a trend component (constant growth over time). Easy enough to fit the I term of an ARIMA model for this. You'll need to do some custom work to integrate this with your cross-regional correlations though. > Average processing time You'll want to model this at least as well as you're modelling the demand. That means seasonal effects, correlations between factories / warehouses, etc. PS any time you're dealing with a two-day weekend you'd better use at least 3 years of data in case major holidays happened to fall on a Saturday and Sunday, blindsiding you when it shows up on Monday this year. > show the holidays You'll definitely need to put together a calendar of major holidays for each region. These models will calculate the effect for each holiday. Specifically, the effect of the holiday AFTER accounting for the day of week, overall growth, time of year (season), and region. You might even get nifty charts like [2] =============== Anyway that's the basics. You can take graduate math courses in just this kind of modelling. Easier - you can contract a decent statistician for a couple weeks to build the model for you. They'll be delighted that you can produce SQL queries with the relevant data, and their models will help you get a lot more value out of those queries. =============== Resources 4 U Generic time series reference: [0] https://en.wikipedia.org/wiki/Panel_data https://en.wikipedia.org/wiki/Panel_data [1] https://en.wikipedia.org/wiki/Mixed_model https://en.wikipedia.org/wiki/Mixed_model [2] https://assets.digitalocean.com/articles/eng_python/prophet/fig-7.png https://assets.digitalocean.com/articles/eng_python/prophet/... [3] https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average https://en.wikipedia.org/wiki/Autoregressive_integrated_movi... R tutorials: [4] https://www.datascience.com/blog/introduction-to-forecasting-with-arima-in-r-learn-data-science-tutorials https://www.datascience.com/blog/introduction-to-forecasting... [5] https://cran.r-project.org/web/views/TimeSeries.html https://cran.r-project.org/web/views/TimeSeries.html (Particularly the "Forecasting and Univariate Modeling" section for your problem) Python tutorials: [6] https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/ https://machinelearningmastery.com/arima-for-time-series-for... [7] http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA-model-in-python/ http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA... [8] https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-prophet-in-python-3 https://www.digitalocean.com/community/tutorials/a-guide-to-...

philliphaydon 9y ago

Haha, I wanted to have a bar graph which were by day of week: Bar 1: Last years deliveries. Bar 2: Predicted this years deliveries (based on % of increase from other markets if no previous years, or percentage of growth from last 2 years) Bar 3: Actual deliveries. Then another graph: Bar 1: Average processing time from previous year. Bar 2: Average processing time for current year. I'm terrible with graphs and such tho, I can get all the data, I really suck at displaying that data tho. (and also show the holidays in that region, India has a lot!)

edraferi 9y ago
You're absolutely on the right track. There are well-established statistics tools that formalize your intuition and carry it forward to its logical conclusion. > "by day of week" This means you have a time series with daily resolution (one observation per day) and you expect Weekly Seasonality to matter. Model this as 7-period lag in your daily series. > based on % of increase from other markets if no previous years There are multiple markets, each with their own time series? Congratulations, you have Panel Data [0]. Do some regions have similar trends? Need to account for that correlation, maybe try a Mixed Model [1] > percentage of growth from last 2 years So there's a trend component (constant growth over time). Easy enough to fit the I term of an ARIMA model for this. You'll need to do some custom work to integrate this with your cross-regional correlations though. > Average processing time You'll want to model this at least as well as you're modelling the demand. That means seasonal effects, correlations between factories / warehouses, etc. PS any time you're dealing with a two-day weekend you'd better use at least 3 years of data in case major holidays happened to fall on a Saturday and Sunday, blindsiding you when it shows up on Monday this year. > show the holidays You'll definitely need to put together a calendar of major holidays for each region. These models will calculate the effect for each holiday. Specifically, the effect of the holiday AFTER accounting for the day of week, overall growth, time of year (season), and region. You might even get nifty charts like [2] =============== Anyway that's the basics. You can take graduate math courses in just this kind of modelling. Easier - you can contract a decent statistician for a couple weeks to build the model for you. They'll be delighted that you can produce SQL queries with the relevant data, and their models will help you get a lot more value out of those queries. =============== Resources 4 U Generic time series reference: [0] https://en.wikipedia.org/wiki/Panel_data https://en.wikipedia.org/wiki/Panel_data [1] https://en.wikipedia.org/wiki/Mixed_model https://en.wikipedia.org/wiki/Mixed_model [2] https://assets.digitalocean.com/articles/eng_python/prophet/fig-7.png https://assets.digitalocean.com/articles/eng_python/prophet/... [3] https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average https://en.wikipedia.org/wiki/Autoregressive_integrated_movi... R tutorials: [4] https://www.datascience.com/blog/introduction-to-forecasting-with-arima-in-r-learn-data-science-tutorials https://www.datascience.com/blog/introduction-to-forecasting... [5] https://cran.r-project.org/web/views/TimeSeries.html https://cran.r-project.org/web/views/TimeSeries.html (Particularly the "Forecasting and Univariate Modeling" section for your problem) Python tutorials: [6] https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/ https://machinelearningmastery.com/arima-for-time-series-for... [7] http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA-model-in-python/ http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA... [8] https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-prophet-in-python-3 https://www.digitalocean.com/community/tutorials/a-guide-to-...