4 ms·

We uh treat models as code, but also have NFS shares setup for the storage and GitLab runner talking to a Slurm cluster to run the models. Results and cross va

We uh treat models as code, but also have NFS shares setup for the storage and GitLab runner talking to a Slurm cluster to run the models. Results and cross validation upload to GitLab. Main thing we haven’t built out yet are performance dashboards for showing improvement across commits, but with the GitLab APIs that’s a script away (currently we do it by hand)

sandGorgon 8y ago
Thanks for your reply. Actually the question was more around "how do you create your models and what do you mean treating them as code", "why slurm and not something like airflow" , "what is the test/performance setup - backtesting, smoke test" etc etc The Gitlab stuff is easier to understand.
- marmaduke 8y ago
  Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not taken too seriously tbh) > why slurm and not something like airflow because the HPC resources we have access to are built with Slurm, which is super fast, supports DAGs of jobs, schedules our jobs reliably and quickly. I don't really want the other stuff on the Airflow feature list to be honest.
  - sandGorgon 8y ago
    super interesting. thanks for sharing. >we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML (https://eng.uber.com/michelangelo-pyml/ https://eng.uber.com/michelangelo-pyml/). What are some of your learnings around jupyter -> production code. A lot of these are around conventions - "write a function called train(), fit(), test()". Is that the basis of your pipeline as well ?
    - marmaduke 8y ago
      It’s not so simple for our models (hierarchical Bayesian time series models, often nonlinear, which may not be typical): we spend a lot of time digging through the data itself, forward simulations of model, and refactoring/tweaking model structure. PyML (as described in the link you provided) doesn’t appear to support the first two parts, which are prerequisites to improving the model IMO. Usually when we are doing more of the train/fit/test cycle, there’s an argparse script to quickly try different parameter values succinctly (which is run and tracked by the above CI setup) I wouldn’t say we’re reinventing since a better solution isn’t very clear (though PyML et al look interesting) edit forward simulation isn't a frequent thing in posts on generic ML algorithms, so just as an example: suppose you run a model and see an oscillatory component along a temporal dimensions in your residual error, and you add a oscillatory component to your model, and rerun it but still see a residual with an oscillation. You can run a forward simulation of your model to see what frequency it's predicting and check against what's seen in the data, and fix it. This is a contrived example but when you have multiple competing priors or model components, this is an effective way to debug their behavior.
    - marmaduke 8y ago
      this article (https://towardsdatascience.com/uber-introduces-pyml-their-secret-weapon-for-rapid-machine-learning-development-c0f40009a617 https://towardsdatascience.com/uber-introduces-pyml-their-se...) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc. Maybe a missing detail is that our models are run-once, once results are QA'd, they are sent to relevant practitioner, so Uber's query-per-second stuff is irrelevant for us (for now), which I can see simplifies the deployment question enormously.
      - dsumenkovic 8y ago
        Hello, Community Advocate from GitLab here. I was reading through your comments and it's great to hear how you use GitLab for your setup. Thanks for sharing your story with the community and we'd love to hear more from you on how GitLab helps you.
ptd 8y ago
What do you think are the best dashboard options for showing improvements?
- marmaduke 8y ago
  I'd probably set up Grafana talking to Elasticsearch or PostGres, but I haven't thought too hard about it.

sandGorgon 8y ago

Thanks for your reply. Actually the question was more around "how do you create your models and what do you mean treating them as code", "why slurm and not something like airflow" , "what is the test/performance setup - backtesting, smoke test" etc etc The Gitlab stuff is easier to understand.

marmaduke 8y ago
Ah right, > how do you create your models and what do you mean treating them as code we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data, where we assess performance with LOO approximations for Bayesian models (notably PSIS) and some labeling from experts (which is not taken too seriously tbh) > why slurm and not something like airflow because the HPC resources we have access to are built with Slurm, which is super fast, supports DAGs of jobs, schedules our jobs reliably and quickly. I don't really want the other stuff on the Airflow feature list to be honest.
- sandGorgon 8y ago
  super interesting. thanks for sharing. >we start with local Jupyter notebooks, and refactor bits of code into modules that get tested, which for our models mainly means recovering parameters from simulations, and then test them on real data This is the part that everyone seems reinventing. Have you looked at PyML (https://eng.uber.com/michelangelo-pyml/ https://eng.uber.com/michelangelo-pyml/). What are some of your learnings around jupyter -> production code. A lot of these are around conventions - "write a function called train(), fit(), test()". Is that the basis of your pipeline as well ?
  - marmaduke 8y ago
    It’s not so simple for our models (hierarchical Bayesian time series models, often nonlinear, which may not be typical): we spend a lot of time digging through the data itself, forward simulations of model, and refactoring/tweaking model structure. PyML (as described in the link you provided) doesn’t appear to support the first two parts, which are prerequisites to improving the model IMO. Usually when we are doing more of the train/fit/test cycle, there’s an argparse script to quickly try different parameter values succinctly (which is run and tracked by the above CI setup) I wouldn’t say we’re reinventing since a better solution isn’t very clear (though PyML et al look interesting) edit forward simulation isn't a frequent thing in posts on generic ML algorithms, so just as an example: suppose you run a model and see an oscillatory component along a temporal dimensions in your residual error, and you add a oscillatory component to your model, and rerun it but still see a residual with an oscillation. You can run a forward simulation of your model to see what frequency it's predicting and check against what's seen in the data, and fix it. This is a contrived example but when you have multiple competing priors or model components, this is an effective way to debug their behavior.
  - marmaduke 8y ago
    this article (https://towardsdatascience.com/uber-introduces-pyml-their-secret-weapon-for-rapid-machine-learning-development-c0f40009a617 https://towardsdatascience.com/uber-introduces-pyml-their-se...) does a better job motivating PyML, or maybe I'm just more awake now. In any case, I see what you mean. The GitLab CI setup we have builds Docker images out of our models, and we use branch names to target datasets, so "production" usage is "just" creating a branch, watching it run, checking results, etc. Maybe a missing detail is that our models are run-once, once results are QA'd, they are sent to relevant practitioner, so Uber's query-per-second stuff is irrelevant for us (for now), which I can see simplifies the deployment question enormously.
    - dsumenkovic 8y ago
      Hello, Community Advocate from GitLab here. I was reading through your comments and it's great to hear how you use GitLab for your setup. Thanks for sharing your story with the community and we'd love to hear more from you on how GitLab helps you.

ptd 8y ago

What do you think are the best dashboard options for showing improvements?

marmaduke 8y ago
I'd probably set up Grafana talking to Elasticsearch or PostGres, but I haven't thought too hard about it.