4 ms·

Personally I've been working on the same declarative approach over the past couple of years at my company. This year I changed things to make heavy use of scik

Personally I've been working on the same declarative approach over the past couple of years at my company. This year I changed things to make heavy use of scikit-learn's `make_column_transformer` and `ColumnTransformer` capabilities. It's nice to see other, (much more) reputable engineering organizations taking a similar approach and treating the construction of different predictive models as an exercise in configuration. Although in my solution I haven't looked at any DL models, and typically default to feeding everything through XGBoost and performing a grid search for the best hyperparameter config there. My product is basically focused entirely around taking a raw dataset and a configuration file and producing an analytic dataset off of which algorithms can be tested. I'd be really interested in hearing others' experiences with this type of stuff.

thanatropism 8y ago
I make little DSLs using TOML to further hide data wrangling and estimator pipelining "hydraulics" in the context of a specific class of models. So for example I'll have a section that says "data" and with variables "binned = ['var1', 'var2']" and likewise for log/power-transformed variables, and have some Python code turn that into column transformers. In other examples there's even more custom logic hidden (some stuff from my master's thesis that I'm not at liberty to discuss) so it's not just a matter of reusing scikit-learn. I use TOML for little languages that are really flexible configuration files, and YAML for situations where there may be multiple similarly-specified objects (because the hierarchical syntax in TOML is somewhat obscure).
- __blacksite__ 8y ago
  I've never seen TOML, but I really like it. In my pretty simple pipeline, I've just been using a JSON-like structure that I can easily import into Python: strategies = [ { 'name': 'column1', 'kind': 'categorical', 'strategy': 'ohe' }, { 'name': 'column2', 'kind': 'continuous', 'strategy': 'center' } ... ] It has worked very well thus far. I received a request from a stakeholder a few weeks back about building a new model using a slightly different target. I told him it'd take a couple of days at least (it was a very similar target), but I had it completed (at least from a re-engineering perspective) in about 5 minutes. I simply changed the target variable config and removed any leaky features after changing the target. I'm convinced that this approach, coupled with configurations for tree-based models like minimum samples per leaf and max depth, is the most efficient way of building predictive models. Those configurations specific to tree-based model software help to skirt things like the rule of five, etc., IMO.
  - w4nderlust 8y ago
    That experience you talked about, being able to really quickly come up with solutions starting from an already established and working configuration, was one of the main motivating factors behind the making of Ludwig too! Basically I had a model, and they asked me first to add an input feature, then to add another one, then to add an output feature. Every time I did it with generality in mind so that the following time they would ask me it would require less time to do it. In the end the final solution is that it just costed me to add a for in a YAML file. :)
  - thanatropism 8y ago
    That's basically my approach. TOML is chosen for readability and easy maintenance, as well for the ability to cut and paste some chunks of python defining constants etc. directly which makes the first steps much less bureaucratic. I'm looking into more general parsing that would allow me to define semi-verbose little languages. In my heart Stata is still the gold standard for rapid fire usability, even at the cost of idiosyncrasy; I'd like to further make the engineering of, say, REST APIs more and more code/language independent and more logically specified, since a lot of the new "data science" crowd coming from stats and applied maths can't code their way out of a Tequila Sunrise.

thanatropism 8y ago

I make little DSLs using TOML to further hide data wrangling and estimator pipelining "hydraulics" in the context of a specific class of models. So for example I'll have a section that says "data" and with variables "binned = ['var1', 'var2']" and likewise for log/power-transformed variables, and have some Python code turn that into column transformers. In other examples there's even more custom logic hidden (some stuff from my master's thesis that I'm not at liberty to discuss) so it's not just a matter of reusing scikit-learn. I use TOML for little languages that are really flexible configuration files, and YAML for situations where there may be multiple similarly-specified objects (because the hierarchical syntax in TOML is somewhat obscure).

__blacksite__ 8y ago
I've never seen TOML, but I really like it. In my pretty simple pipeline, I've just been using a JSON-like structure that I can easily import into Python: strategies = [ { 'name': 'column1', 'kind': 'categorical', 'strategy': 'ohe' }, { 'name': 'column2', 'kind': 'continuous', 'strategy': 'center' } ... ] It has worked very well thus far. I received a request from a stakeholder a few weeks back about building a new model using a slightly different target. I told him it'd take a couple of days at least (it was a very similar target), but I had it completed (at least from a re-engineering perspective) in about 5 minutes. I simply changed the target variable config and removed any leaky features after changing the target. I'm convinced that this approach, coupled with configurations for tree-based models like minimum samples per leaf and max depth, is the most efficient way of building predictive models. Those configurations specific to tree-based model software help to skirt things like the rule of five, etc., IMO.
- w4nderlust 8y ago
  That experience you talked about, being able to really quickly come up with solutions starting from an already established and working configuration, was one of the main motivating factors behind the making of Ludwig too! Basically I had a model, and they asked me first to add an input feature, then to add another one, then to add an output feature. Every time I did it with generality in mind so that the following time they would ask me it would require less time to do it. In the end the final solution is that it just costed me to add a for in a YAML file. :)
- thanatropism 8y ago
  That's basically my approach. TOML is chosen for readability and easy maintenance, as well for the ability to cut and paste some chunks of python defining constants etc. directly which makes the first steps much less bureaucratic. I'm looking into more general parsing that would allow me to define semi-verbose little languages. In my heart Stata is still the gold standard for rapid fire usability, even at the cost of idiosyncrasy; I'd like to further make the engineering of, say, REST APIs more and more code/language independent and more logically specified, since a lot of the new "data science" crowd coming from stats and applied maths can't code their way out of a Tequila Sunrise.