Not just for neural nets - balancing experimentation against building reusable tools is probably the biggest logistical challenge in scientific programming in g

Not just for neural nets - balancing experimentation against building reusable tools is probably the biggest logistical challenge in scientific programming in general. I've converged to a workflow where I maintain a library with a main project pipeline and reusable tools for the project, and do all scripting with jupyter (all notebooks version controlled). I've found that machine learning projects can be pretty effectively parametrized with config dicts for data, training and the model. Each type of config gets it's own pipelined method that does all of the library calls - pipeline_batch_gen, pipeline_train, pipeline_build_model. Example of a poorly organized config from a project: model_config = { 'optimizer': optimizer, 'clip_grad': clip_grads, 'name': model_name, 'residual': residual, 'n_conv_filters': n_conv_filters, 'n_output_hus': n_output_hus, 'activation': activation, 'batch_norm': batch_norm, 'output_bn': output_bn, 'generation': generation, 'data_spec': { 'uniform_frac': uniform_frac, 'include_augment': True, 'batch_size': batch_size, 'bulk_chunk_size' : bulk_chunk_size, 'max_bulk_chunk_size': max_bulk_chunk_size, 'loss_weighter': loss_weight }, 'train_spec': { 'early_stopping_patience': early_stopping_patience, 'lr_plateau_patience': lr_plateau_patience, 'learning_rate': init_lr, 'clip_grads': clip_grads, 'partial_weight': partial_weight } } I've wanted to give Sacred a try https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred - looks promising but haven't tried yet so can't comment. I still tend to keep track of model performance by hand though. But I have always have the notebooks I can go back to for reference. This is something sacred could help a lot with. Another very non-trivial aspect of this kind of work is the compute/storage infrastructure you need to scale beyond a single workstation. We have a nice system here where $HOME lives on NFS and gets mounted when you log into any machine on the network - I can hardcode paths in my code and count on every worker having the same filesystem. I can't imagine how we would do distributed jobs without NFS. That's not a very realistic solution for homegamers though - you need a very fast network and expensive commodity hardware. And sys admins. Does anyone have a solution for that half of the problem? I've seen a number of merkle-tree based data version control solutions recently...