If your neural net config is in a relatively standalone file, or you can mark it with a special comment block, you could have your test runner actually read the

If your neural net config is in a relatively standalone file, or you can mark it with a special comment block, you could have your test runner actually read the source file, regex it out, and concat the source block, date, current git SHA, and performance metrics into a "neural_runs.txt" file. If something else about your data pipeline is changing as well, e.g. filter settings on your image preprocessing, you can throw that in there too. If you check this in, then every commit will include the diff of everything you tried to get there alongside the final source file, and additionally that file will serve as a single historical record for everything you tried for all time. Asking yourself a month later "did I ever try cross entropy" is as easy as grepping the file. Heck, you could insert into a database as well if you really wanted to, and visualize your performance changes over time a la http://isfiberreadyyet.com/ http://isfiberreadyyet.com/ . Sky's the limit.