4 ms·
It is surprising how often we take reproducibility for granted in the software world. With unit tests and continuous integration scenarios it becomes such a fu
by binarymax 11y ago
It is surprising how often we take reproducibility for granted in the software world. With unit tests and continuous integration scenarios it becomes such a fundamental concept that it would be unheard of for developing products without them.
Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess.
Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor.
It is an interesting time to be in software :)
- malux85 11y agoI agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general. I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips? I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
- binarymax 11y agoIf I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!