3 ms·

The worst are the organisations that don't have any reproducibility in their work. Models in production that are total black boxes. There's no way to determini

by malux85 11y ago

The worst are the organisations that don't have any reproducibility in their work. Models in production that are total black boxes. There's no way to deterministically recreate them. But I treated it as an opportunity --- was able to teach some engineering to the data science guys and they taught me some data science :)

binarymax 11y ago
It is surprising how often we take reproducibility for granted in the software world. With unit tests and continuous integration scenarios it becomes such a fundamental concept that it would be unheard of for developing products without them. Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess. Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor. It is an interesting time to be in software :)
- malux85 11y ago
  I agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general. I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips? I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
  - binarymax 11y ago
    If I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!

by malux85 11y ago

binarymax 11y ago

It is surprising how often we take reproducibility for granted in the software world. With unit tests and continuous integration scenarios it becomes such a fundamental concept that it would be unheard of for developing products without them. Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess. Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor. It is an interesting time to be in software :)

malux85 11y ago
I agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general. I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips? I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
- binarymax 11y ago
  If I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!