3 ms·

Test-First Machine Learning

30 points by frag 6y ago

MrLeap 6y ago
This is fun. I managed a data entry project at a previous employer that was kind of like this. We had non technical subject matter experts that I sneakily got to write several thousand unit tests. We had some quantitative and qualitative data we wanted annotated on a substantial dataset of scanned documents. I built a tool that let them type natural language descriptors for everything. I then converted that to javascript unit tests by way of some regex and coffeescript. It was possible to bootstrap classifiers that way! Seems like kind of old tech in an era of unsupervised megaML, but I've not been using that part of my resume for a year or so.
cik2e 6y ago
> Testing model implementation details such as statistical and mathematical soundness are not part of the TFML strategy. Such details should be tested separately and are specific to the family of the model under consideration. For anyone who clicked this thinking this would be related to model validation, it ain't. The article is promoting test-driven development for systems that have an ML component.
nervousDev 6y ago
I think you missed TDD as in Test Driven Development. The article mentions nothing about specific methodologies regarding testing in a ML context. Maybe on the next article? :)
- qayxc 6y ago
  I understood the it more as a call for a test-first methodology than detailed instructions. A blog series on the details would be great, though!
tadeegan 6y ago
Testing model code tends to be very difficult unless you design your training loop with lots of abstractions and dependency injection which makes the code less explicit and difficult to understand. For example, try to look at the Tensorflow Estimator framework. Absolutely awful to use but is well tested.
- qayxc 6y ago
  The idea is not to test model code at all - the blog post explicitly mentions that this as an anti-pattern. The tests aren't concerned with the technical details at all - they should test whether the models work, not how.
  - frag 6y ago
    Exactly. The tests are concerned with integrating the ML component into the broader picture of requirements and engineering. Engineers are not really interested in why the statistics of a model is failing. And should never be (that's why testing model internals in TFML is an anti-pattern)

30 points by frag 6y ago

This is fun. I managed a data entry project at a previous employer that was kind of like this. We had non technical subject matter experts that I sneakily got to write several thousand unit tests. We had some quantitative and qualitative data we wanted annotated on a substantial dataset of scanned documents. I built a tool that let them type natural language descriptors for everything. I then converted that to javascript unit tests by way of some regex and coffeescript. It was possible to bootstrap classifiers that way! Seems like kind of old tech in an era of unsupervised megaML, but I've not been using that part of my resume for a year or so.

cik2e 6y ago

> Testing model implementation details such as statistical and mathematical soundness are not part of the TFML strategy. Such details should be tested separately and are specific to the family of the model under consideration. For anyone who clicked this thinking this would be related to model validation, it ain't. The article is promoting test-driven development for systems that have an ML component.

nervousDev 6y ago

I think you missed TDD as in Test Driven Development. The article mentions nothing about specific methodologies regarding testing in a ML context. Maybe on the next article? :)

qayxc 6y ago
I understood the it more as a call for a test-first methodology than detailed instructions. A blog series on the details would be great, though!

tadeegan 6y ago

Testing model code tends to be very difficult unless you design your training loop with lots of abstractions and dependency injection which makes the code less explicit and difficult to understand. For example, try to look at the Tensorflow Estimator framework. Absolutely awful to use but is well tested.

qayxc 6y ago
The idea is not to test model code at all - the blog post explicitly mentions that this as an anti-pattern. The tests aren't concerned with the technical details at all - they should test whether the models work, not how.
- frag 6y ago
  Exactly. The tests are concerned with integrating the ML component into the broader picture of requirements and engineering. Engineers are not really interested in why the statistics of a model is failing. And should never be (that's why testing model internals in TFML is an anti-pattern)