5 ms·

Firstly, I can't believe you have enough instances to resist the HN hug of death, with so many people presumably running tests. So that is impressive. Secondly

Firstly, I can't believe you have enough instances to resist the HN hug of death, with so many people presumably running tests. So that is impressive. Secondly, I ran the train -> test cycle and I didn't see any error metrics. Is the idea that if we were spinning up our own we would be outputting these ourselves? Or would we have trained up the model somewhere else and we would transfer it to SLAI to do a final test and then package it?

llom2600 5y ago
We tried to keep our testing workflow as flexible as possible. There's a couple of use-cases that we wanted to allow: - User is working with a pre-trained model that already went through extensive testing during training. In this case our test utilities are useful as e2e tests. Once you integrate the model into your handler, you can specify a bunch of test cases to be sure your API is going to behave as expected (like a unit test). - User wants to train the model on our platform - they can add error metrics directly in their training script and prevent the model from being saved if any error metric exceeds a certain threshold. They can then additionally use the test.py script to run tests against the model + handler.