3 ms·
The only reliable final test will be a black box test suite that takes your model, executes it in a sealed environment and gives you a grade back, potentially w
by m_ke 2y ago
The only reliable final test will be a black box test suite that takes your model, executes it in a sealed environment and gives you a grade back, potentially with a performance break down by subject.
No telling companies what the questions look like, what the output format is, what topics are covered, so that there’s no room to make up synthetic data to interpolate from.
- andrewflnr 2y agoA grade is mostly meaningless if you don't know how it was calculated, so no one would "rely" on it. If nothing else, you need to know the grading methodology after the test. It's the same problem with cheating students. Once the test questions are known, they have a very short lifespan before cheaters can make them worthless. Tests have to be refreshed.
- m_ke 2y agoBy grade I mean a score of how many of the tasks were completed successfully. K/N or as a percentage.
- andrewflnr 2y agoIf I don't know what the tasks were, that's almost exactly as useless to me as a unitless number would be. For starters, are they all of equal difficulty? Are you sure? Do you expect to be able to convince me of that without letting me see them?