4 ms·
Why is it hard to come up with tests that closely resemble real-world usecases where the models are being used?
by zkmon 3mo ago
Why is it hard to come up with tests that closely resemble real-world usecases where the models are being used?
- philipwhiuk 3mo agoWould it be any more applicable? We see the same problem in database benchmarking. The good thing about the deliberately-non-real world pelican case is that it gives a general impression of how much the model is improving because it's not likely that it's being specifically targeted at it, rather than a 'real world case' which might have been specifically optimised for.
- simonw 3mo agoIt's not that hard - there are plenty of those. They're mostly not very funny though.