3 ms·
Definitely! There are some applications where traditional methods are just simply good enough. However, when they aren't, it can be incredibly frustrating. F
by mike210 6y ago
Definitely! There are some applications where traditional methods are just simply good enough. However, when they aren't, it can be incredibly frustrating. From our conversations with scientists, this kind of data (3D, histology, difficult tissues, new assays) is increasing in volume.
As for correctness, we've only done mAP scores and traditional accuracy metrics so far to compare with other algorithms, but we also have our own internal metrics and a test set we're building out in-house to cover many edge cases, many of which cover some of the things you are talking about. One thing we're always trying to be sensitive of is fairness. We want to make sure that we're not biasing the test to our algorithm, which would make us look better than we are.
- itamarst 6y agoI guess when I say correctness, I mean "how do I know it _continues_ to be correct on data we've never seen before". That's where metamorphic testing can be valuable, because it lets you at least find incorrectness on real-world data that hasn't been hand-tagged.
- mike210 6y agoAh, yes. We're even looking to use some generative models in order to even do variations based on data and then compare that we do similarly well between cases. I guess the point I was making was that we want to make sure we don't then use this generated or modified data in order to test other algorithms in the space and say we're better. Simply put, it would be unfair for us to make changes to perform better on a hurdle and then put other algorithms through those hurdles. But for internal use, it's definitely great!