4 ms·
Something I'm starting to realize with all these AI APIs is how difficult it is to evaluate and compare the results. How do you know their algorithms are bette
by hbt 11y ago
Something I'm starting to realize with all these AI APIs is how difficult it is to evaluate and compare the results.
How do you know their algorithms are better than the competition? The marketing buzzwords are all there. "Powerful algorithms", "never seen anything comparable" etc.
The worst part is the algorithms might work well with certain data sets but fail miserably with others.
- vikiomega9 11y agoIf you had an internal API as opposed to using an external one you'd have some sort of A/B or multi-armed testing in place to check if the model you've deployed is performing as expected. At this point you don't think about the theoretical best you could perform at for the said model without computing some sort of complexity measure. I assumed one would have to use a similar automated process for external APIs. I don't see why they have to be different per se.