3 ms·

How accurate is the counting? For examples like the case study, I presume your customers will want very close to 100% accuracy, but many other use cases will on

How accurate is the counting? For examples like the case study, I presume your customers will want very close to 100% accuracy, but many other use cases will only need rough numbers. How do you calibrate / verify your app?

morsanu 5y ago
For most of our clients, we get to 100% accuracy. For some more difficult scenarios, a lower accuracy is ok for estimations (e.g. estimating the crop in agriculture). It's also super easy to correct mistakes or to add items that are not visible in the image.
- BorisTheBrave 5y ago
  You don't mean 100% accuracy, that means mistakes are impossible. Perhaps your error rate is very rare, but it's a bit concerning you don't attempt to quantify it.
  - leucineleprec0n 5y ago
    Suppose it depends on which part of the chain he attributes error, plausible lighting or poor positioning - anything that is an employee duty - does throw it off and then they scrub it. But yeah. Still.
  - jmmcd 5y ago
    I disagree. If we consider a single customer with a particular application, we probably imagine an iterative process like this: the customer supplies some sample images, the system gets a few wrong, they add special cases, fine-tune the hyperparameters, whatever, and after all this they literally get 100% on the customer's holdout data. If that happens for several customers then the GP statement is justified (no other number is possible for these customers). Possibly you're thinking of a single error rate across all customers? For other customers, as stated, it's not 100%. But taking an average across multiple customers is not meaningful when some are counting pearls and some are counting crops.
matsemann 5y ago
You have to take into account the error rate of human counting when making the comparison. I can also imagine the new process have other benefits. Like before, if a customer complained of getting fewer than ordered pearls, they would probably have to acknowledge that as a counting error on their side but not knowing. Now they have a picture/data to refer back to. At least that's how it is where I work; the automation leads to reduced errors, and when they happen the amount of data makes it easier to track where it went wrong.
dintech 5y ago
One thing to consider is that humans don't have 100% accuracy for large numbers of things.
klyrs 5y ago
The article shows a photo with the pearls highlighted with circles. I'm generally skeptical of AI, but this system produces a certificate that a human can verify much faster than doing the work themselves. IMO, that's the right way to do it. You can trust the system 99% of the time, and check a random 1% by eye every day. Or, you can keep a human fully in the loop and still save time (though, if the error rate is too low, humans will get bored)

morsanu 5y ago

For most of our clients, we get to 100% accuracy. For some more difficult scenarios, a lower accuracy is ok for estimations (e.g. estimating the crop in agriculture). It's also super easy to correct mistakes or to add items that are not visible in the image.

BorisTheBrave 5y ago
You don't mean 100% accuracy, that means mistakes are impossible. Perhaps your error rate is very rare, but it's a bit concerning you don't attempt to quantify it.
- leucineleprec0n 5y ago
  Suppose it depends on which part of the chain he attributes error, plausible lighting or poor positioning - anything that is an employee duty - does throw it off and then they scrub it. But yeah. Still.
- jmmcd 5y ago
  I disagree. If we consider a single customer with a particular application, we probably imagine an iterative process like this: the customer supplies some sample images, the system gets a few wrong, they add special cases, fine-tune the hyperparameters, whatever, and after all this they literally get 100% on the customer's holdout data. If that happens for several customers then the GP statement is justified (no other number is possible for these customers). Possibly you're thinking of a single error rate across all customers? For other customers, as stated, it's not 100%. But taking an average across multiple customers is not meaningful when some are counting pearls and some are counting crops.

matsemann 5y ago

You have to take into account the error rate of human counting when making the comparison. I can also imagine the new process have other benefits. Like before, if a customer complained of getting fewer than ordered pearls, they would probably have to acknowledge that as a counting error on their side but not knowing. Now they have a picture/data to refer back to. At least that's how it is where I work; the automation leads to reduced errors, and when they happen the amount of data makes it easier to track where it went wrong.

dintech 5y ago

One thing to consider is that humans don't have 100% accuracy for large numbers of things.

klyrs 5y ago

The article shows a photo with the pearls highlighted with circles. I'm generally skeptical of AI, but this system produces a certificate that a human can verify much faster than doing the work themselves. IMO, that's the right way to do it. You can trust the system 99% of the time, and check a random 1% by eye every day. Or, you can keep a human fully in the loop and still save time (though, if the error rate is too low, humans will get bored)