2 ms·
I remember 15 years ago making a neural net captcha OCR cracker (for DNS registrations). And after about 3 hours I have my accuracy, recall and confusion number
by candiodari 9y ago
I remember 15 years ago making a neural net captcha OCR cracker (for DNS registrations). And after about 3 hours I have my accuracy, recall and confusion numbers and I'm not really getting them any better. This was a captcha like from those old PHP libraries. Many colors (sometimes bright yellow on white GRMBL), varying background color, a few things on the background that were clearly not letters (like large circles).
Now in order to train this network I had to download 1000 or so captchas and solve them manually, so I had written a quick program that allows a person to solve captchas. So I figured "let's see how good it is", and used the program to test myself (I figured, worst case I have some more raw data).
Turns out the neural net was 2% better than I was. It actually scored 2% better on captcha solving than I did, and this was when I was trying really hard, pausing when there's doubt and giving it 5 seconds thought. I was a bit surprised that trying really hard or just going with my first guess didn't make more than a percent difference.
Then I got someone who had nothing to do with IT to solve 50 captchas and calculated ... precision was 8% below the neural net (also known as she'd do about 1 in 6 wrong, whereas I only had one in 10-12 or so. The neural only had one in 14-15 wrong. It was a pretty sizable difference).
And lastly I figured, this is not possible. I'm somehow messing this up. After all, I should have made that same 8%+ errors on the original set of 1000 captchas I downloaded, did the network really learn to read better than I can, while being fed close to 10% falsehoods* ? So I generated the set where it consistently errors. And ... Yep. It did. About 40% of those was just me doing the ground truth wrong. I fixed them, but it only improved the CNNs performance slightly.
* of course of those 10% wrong would still almost always have 4 out of 5 letters correct