4 ms·

Probably because 99% accuracy isn't good enough.

by th0ma5 7y ago

scrollaway 7y ago
What accuracy do you think humans have? If I give you 1000 handwritten numbers, do you think you'll make less than 10 mistakes?
- deepnet 7y ago
  They will likely assign the same tasks to multiple people and use an ensemble of human consensus.
- Arbalest 7y ago
  Run both OCR and human in parallel and double check anomalies
  - saagarjha 7y ago
    I would expect humans to make similar errors.
    - Arnt 7y ago
      AIUI the classic procedure for tasks like this is/was: Hide some graduate students. Tell each to transcribe, and to mark the difficult spots. Give each page to two students. Next, have someone else process the page pairs and resolve conflicts and the marked trouble spots. As long as one of the students notices that a particular spot is difficult to read, the error is discovered and can be handled by someone who isn't numb from transcribing pages of simple numbers.
- astrobe_ 7y ago
  The forms are in open access, anyone can fill one form, so they surely did consider variations in accuracy (if not plain sabotage). They are probably submitting the same data to multiple people and the cross-checking the submissions. Also, the totals help to filter out submissions with mistakes. One can also add bound checking for individual values (misplaced decimal point, extra digits etc.). I would love to see how they do it, but they'll probably keep it confidential because trolls and other vandals.
  - reportingsjr 7y ago
    This is exactly how zooniverse works. The first thing you do is go through a few known pieces of data to verify you know how to identify things correctly (more important for galaxy types, etc). As you start labeling data everything has to reach a consensus. I'm not sure exactly how it works, but it does have multiple people verify each piece of data.
labster 7y ago
Haha 99% accuracy in weather data would be an amazing improvement. Noisy data and data holes are all we got to work with. Humidity sensors have a memory, storing moist/dry air inside the ceramic.
adrianN 7y ago
Checking the results of OCR is a lot faster than manually doing the OCR, no?
dx034 7y ago
I've had a look at a few of those and I doubt you'd get 99% accuracy. A lot of the footnotes especially are very hard to decipher. They're also 50 years old, handwriting has changed over time. I'm not sure how well models are trained to read old handwriting, I know that I struggle with it. For humans, much can be deciphered through context. That's much more complex to do with OCR.
- londons_explore 7y ago
  The National Archives in the UK has a research project to try to read > 200 year old handwriting with OCR reliably. They use various tricks like training per-century or per author models which gives a big performance improvement. They also have ways to handle symbols that no longer exist (not in unicode). Give me a shout if anyone here wants to collaborate.
  - GijsjanB 7y ago
    I work on the visualization and publishing side of this kind of data, but I do know our HTR works even better if we combine authors and centuries into one model, instead of separating. Did you try and compare results?
hardlianotion 7y ago
I doubt you would get 99% accuracy with humans, either.
deleted 7y ago
[deleted]
pvaldes 7y ago
Outliers would be easy to spot. Maybe they should split in lines each figure, make a coordinates map of the entire figure or so, make a copy and applying a bulk search with a machine for crossing the map and annotate as many undoubtely identifiable numbers as possible. Then paint it in a different color easy to filter or hide it and remember its position. And then add humans to focuse only in the remaining dificult cases and outliers armed with a reference sample chart. This way an human would need to focus its eyes in 20 characters/image (instead 60 or 100). They would accumulate more completed figures faster and obtain a bigger sense of reward. If the human can't recognize the number, could put a ? in the chain and move on. Thus the machine would annotate for example 1_3, 4_67_, _8 and the human would write: 5?01 for the same line. Just an idea, don't know if designing that is easy-peasy or really defiant but the number of possible values is limited in any case, so should be possible to train a machine to recognize some single characters or even entire numbers. Specially if written for the same people.

Probably because 99% accuracy isn't good enough.

by th0ma5 7y ago

Probably because 99% accuracy isn't good enough.

What accuracy do you think humans have? If I give you 1000 handwritten numbers, do you think you'll make less than 10 mistakes?

deepnet 7y ago
They will likely assign the same tasks to multiple people and use an ensemble of human consensus.
Arbalest 7y ago
Run both OCR and human in parallel and double check anomalies
- saagarjha 7y ago
  I would expect humans to make similar errors.
  - Arnt 7y ago
    AIUI the classic procedure for tasks like this is/was: Hide some graduate students. Tell each to transcribe, and to mark the difficult spots. Give each page to two students. Next, have someone else process the page pairs and resolve conflicts and the marked trouble spots. As long as one of the students notices that a particular spot is difficult to read, the error is discovered and can be handled by someone who isn't numb from transcribing pages of simple numbers.
astrobe_ 7y ago
The forms are in open access, anyone can fill one form, so they surely did consider variations in accuracy (if not plain sabotage). They are probably submitting the same data to multiple people and the cross-checking the submissions. Also, the totals help to filter out submissions with mistakes. One can also add bound checking for individual values (misplaced decimal point, extra digits etc.). I would love to see how they do it, but they'll probably keep it confidential because trolls and other vandals.
- reportingsjr 7y ago
  This is exactly how zooniverse works. The first thing you do is go through a few known pieces of data to verify you know how to identify things correctly (more important for galaxy types, etc). As you start labeling data everything has to reach a consensus. I'm not sure exactly how it works, but it does have multiple people verify each piece of data.

labster 7y ago

Haha 99% accuracy in weather data would be an amazing improvement. Noisy data and data holes are all we got to work with. Humidity sensors have a memory, storing moist/dry air inside the ceramic.

adrianN 7y ago

Checking the results of OCR is a lot faster than manually doing the OCR, no?

dx034 7y ago

I've had a look at a few of those and I doubt you'd get 99% accuracy. A lot of the footnotes especially are very hard to decipher. They're also 50 years old, handwriting has changed over time. I'm not sure how well models are trained to read old handwriting, I know that I struggle with it. For humans, much can be deciphered through context. That's much more complex to do with OCR.

londons_explore 7y ago
The National Archives in the UK has a research project to try to read > 200 year old handwriting with OCR reliably. They use various tricks like training per-century or per author models which gives a big performance improvement. They also have ways to handle symbols that no longer exist (not in unicode). Give me a shout if anyone here wants to collaborate.
- GijsjanB 7y ago
  I work on the visualization and publishing side of this kind of data, but I do know our HTR works even better if we combine authors and centuries into one model, instead of separating. Did you try and compare results?

hardlianotion 7y ago

I doubt you would get 99% accuracy with humans, either.

deleted 7y ago

[deleted]

pvaldes 7y ago

Outliers would be easy to spot. Maybe they should split in lines each figure, make a coordinates map of the entire figure or so, make a copy and applying a bulk search with a machine for crossing the map and annotate as many undoubtely identifiable numbers as possible. Then paint it in a different color easy to filter or hide it and remember its position. And then add humans to focuse only in the remaining dificult cases and outliers armed with a reference sample chart. This way an human would need to focus its eyes in 20 characters/image (instead 60 or 100). They would accumulate more completed figures faster and obtain a bigger sense of reward. If the human can't recognize the number, could put a ? in the chain and move on. Thus the machine would annotate for example 1_3, 4_67_, _8 and the human would write: 5?01 for the same line. Just an idea, don't know if designing that is easy-peasy or really defiant but the number of possible values is limited in any case, so should be possible to train a machine to recognize some single characters or even entire numbers. Specially if written for the same people.