6 ms·

UK scientists ask for help in transcribing 200 years of rainfall data

102 points by billfruit 7y ago

lalaland1125 7y ago
Why hasn't this been automated? Digit recognition technology is pretty good and it seems like the forms are pretty standardized.
- ant6n 7y ago
  Is there a technology that would help ocr-ing standardized forms? I’ve got a couple thousand pages of historical train schedules that i would like to digitize (tabulated, printed data, including symbols/icons), but I’m not sure how to automatically recognize the structured data.
  - GijsjanB 7y ago
    I don't know about out-of-the-box ready software. Depending on how much time you have you'd either train your own model with your own ground truth or you can use an open source model. Libraries and archives sometimes make theirs available (I know the Dutch National Archives do). You can train the model and/or HTR the data with software like Transcribus or Kraken (open source).
  - jasoncabot 7y ago
    (disclaimer MS employee but nothing to do with this product) You could check out the Azure Forms Recognizer. It's designed to do exactly this and make even training a custom model pretty easy. It's also pretty cheap (free up to 500 pages) so you can always do a quick experiment and see if it does what you need it to https://docs.microsoft.com/en-us/azure/cognitive-services/form-recognizer/overview https://docs.microsoft.com/en-us/azure/cognitive-services/fo...
  - martyvis 7y ago
    I remember getting some reference software and methods from NIST maybe 20 years ago on handwriting recognition. Never put it into practice, but maybe they have some updated software available
  - froindt 7y ago
    I'm curious, what's your interest in the train schedules? One day at an antique shop, I came across a book from ~1910 which had hundreds of pages of annual reports from railroads with many metrics we'd expect to see in the 10-K reports public companies file. The book was published annually, but had much of its data in tables with grouped headers and cells, which could make automated OCR-ing with a good (useful) end result challenging. I think it'd be interesting to map out the Railroad consolidation, track all their financial metrics over time, and do some level of forensic accounting to see if/which companies probably had funny business going on.
    - ant6n 7y ago
      I’ve got European rail schedules, which gives an indication of historical travel patterns. Specifically looking into the development of overnight travel.
- th0ma5 7y ago
  Probably because 99% accuracy isn't good enough.
  - scrollaway 7y ago
    What accuracy do you think humans have? If I give you 1000 handwritten numbers, do you think you'll make less than 10 mistakes?
    - deepnet 7y ago
      They will likely assign the same tasks to multiple people and use an ensemble of human consensus.
    - Arbalest 7y ago
      Run both OCR and human in parallel and double check anomalies
      - saagarjha 7y ago
        I would expect humans to make similar errors.
        Arnt 7y ago
        AIUI the classic procedure for tasks like this is/was: Hide some graduate students. Tell each to transcribe, and to mark the difficult spots. Give each page to two students. Next, have someone else process the page pairs and resolve conflicts and the marked trouble spots. As long as one of the students notices that a particular spot is difficult to read, the error is discovered and can be handled by someone who isn't numb from transcribing pages of simple numbers.
    - astrobe_ 7y ago
      The forms are in open access, anyone can fill one form, so they surely did consider variations in accuracy (if not plain sabotage). They are probably submitting the same data to multiple people and the cross-checking the submissions. Also, the totals help to filter out submissions with mistakes. One can also add bound checking for individual values (misplaced decimal point, extra digits etc.). I would love to see how they do it, but they'll probably keep it confidential because trolls and other vandals.
tekknolagi 7y ago
It would be great if the data entry panel and the main document scrolled separately. Right now I have to scroll up and down to enter data, which is frustrating.
- mustardo 7y ago
  On desktop you can drag the document (image) up and down which helps :-)
- matthewheath 7y ago
  I usually scroll down the main document and use the Tab key to tab between each field to enter data - that way, I don't need to keep scrolling back up and down.
tryonlinux 7y ago
What they should do is make a crypto platform like curecoin but for this where you earn coin for transcription of these documents (and others, could be a cool platform).
- viraptor 7y ago
  Or, you know, cut out the middle man and just pay per transcribed page.
teddyuk 7y ago
Mechanical Turk?
pvaldes 7y ago
Why science must always be done for free, and all the other non scientific stuff, including things dificult to explain logically, are gladly and generously overpaid? Universities take an economic profit from this volunteer work. They can just share some and hire a few students to do it. And if there is a scarcity of more volunteers, maybe they should think about if repeatedly firing, whimsically dismantling expert teams, and cut the funds to scientists at the slighest opportunity really pays at long term.
- timthorn 7y ago
  Zooniverse was a direct response to a PhD project that Chris Lintott was supervising. His student was tasked with classifying galaxies and in the end decided that the volume of data was just too great. There is a good supply of volunteers, and the tasks are enjoyed by those who undertake them. Where volunteers identify notable things, they are also credited by name in the resulting papers. > maybe they should think about if repeatedly firing, whimsically dismantling expert teams, and cut the funds to scientists at the slighest opportunity really pays at long term. I don't see much evidence for this happening in the UK, where the project is based.
  - pvaldes 7y ago
    > I don't see much evidence for this happening in the UK, where the project is based. Reduction in a 40% of UK funding applicants in the last years: https://www.bbc.com/news/science-environment-50044659 https://www.bbc.com/news/science-environment-50044659 Lets see how the nice promises will perform at the end https://www.sciencemag.org/news/2020/03/uk-cues-big-funding-increases-rd# https://www.sciencemag.org/news/2020/03/uk-cues-big-funding-...
  - nil-sec 7y ago
    Ha, funny seeing this mentioned on HN, this student was my Professor for Astrophysics during my time in this field. He was a great mentor and taught me (and others) quite a lot, in particular about outreach and science communication. The Zooniverse project is amazing and citizen science in general is a great idea. However, you mention the crux of the issue already: the tasks need to be enjoyable. Looking at galaxies that potentially no other human ever laid eyes on is pretty much as exciting as it may get. A former colleague of mine (in the group of the mentioned student) worked on this interesting structure https://en.wikipedia.org/wiki/Hanny%27s_Voorwerp https://en.wikipedia.org/wiki/Hanny%27s_Voorwerp that was discovered by one citizen scientist in the Zooniverse project (hence the name). It's one of the coolest success stories of crowd-sourced research I've heard about.
martyvis 7y ago
Having collected so many data points, I wonder how it was used at the time? Clearly it was probably summed to give yearly totals and possibly monthly averages, and so on. But were the same sheets revisited year after year then?
alt_f4 7y ago
that's a job. go pay someone for their work.
deleted 7y ago
[deleted]
andrewla 7y ago
https://www.zooniverse.org/projects/edh/rainfall-rescue https://www.zooniverse.org/projects/edh/rainfall-rescue is the direct link where you can start transcribing the data yourself.
ilamont 7y ago
10+ years ago, Ancestry.com hired people in Asia to transcribe U.S. census returns and vital records. They also used volunteers: http://blogs.ancestry.com/circle/?p=2321 http://blogs.ancestry.com/circle/?p=2321 Not sure how it's done now, but I suspect it's very hard to automate owing to inconsistent handwriting styles, unusual names for people and places, footnotes and abbreviations, etc.
simon04 7y ago
Some kind of input validation and user feedback would be welcome. Show a green check mark, if the input annual sum equals the computed sum of the input monthly sums. Typically, all values have two decimal places, thus, validate that there are no more and no less in the user input.

102 points by billfruit 7y ago

Why hasn't this been automated? Digit recognition technology is pretty good and it seems like the forms are pretty standardized.

ant6n 7y ago
Is there a technology that would help ocr-ing standardized forms? I’ve got a couple thousand pages of historical train schedules that i would like to digitize (tabulated, printed data, including symbols/icons), but I’m not sure how to automatically recognize the structured data.
- GijsjanB 7y ago
  I don't know about out-of-the-box ready software. Depending on how much time you have you'd either train your own model with your own ground truth or you can use an open source model. Libraries and archives sometimes make theirs available (I know the Dutch National Archives do). You can train the model and/or HTR the data with software like Transcribus or Kraken (open source).
- jasoncabot 7y ago
  (disclaimer MS employee but nothing to do with this product) You could check out the Azure Forms Recognizer. It's designed to do exactly this and make even training a custom model pretty easy. It's also pretty cheap (free up to 500 pages) so you can always do a quick experiment and see if it does what you need it to https://docs.microsoft.com/en-us/azure/cognitive-services/form-recognizer/overview https://docs.microsoft.com/en-us/azure/cognitive-services/fo...
- martyvis 7y ago
  I remember getting some reference software and methods from NIST maybe 20 years ago on handwriting recognition. Never put it into practice, but maybe they have some updated software available
- froindt 7y ago
  I'm curious, what's your interest in the train schedules? One day at an antique shop, I came across a book from ~1910 which had hundreds of pages of annual reports from railroads with many metrics we'd expect to see in the 10-K reports public companies file. The book was published annually, but had much of its data in tables with grouped headers and cells, which could make automated OCR-ing with a good (useful) end result challenging. I think it'd be interesting to map out the Railroad consolidation, track all their financial metrics over time, and do some level of forensic accounting to see if/which companies probably had funny business going on.
  - ant6n 7y ago
    I’ve got European rail schedules, which gives an indication of historical travel patterns. Specifically looking into the development of overnight travel.
th0ma5 7y ago
Probably because 99% accuracy isn't good enough.
- scrollaway 7y ago
  What accuracy do you think humans have? If I give you 1000 handwritten numbers, do you think you'll make less than 10 mistakes?
  - deepnet 7y ago
    They will likely assign the same tasks to multiple people and use an ensemble of human consensus.
  - Arbalest 7y ago
    Run both OCR and human in parallel and double check anomalies
    - saagarjha 7y ago
      I would expect humans to make similar errors.
      - Arnt 7y ago
        AIUI the classic procedure for tasks like this is/was: Hide some graduate students. Tell each to transcribe, and to mark the difficult spots. Give each page to two students. Next, have someone else process the page pairs and resolve conflicts and the marked trouble spots. As long as one of the students notices that a particular spot is difficult to read, the error is discovered and can be handled by someone who isn't numb from transcribing pages of simple numbers.
  - astrobe_ 7y ago
    The forms are in open access, anyone can fill one form, so they surely did consider variations in accuracy (if not plain sabotage). They are probably submitting the same data to multiple people and the cross-checking the submissions. Also, the totals help to filter out submissions with mistakes. One can also add bound checking for individual values (misplaced decimal point, extra digits etc.). I would love to see how they do it, but they'll probably keep it confidential because trolls and other vandals.

tekknolagi 7y ago

It would be great if the data entry panel and the main document scrolled separately. Right now I have to scroll up and down to enter data, which is frustrating.

mustardo 7y ago
On desktop you can drag the document (image) up and down which helps :-)
matthewheath 7y ago
I usually scroll down the main document and use the Tab key to tab between each field to enter data - that way, I don't need to keep scrolling back up and down.

tryonlinux 7y ago

What they should do is make a crypto platform like curecoin but for this where you earn coin for transcription of these documents (and others, could be a cool platform).

viraptor 7y ago
Or, you know, cut out the middle man and just pay per transcribed page.

teddyuk 7y ago

Mechanical Turk?

pvaldes 7y ago

Why science must always be done for free, and all the other non scientific stuff, including things dificult to explain logically, are gladly and generously overpaid? Universities take an economic profit from this volunteer work. They can just share some and hire a few students to do it. And if there is a scarcity of more volunteers, maybe they should think about if repeatedly firing, whimsically dismantling expert teams, and cut the funds to scientists at the slighest opportunity really pays at long term.

timthorn 7y ago
Zooniverse was a direct response to a PhD project that Chris Lintott was supervising. His student was tasked with classifying galaxies and in the end decided that the volume of data was just too great. There is a good supply of volunteers, and the tasks are enjoyed by those who undertake them. Where volunteers identify notable things, they are also credited by name in the resulting papers. > maybe they should think about if repeatedly firing, whimsically dismantling expert teams, and cut the funds to scientists at the slighest opportunity really pays at long term. I don't see much evidence for this happening in the UK, where the project is based.
- pvaldes 7y ago
  > I don't see much evidence for this happening in the UK, where the project is based. Reduction in a 40% of UK funding applicants in the last years: https://www.bbc.com/news/science-environment-50044659 https://www.bbc.com/news/science-environment-50044659 Lets see how the nice promises will perform at the end https://www.sciencemag.org/news/2020/03/uk-cues-big-funding-increases-rd# https://www.sciencemag.org/news/2020/03/uk-cues-big-funding-...
- nil-sec 7y ago
  Ha, funny seeing this mentioned on HN, this student was my Professor for Astrophysics during my time in this field. He was a great mentor and taught me (and others) quite a lot, in particular about outreach and science communication. The Zooniverse project is amazing and citizen science in general is a great idea. However, you mention the crux of the issue already: the tasks need to be enjoyable. Looking at galaxies that potentially no other human ever laid eyes on is pretty much as exciting as it may get. A former colleague of mine (in the group of the mentioned student) worked on this interesting structure https://en.wikipedia.org/wiki/Hanny%27s_Voorwerp https://en.wikipedia.org/wiki/Hanny%27s_Voorwerp that was discovered by one citizen scientist in the Zooniverse project (hence the name). It's one of the coolest success stories of crowd-sourced research I've heard about.

martyvis 7y ago

Having collected so many data points, I wonder how it was used at the time? Clearly it was probably summed to give yearly totals and possibly monthly averages, and so on. But were the same sheets revisited year after year then?

alt_f4 7y ago

that's a job. go pay someone for their work.

deleted 7y ago

[deleted]

andrewla 7y ago

https://www.zooniverse.org/projects/edh/rainfall-rescue https://www.zooniverse.org/projects/edh/rainfall-rescue is the direct link where you can start transcribing the data yourself.

ilamont 7y ago

10+ years ago, Ancestry.com hired people in Asia to transcribe U.S. census returns and vital records. They also used volunteers: http://blogs.ancestry.com/circle/?p=2321 http://blogs.ancestry.com/circle/?p=2321 Not sure how it's done now, but I suspect it's very hard to automate owing to inconsistent handwriting styles, unusual names for people and places, footnotes and abbreviations, etc.

simon04 7y ago

Some kind of input validation and user feedback would be welcome. Show a green check mark, if the input annual sum equals the computed sum of the input monthly sums. Typically, all values have two decimal places, thus, validate that there are no more and no less in the user input.