5 ms·

> How could vital records and other paper archives at the municipal level be digitized and indexed? The typical business model is for a company to digitize the

> How could vital records and other paper archives at the municipal level be digitized and indexed? The typical business model is for a company to digitize the ones somebody might want to look at someday and offer online access. > What are the technical and legal considerations for such a project? Legal: more than can be articulated here, but the FCRA is sort of the big one that covers much more than you'd think. Technical: buy a scanner or two.

peterburkimsher 8y ago
Please can you elaborate more about the legal aspects? I've scanned a lot of Chinese-language material (especially bilingual English/Chinese and dialects) and found some new Unicode characters that way. I'm reluctant to share it online because I'm scared about copyright.
- bumholio 8y ago
  The original copyright is with the paper for anything published after the second world war (the exact date depends on the country). They have the sole right to decide how and when their archives are published online. Determining copyright status is non-trivial, here is an example for some US periodicals: http://onlinebooks.library.upenn.edu/cce/firstperiod.html http://onlinebooks.library.upenn.edu/cce/firstperiod.html That being said, the paper have usually long since went under and the copyrights will never be asserted. In the extremely unlikely case they will be (for example, if the paper is still in circulation), a free online archive with no advertisements falls squarely under fair use, given the immense historical value of old papers and their nil current commercial value.
  - peterburkimsher 8y ago
    "nil current commercial value" can't be right. "He has, on occasion, been approached by companies looking to partner with him or purchase licenses to his archives. He’s turned them all down, including one offer for half a million dollars. “I knew [my collection] would ultimately be charged for,” Tryniski says nonchalantly, explaining why he declined the offer—for most people, an enormous sum of money. “I really didn’t like the idea of charging a guy to use my site, and then for them to take the biggest profit." I assume the operation used a Russian server because of the copyright issue.
ilamont 8y ago
In terms of scanners, are there models that can handle odd-sized books and ledgers, many in fragile condition? This is where many old vital records and documents reside. Another issue for indexing: 18th, 19th and early 20th century handwriting styles. Many clerks had good handwriting, but with lots of variation in style, written over printed templates (marriage certificates, property and tax records, etc). How good are modern OCR technologies recognizing variations in handwriting?
- superhuzza 8y ago
  1. If documents are odd sized, or fragile, cameras usually the best choice for digitization. Doesn't have to touch the pages, software can handle differently sized formats etc. 2. OCR still isn't even 100% accurate for printed text. Handwritten text accuracy is worse. Cursive is even less accurate.
- justin66 8y ago
  There are a few different levels of digitization here if you're talking about something like a county clerk or a state office or something. You can digitize the documents with OCR (or by hand) and process them with a level of precision great enough that you can archive or destroy the original documents and just print out a new document when you need it. One example would be the states that no longer issue a "real" birth certificate and just print one out on official letterhead when there is a demand. You can index the documents and stuff them in the basement, which is pretty much the default for many county records. The computer is basically equivalent to a card catalog in that case. The database might very well be a scanned card catalog... You can make a nice, pretty, OCR-enabled picture of the original document and store it on a database. Searchability can vary. Things like signatures and cursive writing will often just need to be interpreted by a human during the process of digitization. If such records aren't important enough to justify a few minutes of someone's time, they might not ever be digitized. There's a lot of variance here. A state agency like a Department of Motor Vehicles has a strong motivation to make things more efficient and the advantage that none of the records are really ancient. A county clerk in a poor, rural county has completely different resources and needs.
userbinator 8y ago
Actually, I think cameras are the preferred method for digitisation now --- they've gotten much better in terms of resolution, and have always been many times faster.
- vinceguidry 8y ago
  Dropbox has a document scanner that has completely obliterated the anxiety I get over whether I should keep a particular document or not. If it doesn't need to be physically presented anywhere, it can be scanned and tossed, and Dropbox offers a convenient, perfectly adequate solution to do that. Documents that do need physical presentation can always be reacquired, but there's only a handful of those, easy to put them all in an envelope and stow 'em.
  - toomuchtodo 8y ago
    Yes! Dropbox really nailed this. I keep hoping it’s a feature iOS ends up implementing natively in the camera app with the ability to save to iCloud Files.
- crispyambulance 8y ago
  That's interesting. I guess it would happen eventually. I knew that a while back drum scanners were the ultimate for scanning large documents/artworks. I wonder if now they're doing image stitching with cameras?
  - Kadin 8y ago
    You can certainly do image stitching with camera images. From the software's perspective it's no different than putting together a panorama, really. (There are some minor differences, since the camera is not being panned from a static position, but they are not big.) Drum scanners were always the ne plus ultra when it came to scanning film, particularly large format film like 4x5 negatives. But as CMOS sensors have gotten better, the advantage has diminished (the design of drum scanners was driven by PMTs and later, linear CCDs), and I don't know if even 4x5 format photographers are bothering anymore. They are a real pain to work with, especially the ones that required a "wet mount" of the negative. Historical negatives were never a good match for them. It's nice that the advancements in the consumer market have made it much easier for archivists doing copy stand work.

peterburkimsher 8y ago

Please can you elaborate more about the legal aspects? I've scanned a lot of Chinese-language material (especially bilingual English/Chinese and dialects) and found some new Unicode characters that way. I'm reluctant to share it online because I'm scared about copyright.

bumholio 8y ago
The original copyright is with the paper for anything published after the second world war (the exact date depends on the country). They have the sole right to decide how and when their archives are published online. Determining copyright status is non-trivial, here is an example for some US periodicals: http://onlinebooks.library.upenn.edu/cce/firstperiod.html http://onlinebooks.library.upenn.edu/cce/firstperiod.html That being said, the paper have usually long since went under and the copyrights will never be asserted. In the extremely unlikely case they will be (for example, if the paper is still in circulation), a free online archive with no advertisements falls squarely under fair use, given the immense historical value of old papers and their nil current commercial value.
- peterburkimsher 8y ago
  "nil current commercial value" can't be right. "He has, on occasion, been approached by companies looking to partner with him or purchase licenses to his archives. He’s turned them all down, including one offer for half a million dollars. “I knew [my collection] would ultimately be charged for,” Tryniski says nonchalantly, explaining why he declined the offer—for most people, an enormous sum of money. “I really didn’t like the idea of charging a guy to use my site, and then for them to take the biggest profit." I assume the operation used a Russian server because of the copyright issue.

ilamont 8y ago

In terms of scanners, are there models that can handle odd-sized books and ledgers, many in fragile condition? This is where many old vital records and documents reside. Another issue for indexing: 18th, 19th and early 20th century handwriting styles. Many clerks had good handwriting, but with lots of variation in style, written over printed templates (marriage certificates, property and tax records, etc). How good are modern OCR technologies recognizing variations in handwriting?

superhuzza 8y ago
1. If documents are odd sized, or fragile, cameras usually the best choice for digitization. Doesn't have to touch the pages, software can handle differently sized formats etc. 2. OCR still isn't even 100% accurate for printed text. Handwritten text accuracy is worse. Cursive is even less accurate.
justin66 8y ago
There are a few different levels of digitization here if you're talking about something like a county clerk or a state office or something. You can digitize the documents with OCR (or by hand) and process them with a level of precision great enough that you can archive or destroy the original documents and just print out a new document when you need it. One example would be the states that no longer issue a "real" birth certificate and just print one out on official letterhead when there is a demand. You can index the documents and stuff them in the basement, which is pretty much the default for many county records. The computer is basically equivalent to a card catalog in that case. The database might very well be a scanned card catalog... You can make a nice, pretty, OCR-enabled picture of the original document and store it on a database. Searchability can vary. Things like signatures and cursive writing will often just need to be interpreted by a human during the process of digitization. If such records aren't important enough to justify a few minutes of someone's time, they might not ever be digitized. There's a lot of variance here. A state agency like a Department of Motor Vehicles has a strong motivation to make things more efficient and the advantage that none of the records are really ancient. A county clerk in a poor, rural county has completely different resources and needs.

userbinator 8y ago

Actually, I think cameras are the preferred method for digitisation now --- they've gotten much better in terms of resolution, and have always been many times faster.

vinceguidry 8y ago
Dropbox has a document scanner that has completely obliterated the anxiety I get over whether I should keep a particular document or not. If it doesn't need to be physically presented anywhere, it can be scanned and tossed, and Dropbox offers a convenient, perfectly adequate solution to do that. Documents that do need physical presentation can always be reacquired, but there's only a handful of those, easy to put them all in an envelope and stow 'em.
- toomuchtodo 8y ago
  Yes! Dropbox really nailed this. I keep hoping it’s a feature iOS ends up implementing natively in the camera app with the ability to save to iCloud Files.
crispyambulance 8y ago
That's interesting. I guess it would happen eventually. I knew that a while back drum scanners were the ultimate for scanning large documents/artworks. I wonder if now they're doing image stitching with cameras?
- Kadin 8y ago
  You can certainly do image stitching with camera images. From the software's perspective it's no different than putting together a panorama, really. (There are some minor differences, since the camera is not being panned from a static position, but they are not big.) Drum scanners were always the ne plus ultra when it came to scanning film, particularly large format film like 4x5 negatives. But as CMOS sensors have gotten better, the advantage has diminished (the design of drum scanners was driven by PMTs and later, linear CCDs), and I don't know if even 4x5 format photographers are bothering anymore. They are a real pain to work with, especially the ones that required a "wet mount" of the negative. Historical negatives were never a good match for them. It's nice that the advancements in the consumer market have made it much easier for archivists doing copy stand work.