4 ms·
What documents (books, scientific articles) benefit from specifically is a number of highly consistent, highly accurate identifiers: DOI (scientific articles),
by dredmorbius 5y ago
What documents (books, scientific articles) benefit from specifically is a number of highly consistent, highly accurate identifiers: DOI (scientific articles), ISBN (published books), and others (OCLC identifier, Library of Congress Catalogue Number, etc.)
With the addition of hashsums (even MD5 ad SHA1, though longer and more robust hashsums are preferred), a pretty reliable archive of content can be made. It's a curious case where increased legibility seems to be breaking rather than creating a gatekeeper monopoly.
I've been interested in the notion of more reliable content-based identifiers or fingerprints themselves, though I've found little reliable reference on this. Ngram tuples of 4-5 words are often sufficient to identify a work, particularly if a selection of several are made. Aggreeing on which tuples to use, how many, and how to account for potential noise / variations (special characters, whitespace variance, OCR inaccuracy) is also a stumbling point.
- spicybright 5y agoWhy map anything to words for strict identification? Words and language are very error prone vs an id number or hash.
- mandelken 5y agoProbably because for written text the words identify the content while the hash relates more to the digital carrier format (pdf vs epub) and id number can change between publications, countries, etc.
- dredmorbius 5y agoBingo. And to drag in metadata, it may: - Not be present. - Be inaccurately applied to the correct work (metadata say the work is different, work is in fact related/same). - Be inaccurately applied to the wrong work (metadata say the works are the same/related, they are not).
- dredmorbius 5y agoIt's a bit of an itch I've been scratching for a few years. Most especially, given two or more instances of what you suspect to be the same or a substantively similar work, how can you assess this in a robust and format-independent manner, programmatically? For works with well-formed metadata, this isn't an issue. For identical duplicate copies of the same file, a hash is effective. But for the circumstance most often encountered in reality --- different forms and formats derived from different sources but containing substantially the same work --- there is no simple solution of which I'm aware. As examples, say you have a reference document The Reference Document. How do I determine that: - An ACSCII-only textfile - Markdown, HTML, DocBook, and LaTeX sources - PDF, MS Word (which version), PS, DJVU, ePub, or .mobi files (sling any other formats you care to mention). - Hardbound and paperback physical copies - Scans made from the same or different physical books or instances, versions, and/or translations. - Audiobooks based on a work. By the same or different readers. - Dramatic performances, films, video series, comic-book adaptations, etc., of a work. (Say: Hamlet or Romeo and Juliet. What is the relationship to "West Side Story" (and which version), or Pyramus and Thisbe?) - Re-typed or OCRed text ... all refer to the same work? How do you define "work"? How do you define "differences between works"? How do you distinguish intentional, accidental, and incidental differences between instances? (Say: translations, errata, corrections, additions for the one, transcription errors for the second, and scanning or rendering artefacts for the third.) If you're working in an environment in which instances of works come from different sources with different provenances, these questions arise. At least some of these questions are prominent in library science itself. It's the technical mapping of digitised formats I'm focusing on most closely, so the physical instantiations aren't as critical here, though the presumption is that these could be converted to some machine-readable form. In bibliographic / library science, the term is "work, expression, manifestation" https://www.loc.gov/marc/marbi/2011/2011-dp03.html https://www.loc.gov/marc/marbi/2011/2011-dp03.html
- mathnmusic 5y agoI have been dealing with the same problem for curating resources at https://learnawesome.org https://learnawesome.org. Projects like Openlibrary do collect unique identifiers for _books_, but for everything else, it mostly takes manual effort. For example, I collect talks/podcasts by the author where they discuss ideas from their books. Then there are summaries written by others.
- fnord77 5y agotext to speech the doc then an acoustic fingerprint on the audio :)
- dredmorbius 5y agoYou'd all but certainly be better going in the other direction. Text is a more constrained state space than speech/audio.
- deleted 5y ago[deleted]