6 ms·

I have a similar, "One day I will do it" project. The idea is that somebody cares enough to make the semantic web work--it's just not the people with write acc

I have a similar, "One day I will do it" project. The idea is that somebody cares enough to make the semantic web work--it's just not the people with write access to the data. I think we can use CTPH algorithms to fingerprint the data independently of whatever names are used, and then we can use that to find representations of the "same data" submitted by other users. Probably there would be some reputation stuff involved, a web of trust, etc. The flow would go something like this: COGNIZE: 0. Encounter messy data in the wild (has pagination, timestamps of access, etc), need other representation (human/computer/whatever) 1. Calculate CTPH fingerprints, use them to search for link: miss 3. Clean data the hard way and publish canonical representation (ipfs?) 4. Generate missing representation the hard way, publish that too 5. Calculate the fingerprints common to both the missing representation and the canonical representation, and publish it as a "link" between the two. Unlike traditional web links, this one is bidirectional. RECOGNIZE 1. Different user encounters "same" data in the wild 2. Calculate fingerprints, use them to search for canonical representation: hit 3. Find further links to see what other representations of the "same data" are available, download them if desired. The fingerprint stuff works, but there's a lot of work left to be done re: mapping fuzzy hashes of "in the wild" data to cryptographic ones of "canonical representations" and finding ways to incentivize users to go through the hassle of the "cognize" step so that other users can benefit from the "recognize" step. Sorry for talking your ear off, it just feels good to know I'm not the only one working on something like this, even if our approaches are quite different (mine works on PDF's only because it works on arbitrary bytes). Good luck with yours.

melony 5y ago
Manual curation was why Berners-lee's Semantic Web failed. Unlike ARIA annotations which is required to serve certain clientele, the semantic web offers no inherent value to the developer. There is an entire field of machine learning dedicated to automatically generating knowledge graphs, I would start there rather than trying to manually curate and annotate.
- __MatrixMan__ 5y ago
  That's a good point. Once the tooling is there for the manual workflow, I expect that an ML-driven approach will plug in to the same hooks without fuss. But I don't think it changes the main problem to be solved, which is that it's data consumers who want the semantic web, but so far it's been up to data providers to implement it. We need to be able to create links between data with without the participation of whoever hosts it.
  - oever 5y ago
    WebAnnotations are meant for that. https://www.w3.org/TR/annotation-model/ https://www.w3.org/TR/annotation-model/
    - __MatrixMan__ 5y ago
      Hmm, it looks like the use of "target" as a uri means that if the file gets rehosted elsewhere under a different name, the annotations will fail to adhere. I'm going for more of a virus-scanner approach, except I'm looking for the good stuff instead of the bad. But otherwise yes. I'll have to give that a closer look and use as much of it as I can.
      - oever 5y ago
        You can use a magnet or ipfs uri or add a triple that specifies other properties of the file that let you find it. Semantic web uses IRIs in triples. They serve as identifiers, not locators (URL). This is because semantic web talks about more than just files. The Web Annotation spec talks about web resources which should be dereferencable. That is a naive assumption that makes the approach brittle. But one can annotate 'sourceDate' and 'cached' which indicate the date for the annotated version and a location for a copy. When you're annotation a resource it's best to also store a copy. The Web Annotations specification allows this.
        zozbot234 5y ago
        It's preferrable to use the ni:// URI scheme since it is standardized, unlike magnet links. There is a draft standard https://datatracker.ietf.org/doc/html/draft-hallambaker-decade-ni-params-01 https://datatracker.ietf.org/doc/html/draft-hallambaker-deca... defining common extensions that enable much the same featureset as magnet.
wnkrshm 5y ago
What you describe has classically been the work of librarians or archivists.
- __MatrixMan__ 5y ago
  Ah well I guess I'm looking to make freelance librarian into something you can be. I figure if you configure the client to keep track of whose annotations you have a history of using, you've got a real granular view of which content providers to pay. I'm imagining a game where we all put $5 in at the start of the month and we all pay each other based on whose content we use the most. Some users will pay more than they make, others will make more than they pay.

melony 5y ago

Manual curation was why Berners-lee's Semantic Web failed. Unlike ARIA annotations which is required to serve certain clientele, the semantic web offers no inherent value to the developer. There is an entire field of machine learning dedicated to automatically generating knowledge graphs, I would start there rather than trying to manually curate and annotate.

__MatrixMan__ 5y ago
That's a good point. Once the tooling is there for the manual workflow, I expect that an ML-driven approach will plug in to the same hooks without fuss. But I don't think it changes the main problem to be solved, which is that it's data consumers who want the semantic web, but so far it's been up to data providers to implement it. We need to be able to create links between data with without the participation of whoever hosts it.
- oever 5y ago
  WebAnnotations are meant for that. https://www.w3.org/TR/annotation-model/ https://www.w3.org/TR/annotation-model/
  - __MatrixMan__ 5y ago
    Hmm, it looks like the use of "target" as a uri means that if the file gets rehosted elsewhere under a different name, the annotations will fail to adhere. I'm going for more of a virus-scanner approach, except I'm looking for the good stuff instead of the bad. But otherwise yes. I'll have to give that a closer look and use as much of it as I can.
    - oever 5y ago
      You can use a magnet or ipfs uri or add a triple that specifies other properties of the file that let you find it. Semantic web uses IRIs in triples. They serve as identifiers, not locators (URL). This is because semantic web talks about more than just files. The Web Annotation spec talks about web resources which should be dereferencable. That is a naive assumption that makes the approach brittle. But one can annotate 'sourceDate' and 'cached' which indicate the date for the annotated version and a location for a copy. When you're annotation a resource it's best to also store a copy. The Web Annotations specification allows this.
      - zozbot234 5y ago
        It's preferrable to use the ni:// URI scheme since it is standardized, unlike magnet links. There is a draft standard https://datatracker.ietf.org/doc/html/draft-hallambaker-decade-ni-params-01 https://datatracker.ietf.org/doc/html/draft-hallambaker-deca... defining common extensions that enable much the same featureset as magnet.

wnkrshm 5y ago

What you describe has classically been the work of librarians or archivists.

__MatrixMan__ 5y ago
Ah well I guess I'm looking to make freelance librarian into something you can be. I figure if you configure the client to keep track of whose annotations you have a history of using, you've got a real granular view of which content providers to pay. I'm imagining a game where we all put $5 in at the start of the month and we all pay each other based on whose content we use the most. Some users will pay more than they make, others will make more than they pay.