4 ms·

You build a system based on trust but verify. If the output is the result of a known deterministic program on a known input, anyone can verify it. So they just

You build a system based on trust but verify. If the output is the result of a known deterministic program on a known input, anyone can verify it. So they just have to sign their work. If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. The second associated problem is how would one prevent them from appropriating the work of others that would just re-sign it. One way would be to allow the worker to introduce a few voluntary errors but have a secret joker that allow him to pass the challenge of a failed verification. One alternative way is based on data malleability. The worker pick a secret one way function and compute is F( data + secretFunction(data,epsilon) ) ~ F(data) and publish the values of the secretFunction(data,epsilon) but not the secretFunction. Only someone with knowledge of the secretFunction can make a claim on the work done. If there is a challenge only the real worker will be able to publish the secret of the secretFunction (Or use some zero knowledge proof to convince you they know it).

charcircuit 5y ago
>If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. A web page isn't an immutable piece of text. It can change on every visit and it can sometimes returns errors.
- GistNoesis 5y ago
  That's why you don't index the webpage but a snapshot of it. For example you index the commoncrawl archives, or some content addressable storage like ipfs or torrent file.
  - charcircuit 5y ago
    What's the point of crawling the common crawl archive? It's pointless. You can simply download it.
    - GistNoesis 5y ago
      I think crawling and indexing should be treated differently. Indexing is about extracting value from the data, while crawling is about gathering data. Once a reference snapshot has been crawled, the indexing task is more easily verifiable. The crawling task is harder to verify, because external website could lie to the crawler. So the sensible thing to do is have multiple people crawl the same site and compare their results. Every crawler will publish its snapshots (which may contain some errors or not), and then that's the job of the indexer to combine multiple snapshots of various crawler and filter the errors out and do the de-duplication. The crawling task is less necessary now than it was a few years ago, because there is already plenty of available data. Also most of the valuable data is locked in walled garden, and companies like Cloudflare make the crawling difficult for the rest of the fat tail. So it's better to only have data submitted to you, and outsource the crawling.

charcircuit 5y ago

>If later someone find that they have lied and provided a false result, they lose reputation/stacked coins. A web page isn't an immutable piece of text. It can change on every visit and it can sometimes returns errors.

GistNoesis 5y ago
That's why you don't index the webpage but a snapshot of it. For example you index the commoncrawl archives, or some content addressable storage like ipfs or torrent file.
- charcircuit 5y ago
  What's the point of crawling the common crawl archive? It's pointless. You can simply download it.
  - GistNoesis 5y ago
    I think crawling and indexing should be treated differently. Indexing is about extracting value from the data, while crawling is about gathering data. Once a reference snapshot has been crawled, the indexing task is more easily verifiable. The crawling task is harder to verify, because external website could lie to the crawler. So the sensible thing to do is have multiple people crawl the same site and compare their results. Every crawler will publish its snapshots (which may contain some errors or not), and then that's the job of the indexer to combine multiple snapshots of various crawler and filter the errors out and do the de-duplication. The crawling task is less necessary now than it was a few years ago, because there is already plenty of available data. Also most of the valuable data is locked in walled garden, and companies like Cloudflare make the crawling difficult for the rest of the fat tail. So it's better to only have data submitted to you, and outsource the crawling.