4 ms·

You might be able to use a blockchain here. Each block in the chain stores "transactions" that are cryptographically secure hashes of the content of a URL. To

You might be able to use a blockchain here. Each block in the chain stores "transactions" that are cryptographically secure hashes of the content of a URL. To archive an entire page, which may link to various resource URLs, you would need multiple transactions; one for each resource. Effectively, a transaction acts like a claim that a page looked a certain way on a certain date. Mining operates like a normal blockchain where blocks are validated by other parties based on downloading the content of the URL, comparing it to the hash, and discarding the block if the content doesn't hash to the same value. Miners are compensated either with a type of in-chain currency or a smart contract granting them another cryptocurrency based on how many blocks they mine. This system still has a few flaws: 1) Websites could still prevent their archival by changing the website on each access. Since each node in the network downloads the page independently, the downloads must all serve the same content for the transaction to validate. 2) Malicious miners could request archival of a website, then change the content of the website as soon as a new block is mined. With enough luck, they can prevent the verification of a new block, stopping the original miner from claiming their reward.

captn3m0 6y ago
1 is a pretty big flaw. As a trivial example, the "Date" header changes for every request on most web servers. News websites have minute to minute updates. The problem statement in the blog post covers these issues and sticks to validatation of a already-downloaded cache.
SahAssar 6y ago
That seems like a massively overengineered solution which does not solve the actual question: > If I want to contribute my own crawl results, how can the Archive verify that I didn’t forge the pages I’m submitting? Using a blockchain for this solves nothing and only introduces new problems.

captn3m0 6y ago

1 is a pretty big flaw. As a trivial example, the "Date" header changes for every request on most web servers. News websites have minute to minute updates. The problem statement in the blog post covers these issues and sticks to validatation of a already-downloaded cache.

SahAssar 6y ago

That seems like a massively overengineered solution which does not solve the actual question: > If I want to contribute my own crawl results, how can the Archive verify that I didn’t forge the pages I’m submitting? Using a blockchain for this solves nothing and only introduces new problems.