3 ms·

Friendly reminder that archive box exists to let you self host your own archive service. https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox

Friendly reminder that archive box exists to let you self host your own archive service. https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox I dream of a day where archivebox becomes a fleet of homelabs all over the world making it drastically harder to block them all.

codedokode 1y ago
I think about the opposite, people reading in the news that FBI is after archiving sites, will not want to launch their own site, except maybe the radical types.
- wartywhoa23 1y ago
  1960s: FBI/CIA invents the term "conspiracy theorist" 2020s: FBI/CIA invents the term "radical archivist"
nikisweeting 1y ago
I've been mulling over how to take ArchiveBox in this direction for years, but it's a really hard problem to tackle because of privacy. https://docs.sweeting.me/s/cookie-dilemma https://docs.sweeting.me/s/cookie-dilemma Most content is going behind logins these days, and if you include the PII of the person doing the archiving in the archives then it's A. really easy for providers to block that account B. potentially dangerous to dox the person doing the archiving. The problem is removing PII from logged in sites is that it's not as simple as stripping some EXIF data, the html and JS is littered with secret tokens, usernames, user-specific notifications, etc. that would reveal the ID of the archivist and cant be removed without breaking page behavior on replay. My latest progress is that it might be possible to anonymize logged in snapshots by using the intersection of two different logged-in snapshots, making them easier to share over a distributed system like Bittorrent or IPFS without doxxing the archivist. More here: https://github.com/pirate/html-private-set-intersection https://github.com/pirate/html-private-set-intersection
e2le 1y ago
Out of curiosity, does ArchiveBox integrate some way of verifying the contents of the archived page(s) are legitimate and unmodified?
- nikisweeting 1y ago
  ArchiveBox open source does not, but I have set it up for paying clients in the past using TLSNotary. This is actually a very hard problem and is not as simple as saving traffic hashes + original SSL certs (because HTTPS connections use a symmetric key after the initial handshake, the archivist can forge server responses and claim the server sent things that it did not). There is only 1 reasonable approach that I know of as of today: https://tlsnotary.org/docs/intro https://tlsnotary.org/docs/intro, and it still involves trusting a third party with reputation (though it cleverly uses a zk algorithm so that the third party doesn't have to see the cleartext). Anyone claiming to provide "verifyable" web archives is likely lying or overstating it unless they are using TLSNotary or a similar approach. I've seen far to many companies make impossible claims about "signed" or "verified" web archives over the last decade, be very critial any time you see someone claiming that unless they talk explicitly about the "TLS Non-Repudiation Problem" and how they solve it: https://security.stackexchange.com/questions/103645/does-ssl-tls-provide-non-repudiation-service https://security.stackexchange.com/questions/103645/does-ssl...
  - giancarlostoro 1y ago
    If web pages were signed the way emails were, it would authenticate if an archived copy of a web page is indeed authentic, but good luck getting such a major change all the way across the entire web. Why would anyone who would gladly retract / redact information on a whim even subscribe to this technology? Would be nice if they all did though.
    - galaxy_gas 1y ago
      Have you see this before? https://web.dev/articles/signed-exchanges https://web.dev/articles/signed-exchanges Its not for authenticity but instead to prevent tampering of adblock,ad and tracking removals more if there is history in search many complains in this

codedokode 1y ago

I think about the opposite, people reading in the news that FBI is after archiving sites, will not want to launch their own site, except maybe the radical types.

wartywhoa23 1y ago
1960s: FBI/CIA invents the term "conspiracy theorist" 2020s: FBI/CIA invents the term "radical archivist"

nikisweeting 1y ago

I've been mulling over how to take ArchiveBox in this direction for years, but it's a really hard problem to tackle because of privacy. https://docs.sweeting.me/s/cookie-dilemma https://docs.sweeting.me/s/cookie-dilemma Most content is going behind logins these days, and if you include the PII of the person doing the archiving in the archives then it's A. really easy for providers to block that account B. potentially dangerous to dox the person doing the archiving. The problem is removing PII from logged in sites is that it's not as simple as stripping some EXIF data, the html and JS is littered with secret tokens, usernames, user-specific notifications, etc. that would reveal the ID of the archivist and cant be removed without breaking page behavior on replay. My latest progress is that it might be possible to anonymize logged in snapshots by using the intersection of two different logged-in snapshots, making them easier to share over a distributed system like Bittorrent or IPFS without doxxing the archivist. More here: https://github.com/pirate/html-private-set-intersection https://github.com/pirate/html-private-set-intersection

e2le 1y ago

Out of curiosity, does ArchiveBox integrate some way of verifying the contents of the archived page(s) are legitimate and unmodified?

nikisweeting 1y ago
ArchiveBox open source does not, but I have set it up for paying clients in the past using TLSNotary. This is actually a very hard problem and is not as simple as saving traffic hashes + original SSL certs (because HTTPS connections use a symmetric key after the initial handshake, the archivist can forge server responses and claim the server sent things that it did not). There is only 1 reasonable approach that I know of as of today: https://tlsnotary.org/docs/intro https://tlsnotary.org/docs/intro, and it still involves trusting a third party with reputation (though it cleverly uses a zk algorithm so that the third party doesn't have to see the cleartext). Anyone claiming to provide "verifyable" web archives is likely lying or overstating it unless they are using TLSNotary or a similar approach. I've seen far to many companies make impossible claims about "signed" or "verified" web archives over the last decade, be very critial any time you see someone claiming that unless they talk explicitly about the "TLS Non-Repudiation Problem" and how they solve it: https://security.stackexchange.com/questions/103645/does-ssl-tls-provide-non-repudiation-service https://security.stackexchange.com/questions/103645/does-ssl...
- giancarlostoro 1y ago
  If web pages were signed the way emails were, it would authenticate if an archived copy of a web page is indeed authentic, but good luck getting such a major change all the way across the entire web. Why would anyone who would gladly retract / redact information on a whim even subscribe to this technology? Would be nice if they all did though.
  - galaxy_gas 1y ago
    Have you see this before? https://web.dev/articles/signed-exchanges https://web.dev/articles/signed-exchanges Its not for authenticity but instead to prevent tampering of adblock,ad and tracking removals more if there is history in search many complains in this