5 ms·

I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptogra

I recommend Browsertrix for WARC creation, I think they are the best currently available for WARC/WACZ. ArchiveBox is also gearing up to support real cryptographic signing of archives using https://tlsnotary.org/ https://tlsnotary.org/ in an upcoming plugin. (in a way that actually solves the TLS non-repudation issue, which traditional "signing a WARC" does not, more info: https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_09-4_Ritzdorf_paper.pdf https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd...)

deleted 2y ago
[deleted]
toomuchtodo 2y ago
Keep in mind, what signing methodology you use is a function of who accepts it. If I can confirm "ArchiveTeam ripped this", that is is superior to whatever tlsnotary is doing with MPC, blockchain, distributed ledger, whatever (in my use case). Have to trust someone at the end of the day. ArchiveTeam's Warrior doesn't use tlsnotary, for example, and rips entire sites just fine.
- nikisweeting 2y ago
  The idea with TLSNotary is that you can have several universities or central agencies running signing servers but you dont have to share the cleartext content of your archives with them to get it signed. This dramatically changes what is possible with signing because previously to get ArchiveTeam's signature of approval, they would have to see the content themselves to archive it. With TLSNotary they can sign without needing to see the content/access the cookies/etc.
  - viraptor 2y ago
    Isn't that already possible with any kind of notary by giving them a sha256 of the content only? Or am I missing some distinction?
    - nikisweeting 2y ago
      You can do that but it proves nothing because TLS session keys are symmetric, so the archiver can forge server responses and falsely attest that the server sent them. Look up "TLS non repudiation" A real solution like TLSNotary involves a neutral, reputable third party that can't see the cleartext attesting to the cyphertext using a ZK proof. The neutral third party doing attestation can't see the content so they can't easily tamper with it, and attempts to tamper indiscriminately would be easily detected and ding their reputation.
digitaldragon 2y ago
Unfortunately, Browsertrix relies on the Chrome Devtools Protocol, which strips transfer encoding (and possibly transforms the data in other ways). This results in Browsertrix writing noncompliant WARC files, because the spec requires that the original transfer encoding be preserved.
- ikreymer 2y ago
  Unfortunately, there is not much we can do about transfer-encoding, but the data is otherwise exactly as is returned from the browser. Browsertrix uses the browser to create web archives, so users get an accurate representation of what they see in their browser, which is generally what people want from archives. We do the best we can with a limited standard that is difficult to modify. Archiving is always lossy, we try to reduce that as much as possible, but there are limits. People create web archives because they care about not losing their stuff online, not because they need an accurate record of transfer-encoding property in an HTTP connection. If storing the transfer-encoding is the most important thing, then yes, there are better tools for that.
  - CorentinB 2y ago
    You could use a proxy. "Archiving is always lossy" No.
    - nikisweeting 2y ago
      You're talking to the guy who built the best proxy recorder in the archiving industry ;) ikreymer created https://pywb.readthedocs.io/en/latest/ https://pywb.readthedocs.io/en/latest/ I think he has more context than any of us on the limits of proxy archiving vs browser based archiving. But also if you really need perfect packet-level replication, just wireshark it as he said. Why bother with WARCs at all?
      - pabs3 2y ago
        pywb has WARC issues too, due to use of warcio: https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem
fasa99 2y ago
>ArchiveBox is also gearing up to support real cryptographic signing of archives That's a really interesting point. The gut reaction is "why are we wasting time on adding a nice-to-have such as a very fancy cousin of the MD5 checksum when the real meat of the time & effort is maximizing data download and scale" But then, then go read the book 1984 and it may become clear the importance of ensuring the data is unchanged down the road. But if this is a hedge against hypothetical future 1984 world, one would have to ask - what if the only file available has the wrong md5sum? Because then most people would say, "welp, something is better than nothing" and that's it. Perhaps something that might provide additional information about what/how/where something was changed in more detail.

deleted 2y ago

[deleted]

toomuchtodo 2y ago

Keep in mind, what signing methodology you use is a function of who accepts it. If I can confirm "ArchiveTeam ripped this", that is is superior to whatever tlsnotary is doing with MPC, blockchain, distributed ledger, whatever (in my use case). Have to trust someone at the end of the day. ArchiveTeam's Warrior doesn't use tlsnotary, for example, and rips entire sites just fine.

nikisweeting 2y ago
The idea with TLSNotary is that you can have several universities or central agencies running signing servers but you dont have to share the cleartext content of your archives with them to get it signed. This dramatically changes what is possible with signing because previously to get ArchiveTeam's signature of approval, they would have to see the content themselves to archive it. With TLSNotary they can sign without needing to see the content/access the cookies/etc.
- viraptor 2y ago
  Isn't that already possible with any kind of notary by giving them a sha256 of the content only? Or am I missing some distinction?
  - nikisweeting 2y ago
    You can do that but it proves nothing because TLS session keys are symmetric, so the archiver can forge server responses and falsely attest that the server sent them. Look up "TLS non repudiation" A real solution like TLSNotary involves a neutral, reputable third party that can't see the cleartext attesting to the cyphertext using a ZK proof. The neutral third party doing attestation can't see the content so they can't easily tamper with it, and attempts to tamper indiscriminately would be easily detected and ding their reputation.

digitaldragon 2y ago

Unfortunately, Browsertrix relies on the Chrome Devtools Protocol, which strips transfer encoding (and possibly transforms the data in other ways). This results in Browsertrix writing noncompliant WARC files, because the spec requires that the original transfer encoding be preserved.

ikreymer 2y ago
Unfortunately, there is not much we can do about transfer-encoding, but the data is otherwise exactly as is returned from the browser. Browsertrix uses the browser to create web archives, so users get an accurate representation of what they see in their browser, which is generally what people want from archives. We do the best we can with a limited standard that is difficult to modify. Archiving is always lossy, we try to reduce that as much as possible, but there are limits. People create web archives because they care about not losing their stuff online, not because they need an accurate record of transfer-encoding property in an HTTP connection. If storing the transfer-encoding is the most important thing, then yes, there are better tools for that.
- CorentinB 2y ago
  You could use a proxy. "Archiving is always lossy" No.
  - nikisweeting 2y ago
    You're talking to the guy who built the best proxy recorder in the archiving industry ;) ikreymer created https://pywb.readthedocs.io/en/latest/ https://pywb.readthedocs.io/en/latest/ I think he has more context than any of us on the limits of proxy archiving vs browser based archiving. But also if you really need perfect packet-level replication, just wireshark it as he said. Why bother with WARCs at all?
    - pabs3 2y ago
      pywb has WARC issues too, due to use of warcio: https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem

fasa99 2y ago

>ArchiveBox is also gearing up to support real cryptographic signing of archives That's a really interesting point. The gut reaction is "why are we wasting time on adding a nice-to-have such as a very fancy cousin of the MD5 checksum when the real meat of the time & effort is maximizing data download and scale" But then, then go read the book 1984 and it may become clear the importance of ensuring the data is unchanged down the road. But if this is a hedge against hypothetical future 1984 world, one would have to ask - what if the only file available has the wrong md5sum? Because then most people would say, "welp, something is better than nothing" and that's it. Perhaps something that might provide additional information about what/how/where something was changed in more detail.