3 ms·

Are you guys concerned with decay at all? How do you know the file stored 2 years ago is still the same file? what happens if its become corrupted?

textfiles 11y ago
Currently, the system requires you to check in (using the git-annex facility) on a regular basis. If you don't check in (it's a script run by cron) within 2 weeks, you're considered decayed, and after 30 days, your contributions fall back into the pool and are taken in elsewhere.
- meesterdude 11y ago
  I'm talking more about the checksums of files over time, and error correction should they deviate. I recently read about facebooks cold storage and it got me thinking about it (https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...) though for them they just want one copy total.
  - textfiles 11y ago
    git-annex is the engine behind this project currently, and a lot of bugfixes, feature-adds and work has happened as a result. https://git-annex.branchable.com/ https://git-annex.branchable.com/
j_s 11y ago
I would be interested to hear about protections against active misinformation attacks, where someone is attempting to change archived content maliciously.
- joeyh 11y ago
  The checksums we have from the IA are md5sums, which are not ideal, so a preimage attack is possible, but AFAICS you could only use it if you're generating the original file that is stored in the IA, and are planning to replace that with a colliding version in the future. Otherwise, we can detect falsified files. There are some potential attacks of putting false information into the git-annex repositories, that we use for tracking which clients are storing which files. We'll eventually need post-receive hooks to validate that pushes only change information about the client making the push. For now, if someone attempts this attack, we can revert their malicious changes after the fact.
  - gojomo 11y ago
    ...if you're generating the original file... Maybe for the time being, with published attacks. But MD5's strength against attacks has been sufficiently dubious since early 1996 – before the founding of the Internet Archive – that experts have recommended against using it in new applications. By 2009, the CMU Software Engineering Institute (responsible for CERT security notices) wrote: Software developers, Certification Authorities, website owners, and users should avoid using the MD5 algorithm in any capacity. As previous research has demonstrated, it should be considered cryptographically broken and unsuitable for further use. So they're worse than 'not ideal'. They're dangerously obsolete for the purpose of content-authenticity, and any reliance upon them (under hand-wavy conditions) serves to block the migration to available hashes that would actually work. If you're making the backup to be sent through a wormhole to 1999, use MD5. If you're creating the backup for trust and availability through to end-of-year 2015, or 2025, or 2115, ditch MD5 ASAP. [1] http://www.kb.cert.org/vuls/id/836068 http://www.kb.cert.org/vuls/id/836068

textfiles 11y ago

Currently, the system requires you to check in (using the git-annex facility) on a regular basis. If you don't check in (it's a script run by cron) within 2 weeks, you're considered decayed, and after 30 days, your contributions fall back into the pool and are taken in elsewhere.

meesterdude 11y ago
I'm talking more about the checksums of files over time, and error correction should they deviate. I recently read about facebooks cold storage and it got me thinking about it (https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...) though for them they just want one copy total.
- textfiles 11y ago
  git-annex is the engine behind this project currently, and a lot of bugfixes, feature-adds and work has happened as a result. https://git-annex.branchable.com/ https://git-annex.branchable.com/

j_s 11y ago

I would be interested to hear about protections against active misinformation attacks, where someone is attempting to change archived content maliciously.

joeyh 11y ago
The checksums we have from the IA are md5sums, which are not ideal, so a preimage attack is possible, but AFAICS you could only use it if you're generating the original file that is stored in the IA, and are planning to replace that with a colliding version in the future. Otherwise, we can detect falsified files. There are some potential attacks of putting false information into the git-annex repositories, that we use for tracking which clients are storing which files. We'll eventually need post-receive hooks to validate that pushes only change information about the client making the push. For now, if someone attempts this attack, we can revert their malicious changes after the fact.
- gojomo 11y ago
  ...if you're generating the original file... Maybe for the time being, with published attacks. But MD5's strength against attacks has been sufficiently dubious since early 1996 – before the founding of the Internet Archive – that experts have recommended against using it in new applications. By 2009, the CMU Software Engineering Institute (responsible for CERT security notices) wrote: Software developers, Certification Authorities, website owners, and users should avoid using the MD5 algorithm in any capacity. As previous research has demonstrated, it should be considered cryptographically broken and unsuitable for further use. So they're worse than 'not ideal'. They're dangerously obsolete for the purpose of content-authenticity, and any reliance upon them (under hand-wavy conditions) serves to block the migration to available hashes that would actually work. If you're making the backup to be sent through a wormhole to 1999, use MD5. If you're creating the backup for trust and availability through to end-of-year 2015, or 2025, or 2115, ditch MD5 ASAP. [1] http://www.kb.cert.org/vuls/id/836068 http://www.kb.cert.org/vuls/id/836068