3 ms·

I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with lo

I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original executable is either not available or not able to be executed.

xoa 7y ago
I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time was more expensive, storage capacity was far more expensive, the ratio of software to data size was higher, etc. The latter seems to tie into "must assume the original executable is either not available or not able to be executed." For serious archiving, does it no make sense to just bundle in not merely the executable, but in some instances an entire environment? In a "fall of civilization" type scenario that may not be helpful vs a clear simple spec and ease of bootstrapping, but for situations where technological continuity is a limiting factor for some reason anyway is it safe to simply assume basics like at this point x86 will never go away as something that is at least virtualized? In my own experience there is a pretty clear cutoff date after which I can continue to run the entire environment in a VM. Again this is shooting the breeze a bit, article is discussing a case where there should be the freedom to choose better formats. But for a lot of important archive material, including software itself, are we getting to the point where many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image, which is then stored with a solid amount of parity and replication?
- matt-attack 7y ago
  > many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image Unfortunately, any such image would presume you have access to the hardware, or it has low-level instruction sets/processor design baked in. Think how many PDP-11's are around today. And in terms of an archive it's only been 50 years since the PDP-11 was invented. That's a blink of an eye in terms of archival standards.
  - theamk 7y ago
    > Think how many PDP-11's are around today. Why does it matter how many physical machines are alive? There are tons of emulators around. There is even one in Javascript, with an ability to load disk images as well. The hard part is hardware - the drives go bad, the computers fail. But disks grow, and it is getting simpler and cheaper to store lots of data. As long as you keep copying the files to modern media every 10 years or so, you should no longer have anybdata loss. (The only exception is proprietary data formats which cannot be opened except by original program which cannot be run in VM easily. Those should be avoided at all costs)
ltbarcly3 7y ago
Why would they need to recreate it solely based on 'documentation'? It is open source, the source code is the documentation. It seems just as likely that the source would survive as it is likely that some complete technical documentation would survive. Maybe they wouldn't be able to compile it (probably they would be able to compile it, I don't see why they wouldn't have some kind of computer emulator available), but it's better than any other kind of documentation you could provide. The src/ tree of xz is 335k (compressed with gzip). If you are worried future digital historians won't be able to figure out the xz format, throw a copy of the gzip'd source onto every drive you store archives on, it would basically be free and would almost guarantee they would have a complete copy of exactly what they would need to decompress the files.
- matt-attack 7y ago
  You're exhibiting shortsightedness when it comes to "source". If I give you some RPG [1] or maybe some ALGO 58 [2] source code are you going to just compile and run it no problem? How about some FLOW-MATIC [3]? Point being that computer languages come and go. [1] https://en.wikipedia.org/wiki/IBM_RPG https://en.wikipedia.org/wiki/IBM_RPG [2] https://en.wikipedia.org/wiki/ALGOL_58 https://en.wikipedia.org/wiki/ALGOL_58 [3] https://en.wikipedia.org/wiki/FLOW-MATIC https://en.wikipedia.org/wiki/FLOW-MATIC
  - ltbarcly3 7y ago
    Yes, programming languages come and go, but I don't see how that matters. Some future historian will either have access to a working copy of xz or they will not. If they don't, and they want to implement it, having a copy of the source code is far better than anything else you could give them. Sure, future programming languages will be quite different, but humans will certainly be able to read and understand C code. If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer? Human languages come and go also.. https://en.wikipedia.org/wiki/Egyptian_hieroglyphs https://en.wikipedia.org/wiki/Egyptian_hieroglyphs https://en.wikipedia.org/wiki/Judaeo-Aragonese https://en.wikipedia.org/wiki/Judaeo-Aragonese https://en.wikipedia.org/wiki/Latin https://en.wikipedia.org/wiki/Latin Any argument you can make about historians being able to recover dead languages you can make the exact same argument for their ability to recover dead computer languages, and there is no better or more accurate specification than the actual code. So let me add to my recommendation, in addition to a copy of the xz source code, include a plain text copy of any 'how to program in C' book, or just the wikipedia page for the C language. That is more than enough for them to construct a program that can decompress xz files, once they relearn how to read whatever long dead language the book is written in (Ancient Pre-Cataclysm Earth English for example).

xoa 7y ago

I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time was more expensive, storage capacity was far more expensive, the ratio of software to data size was higher, etc. The latter seems to tie into "must assume the original executable is either not available or not able to be executed." For serious archiving, does it no make sense to just bundle in not merely the executable, but in some instances an entire environment? In a "fall of civilization" type scenario that may not be helpful vs a clear simple spec and ease of bootstrapping, but for situations where technological continuity is a limiting factor for some reason anyway is it safe to simply assume basics like at this point x86 will never go away as something that is at least virtualized? In my own experience there is a pretty clear cutoff date after which I can continue to run the entire environment in a VM. Again this is shooting the breeze a bit, article is discussing a case where there should be the freedom to choose better formats. But for a lot of important archive material, including software itself, are we getting to the point where many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image, which is then stored with a solid amount of parity and replication?

matt-attack 7y ago
> many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image Unfortunately, any such image would presume you have access to the hardware, or it has low-level instruction sets/processor design baked in. Think how many PDP-11's are around today. And in terms of an archive it's only been 50 years since the PDP-11 was invented. That's a blink of an eye in terms of archival standards.
- theamk 7y ago
  > Think how many PDP-11's are around today. Why does it matter how many physical machines are alive? There are tons of emulators around. There is even one in Javascript, with an ability to load disk images as well. The hard part is hardware - the drives go bad, the computers fail. But disks grow, and it is getting simpler and cheaper to store lots of data. As long as you keep copying the files to modern media every 10 years or so, you should no longer have anybdata loss. (The only exception is proprietary data formats which cannot be opened except by original program which cannot be run in VM easily. Those should be avoided at all costs)

ltbarcly3 7y ago

Why would they need to recreate it solely based on 'documentation'? It is open source, the source code is the documentation. It seems just as likely that the source would survive as it is likely that some complete technical documentation would survive. Maybe they wouldn't be able to compile it (probably they would be able to compile it, I don't see why they wouldn't have some kind of computer emulator available), but it's better than any other kind of documentation you could provide. The src/ tree of xz is 335k (compressed with gzip). If you are worried future digital historians won't be able to figure out the xz format, throw a copy of the gzip'd source onto every drive you store archives on, it would basically be free and would almost guarantee they would have a complete copy of exactly what they would need to decompress the files.

matt-attack 7y ago
You're exhibiting shortsightedness when it comes to "source". If I give you some RPG [1] or maybe some ALGO 58 [2] source code are you going to just compile and run it no problem? How about some FLOW-MATIC [3]? Point being that computer languages come and go. [1] https://en.wikipedia.org/wiki/IBM_RPG https://en.wikipedia.org/wiki/IBM_RPG [2] https://en.wikipedia.org/wiki/ALGOL_58 https://en.wikipedia.org/wiki/ALGOL_58 [3] https://en.wikipedia.org/wiki/FLOW-MATIC https://en.wikipedia.org/wiki/FLOW-MATIC
- ltbarcly3 7y ago
  Yes, programming languages come and go, but I don't see how that matters. Some future historian will either have access to a working copy of xz or they will not. If they don't, and they want to implement it, having a copy of the source code is far better than anything else you could give them. Sure, future programming languages will be quite different, but humans will certainly be able to read and understand C code. If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer? Human languages come and go also.. https://en.wikipedia.org/wiki/Egyptian_hieroglyphs https://en.wikipedia.org/wiki/Egyptian_hieroglyphs https://en.wikipedia.org/wiki/Judaeo-Aragonese https://en.wikipedia.org/wiki/Judaeo-Aragonese https://en.wikipedia.org/wiki/Latin https://en.wikipedia.org/wiki/Latin Any argument you can make about historians being able to recover dead languages you can make the exact same argument for their ability to recover dead computer languages, and there is no better or more accurate specification than the actual code. So let me add to my recommendation, in addition to a copy of the xz source code, include a plain text copy of any 'how to program in C' book, or just the wikipedia page for the C language. That is more than enough for them to construct a program that can decompress xz files, once they relearn how to read whatever long dead language the book is written in (Ancient Pre-Cataclysm Earth English for example).