3 ms·

Retaining information in an organised way that scales is a huge unsolved problem.

tablespoon 5y ago
> Retaining information in an organised way that scales is a huge unsolved problem. How so? Libraries and archives have been around for a long time, and seem to work pretty well.
- anfilt 5y ago
  Libaries have screwed this up as well numerous times. less so for proper archives. However lets take a wide example micro-film. Many libraries started storing periodicals on microfilm. Thing is for a lot of older local news papers and such thats now all that exists. (no known original left) It also takes money to put those micro-film scans on something that is not obsolete. You still run into libraries keeping a microfilm reader on hand since they don't have the funds to convert everything. While microfilm was seen to have advantages durning its peak as its become more obsolete it become obvious it has draw backs compared to keeping original documents. At least its relatively simple in practice just needing about 100x microscope to read. The problem with digital information is worse especially you if throw in proprietary disk formats and proprietary file formats in the mix. Let alone keeping hardware operating that can use older interfaces for the storage medium, the alternative is keep copying the data to newer media and ensure that no errors happened during that process I guess my point is that keeping a archived/preserved piece of data accessible always has an upkeep cost. Even books and other physical things have an upkeep cost mainly keeping a safe environment to store said items to keep them safe things like uv that may fade items or humidity that make them mold ect... Let alone protecting items from cataclysmic events fire ect.. Any break down along this line for a library or archive means information may be lost or no longer accessible. The problem is the same for companies except now they have to weigh spending money on this for documentation and such for products they no longer sell and likely do not even support any more if old enough.
  - pm90 5y ago
    Seems like libraries should band together and create archiving standards, and just build on top of a common archiving infrastructure?
    - AlotOfReading 5y ago
      They have [1]. However, libraries are incredibly decentralized, can have massive collections of "things" (not simply books) measured in the millions, and possess the kind of institutional inertia that comes with operating for literal centuries on a relatively shoestring budget. It's somewhat difficult getting everyone around the world to agree on single system that can potentially catalogue every human creation in existence, let alone to implement it as common infrastructure. [1] https://www.loc.gov/librarians/standards https://www.loc.gov/librarians/standards
MathMonkeyMan 5y ago
I believe you, but what about snapshots? I'm thinking something like the internet archive. Documentation websites can go about their business however they like, but there's always a "show me (most of) MSDN as of October 2016" option. Internal bureaucracy would have to be in place to make sure that the feature doesn't get completely broken on a whim by a disinterested branch of the company, but I think it's feasible. It's justifying this expense that would be difficult.
chalst 5y ago
A former girlfriend of mine used to work in the archival department of the BBC, which covered content and legally sensitive communications. They had big archives and their system worked; their biggest problem was educating new employees throughout the organisation, a vast bureaucracy, to use the system: that's a compliance issue, not a technical issue. You're missing "at a price that most companies are willing to pay" in there.
- ghaff 5y ago
  I saw a presentation on The New York Times' photo "morgue" a while back. They've been digitizing it inclusing putting lots of effort into metadata because it's pretty much useless otherwise. https://www.nytimes.com/2018/11/10/reader-center/past-tense-photos-history-morgue.html https://www.nytimes.com/2018/11/10/reader-center/past-tense-... There are also a lot of issues with making it generally accessible because the NYT doesn't own the rights to simply opening up the photos to the world.

> Retaining information in an organised way that scales is a huge unsolved problem. How so? Libraries and archives have been around for a long time, and seem to work pretty well.

anfilt 5y ago
Libaries have screwed this up as well numerous times. less so for proper archives. However lets take a wide example micro-film. Many libraries started storing periodicals on microfilm. Thing is for a lot of older local news papers and such thats now all that exists. (no known original left) It also takes money to put those micro-film scans on something that is not obsolete. You still run into libraries keeping a microfilm reader on hand since they don't have the funds to convert everything. While microfilm was seen to have advantages durning its peak as its become more obsolete it become obvious it has draw backs compared to keeping original documents. At least its relatively simple in practice just needing about 100x microscope to read. The problem with digital information is worse especially you if throw in proprietary disk formats and proprietary file formats in the mix. Let alone keeping hardware operating that can use older interfaces for the storage medium, the alternative is keep copying the data to newer media and ensure that no errors happened during that process I guess my point is that keeping a archived/preserved piece of data accessible always has an upkeep cost. Even books and other physical things have an upkeep cost mainly keeping a safe environment to store said items to keep them safe things like uv that may fade items or humidity that make them mold ect... Let alone protecting items from cataclysmic events fire ect.. Any break down along this line for a library or archive means information may be lost or no longer accessible. The problem is the same for companies except now they have to weigh spending money on this for documentation and such for products they no longer sell and likely do not even support any more if old enough.
- pm90 5y ago
  Seems like libraries should band together and create archiving standards, and just build on top of a common archiving infrastructure?
  - AlotOfReading 5y ago
    They have [1]. However, libraries are incredibly decentralized, can have massive collections of "things" (not simply books) measured in the millions, and possess the kind of institutional inertia that comes with operating for literal centuries on a relatively shoestring budget. It's somewhat difficult getting everyone around the world to agree on single system that can potentially catalogue every human creation in existence, let alone to implement it as common infrastructure. [1] https://www.loc.gov/librarians/standards https://www.loc.gov/librarians/standards

MathMonkeyMan 5y ago

I believe you, but what about snapshots? I'm thinking something like the internet archive. Documentation websites can go about their business however they like, but there's always a "show me (most of) MSDN as of October 2016" option. Internal bureaucracy would have to be in place to make sure that the feature doesn't get completely broken on a whim by a disinterested branch of the company, but I think it's feasible. It's justifying this expense that would be difficult.

chalst 5y ago

A former girlfriend of mine used to work in the archival department of the BBC, which covered content and legally sensitive communications. They had big archives and their system worked; their biggest problem was educating new employees throughout the organisation, a vast bureaucracy, to use the system: that's a compliance issue, not a technical issue. You're missing "at a price that most companies are willing to pay" in there.

ghaff 5y ago
I saw a presentation on The New York Times' photo "morgue" a while back. They've been digitizing it inclusing putting lots of effort into metadata because it's pretty much useless otherwise. https://www.nytimes.com/2018/11/10/reader-center/past-tense-photos-history-morgue.html https://www.nytimes.com/2018/11/10/reader-center/past-tense-... There are also a lot of issues with making it generally accessible because the NYT doesn't own the rights to simply opening up the photos to the world.