4 ms·

>I'd argue that saving absolutely everything likely isn't useful, at least for specified domains I strongly agree, but spinning disk space is very cheap, and y

>I'd argue that saving absolutely everything likely isn't useful, at least for specified domains I strongly agree, but spinning disk space is very cheap, and you never know what you're going to want later... so at least a full month of time there before auto-purge seems quite doable for everything 8) I'd build a system such that if you refer to a given file/page in that timeframe, then the reference count goes up, and it gets moved to permanent storage. 8)

dredmorbius 4y ago
I've been reading up on library functions and operations, much of it through the US Librarian of Congress's annual report to Congress. (I'd posted a link to the archive of these on HN a few weeks back.) Among questions are: - Curation of archive: what is selected, how, and why? - Cataloguing of archive: how do you find what's within it? What purposes does the catalogue serve? - Weeding of archive: what materials are removed, when, and why? - Classification of archive: how are materials organised (especially a concern where physical storage is necessary) and grouped? - Characteristics of archive: what is included, what's been added or removed, what's been circulated, etc., etc. From the LoC sources there's a lot of discussion of how one such system evolved, and what the rationales driving that evolution were. There's also discussion of what usage patterns and concerns were, at least for a physical-book oriented limited-access (members of Congress and other US Government officials principally, members of the public on a limited basis). As of the early 20th century, it seems that very roughly 1% of the collection actually circulated. I should compare that with more recent data.... Which is prelude to some observations for a next-gen Web / online document personal archive curation: - Web-surfing activity is itself a form of curation and selection. One that is, it should be noted, highly influenced by exogenous factors: search-engine biases, site-aggregator preferences (such as HN), and social media trends and fads, which have their own group and algorithmic foundations.... That said, it's a first cut. - Very little Web content is in ideal long-term storage format as published. In general, I'd strongly prefer tools which stripped page metastructure (headers, sidebars, etc.) which are unrelated to the core article text, ensure that metadata (title, author, publication date, publisher, references/links, and referrers) are preserved and standardised, and that interstitial intrusions (recommendations and advertising) can at least be suppressed, if not eliminated entirely. Hotlinked references should be dereferenced and localised, to prevent future breakage or abuse. Yes, capturing a snapshot view of all the original sins of an article is probably useful for archival, but it's not the preferred form of most content. Guiding principle: the author/publisher/aggregator is an ass. Also known as FYWD, an acronym ambiguously defined as "fine young western dinosuars" or "fuck your web design". Either is equally correct. - Indexing and cataloguing should be separate functions and roles than authoring or publishing. We see this in libraries already, and for a good reason. The interests of the archivist are not the same as that of the author or publisher, and tend to align more strongly with those of the reader, though with added domain-specific knowledge and expertise. Tools such as full-text search (a form of indexing) work to an extent ... until publishers discover that they can game this for benefit, at which point the arms race begins. - Rather than a fixed duration storage, a fixed size cache might be used, with items evicted from it according to some basis. Given the sparsity of indicia, random eviction or a randomised weighted eviction might even be reasonable. Otherwise there are a few characteristics which might be used: 1. Specifically-listed domains or URL patterns. E.g., everything from a specific blog or news site. Everything matching a specific pattern, say, your own commonly-used username(s). 2. Keywords of interest. 3. Weighting by which search patterns have matched content. (Here: local searches against archive.) 4. Revisit counts. How often has a particular item been revisited and re-read? This will all but certainly follow a Zipf function, and probably be low. I'd be surprised if > 1% of archived content is revisited on an annual basis. This might be generalised to revisits against domain, keywords, topics, etc. 5. Indicators when initially visited. E.g., a "save this forever" option. The key difference between present "save page" features is that the auto-archival would have its own standardised structure and metadata. I'm thinking that borrowing heavily from wget's WARC functionality would be a good general basis for this and the archival in general. 6. References between items. E.g., if you've saved a page A and pages B, C, and D reference it, this upweights retention for A. 7. Project / process references. If you'e actively. 8. Specified site mirroring. One might designate specific websites to be mirrored / replicated on an ongoing basis. Typically blogs or news sites, though others might also be included. I don't know if this is overly complex or useful. I think if sufficiently implemented as a coded system it would be useful. On random eviction: this is often used in other caching contexts. It tends to work if the material can be restored from a canonical data store. The Web has proved a not-very-reliable such store. There is the option of incorporating archival to services such as the Internet Archive (this can be automated) or Archive.Today (there are manual steps, though the process can be streamlined, and I've done so for collections of thousands of documents for which I've had a specific interest in preservation).

dredmorbius 4y ago

I've been reading up on library functions and operations, much of it through the US Librarian of Congress's annual report to Congress. (I'd posted a link to the archive of these on HN a few weeks back.) Among questions are: - Curation of archive: what is selected, how, and why? - Cataloguing of archive: how do you find what's within it? What purposes does the catalogue serve? - Weeding of archive: what materials are removed, when, and why? - Classification of archive: how are materials organised (especially a concern where physical storage is necessary) and grouped? - Characteristics of archive: what is included, what's been added or removed, what's been circulated, etc., etc. From the LoC sources there's a lot of discussion of how one such system evolved, and what the rationales driving that evolution were. There's also discussion of what usage patterns and concerns were, at least for a physical-book oriented limited-access (members of Congress and other US Government officials principally, members of the public on a limited basis). As of the early 20th century, it seems that very roughly 1% of the collection actually circulated. I should compare that with more recent data.... Which is prelude to some observations for a next-gen Web / online document personal archive curation: - Web-surfing activity is itself a form of curation and selection. One that is, it should be noted, highly influenced by exogenous factors: search-engine biases, site-aggregator preferences (such as HN), and social media trends and fads, which have their own group and algorithmic foundations.... That said, it's a first cut. - Very little Web content is in ideal long-term storage format as published. In general, I'd strongly prefer tools which stripped page metastructure (headers, sidebars, etc.) which are unrelated to the core article text, ensure that metadata (title, author, publication date, publisher, references/links, and referrers) are preserved and standardised, and that interstitial intrusions (recommendations and advertising) can at least be suppressed, if not eliminated entirely. Hotlinked references should be dereferenced and localised, to prevent future breakage or abuse. Yes, capturing a snapshot view of all the original sins of an article is probably useful for archival, but it's not the preferred form of most content. Guiding principle: the author/publisher/aggregator is an ass. Also known as FYWD, an acronym ambiguously defined as "fine young western dinosuars" or "fuck your web design". Either is equally correct. - Indexing and cataloguing should be separate functions and roles than authoring or publishing. We see this in libraries already, and for a good reason. The interests of the archivist are not the same as that of the author or publisher, and tend to align more strongly with those of the reader, though with added domain-specific knowledge and expertise. Tools such as full-text search (a form of indexing) work to an extent ... until publishers discover that they can game this for benefit, at which point the arms race begins. - Rather than a fixed duration storage, a fixed size cache might be used, with items evicted from it according to some basis. Given the sparsity of indicia, random eviction or a randomised weighted eviction might even be reasonable. Otherwise there are a few characteristics which might be used: 1. Specifically-listed domains or URL patterns. E.g., everything from a specific blog or news site. Everything matching a specific pattern, say, your own commonly-used username(s). 2. Keywords of interest. 3. Weighting by which search patterns have matched content. (Here: local searches against archive.) 4. Revisit counts. How often has a particular item been revisited and re-read? This will all but certainly follow a Zipf function, and probably be low. I'd be surprised if > 1% of archived content is revisited on an annual basis. This might be generalised to revisits against domain, keywords, topics, etc. 5. Indicators when initially visited. E.g., a "save this forever" option. The key difference between present "save page" features is that the auto-archival would have its own standardised structure and metadata. I'm thinking that borrowing heavily from wget's WARC functionality would be a good general basis for this and the archival in general. 6. References between items. E.g., if you've saved a page A and pages B, C, and D reference it, this upweights retention for A. 7. Project / process references. If you'e actively. 8. Specified site mirroring. One might designate specific websites to be mirrored / replicated on an ongoing basis. Typically blogs or news sites, though others might also be included. I don't know if this is overly complex or useful. I think if sufficiently implemented as a coded system it would be useful. On random eviction: this is often used in other caching contexts. It tends to work if the material can be restored from a canonical data store. The Web has proved a not-very-reliable such store. There is the option of incorporating archival to services such as the Internet Archive (this can be automated) or Archive.Today (there are manual steps, though the process can be streamlined, and I've done so for collections of thousands of documents for which I've had a specific interest in preservation).