4 ms·

Last time I checked it was between 40TB - 60TB for just the articles, not including the libgen books. At that time I calculated that it would cost about $1000

Last time I checked it was between 40TB - 60TB for just the articles, not including the libgen books. At that time I calculated that it would cost about $1000 in refurbished 4TB external hard drives. Which is probably not the best way to store it. Or about 2000 Empty Blu-Rays if you want to make searching for an article a nightmare.

deleted 9y ago
[deleted]
dredmorbius 9y ago
Search indices != primary storage. Ask Google about that.
- snowpanda 9y ago
  Do you mind expanding on that? What are you suggesting I search for on Google? I genuinely am trying to understand.
  - Skuzzzy 9y ago
    Forgive my brevity, typos, and poor writing as I'm writing this on my phone. He is stating that you would never actually search on your primary storage mediums. You would build your search index off the primary storage medium such that when looking up an article you do not search over the primary medium and only search the index and then once the article is found you read from said primary medium.
    - dredmorbius 9y ago
      This, yes.
    - Redoubts 9y ago
      Well sure, but then you have to crawl though thousands of pieces of physical medium to load the file you want...
      - sli 9y ago
        Do search indexes typically not include an identifier for a given resource? Seems to me that an index would be almost useless without them, as the match itself is more often than not far less useful than the document that contains it. Unless the information you need happens to be in the surrounding blurb. So yeah, I'm not sure why you'd have to "crawl" through physical media when a search index ought to tell you right where your match is located. Is that not the entire point of indexing and searching?
      - dredmorbius 9y ago
        An index maps a query to a specific item. Say, a file reference within a storage hierarchy or record within a database. (The concepts are fundamentally identical.)
    - snowpanda 9y ago
      Thank you for explaining :)
  - dredmorbius 9y ago
    If I were to, say, build a document-management-system-as-a-filesystem,[1] I might approach it something like this: 1. Provide an independent storage tier for the actual content. Say, "stacks". This would store various formats of documents, uniquely identified (say, by a corresponding hash), and maintained independently of access. 2. I'd have an interface tier that would, simply put, be a search interface. The concept of "filesystem as document management" could present this in a familiar, apparently hierarchical fashion, but the hierarchy would actually be searches against various metadata. Say: author, publisher, title, publication date (as range), subject, or keywords. 3. These metadata would be comprised of one or more indices of the actual contents. The indices are what you want on fast storage, in memory if at all possible. Various forms of caching and distribution might provide for this. 4. In order to associate metadata with works and their attributes, you'd need an intake process. This means that as works are onboarded, the would be, effectively, catalogued and classified, with metadata fields supplied. It turns out that there are available metadata repositories for a large class of works (the US Library of Congress and OCLC are two organisations providing same), and it's possible that various distributed, crowdsourced, automated, AI, or similar mechanisms could be used to fill in metadata for works not otherwise categorised. 5. Further capabilities, access, reports, summaries, groupings, workflows, workgroups, projects, security layers, publication, editing, revision, editing, reference, exploration, etc., might be provided by filesystem analogues as possible, other means where not. If the overal system sounds a lot like a library and cataloguing system, that might be because it rather much is. Google's web index operates in much the same way. Primary storage is the origin server of a given URL. The index is maintained and accessed by Google within its own systems. Google typically returns a response in a few hundredths of a second. Retrieval of the referenced source typically takes a number of seconds, or roughly 2-3 orders of magnitude slower. ________________________________ Notes: 1. See: https://redd.it/6bgowu https://redd.it/6bgowu
userbinator 9y ago
...or about 10 LTO-7 tapes (6TB each --- "15TB" compressed, but I assume the article collection is already compressed), at slightly less cost for the media, and lot more for the drive; but with better reliability and even worse access times...
m-p-3 9y ago
Oh damn, this is way bigger than I imagined.

deleted 9y ago

[deleted]

dredmorbius 9y ago

Search indices != primary storage. Ask Google about that.

snowpanda 9y ago
Do you mind expanding on that? What are you suggesting I search for on Google? I genuinely am trying to understand.
- Skuzzzy 9y ago
  Forgive my brevity, typos, and poor writing as I'm writing this on my phone. He is stating that you would never actually search on your primary storage mediums. You would build your search index off the primary storage medium such that when looking up an article you do not search over the primary medium and only search the index and then once the article is found you read from said primary medium.
  - dredmorbius 9y ago
    This, yes.
  - Redoubts 9y ago
    Well sure, but then you have to crawl though thousands of pieces of physical medium to load the file you want...
    - sli 9y ago
      Do search indexes typically not include an identifier for a given resource? Seems to me that an index would be almost useless without them, as the match itself is more often than not far less useful than the document that contains it. Unless the information you need happens to be in the surrounding blurb. So yeah, I'm not sure why you'd have to "crawl" through physical media when a search index ought to tell you right where your match is located. Is that not the entire point of indexing and searching?
    - dredmorbius 9y ago
      An index maps a query to a specific item. Say, a file reference within a storage hierarchy or record within a database. (The concepts are fundamentally identical.)
  - snowpanda 9y ago
    Thank you for explaining :)
- dredmorbius 9y ago
  If I were to, say, build a document-management-system-as-a-filesystem,[1] I might approach it something like this: 1. Provide an independent storage tier for the actual content. Say, "stacks". This would store various formats of documents, uniquely identified (say, by a corresponding hash), and maintained independently of access. 2. I'd have an interface tier that would, simply put, be a search interface. The concept of "filesystem as document management" could present this in a familiar, apparently hierarchical fashion, but the hierarchy would actually be searches against various metadata. Say: author, publisher, title, publication date (as range), subject, or keywords. 3. These metadata would be comprised of one or more indices of the actual contents. The indices are what you want on fast storage, in memory if at all possible. Various forms of caching and distribution might provide for this. 4. In order to associate metadata with works and their attributes, you'd need an intake process. This means that as works are onboarded, the would be, effectively, catalogued and classified, with metadata fields supplied. It turns out that there are available metadata repositories for a large class of works (the US Library of Congress and OCLC are two organisations providing same), and it's possible that various distributed, crowdsourced, automated, AI, or similar mechanisms could be used to fill in metadata for works not otherwise categorised. 5. Further capabilities, access, reports, summaries, groupings, workflows, workgroups, projects, security layers, publication, editing, revision, editing, reference, exploration, etc., might be provided by filesystem analogues as possible, other means where not. If the overal system sounds a lot like a library and cataloguing system, that might be because it rather much is. Google's web index operates in much the same way. Primary storage is the origin server of a given URL. The index is maintained and accessed by Google within its own systems. Google typically returns a response in a few hundredths of a second. Retrieval of the referenced source typically takes a number of seconds, or roughly 2-3 orders of magnitude slower. ________________________________ Notes: 1. See: https://redd.it/6bgowu https://redd.it/6bgowu

userbinator 9y ago

...or about 10 LTO-7 tapes (6TB each --- "15TB" compressed, but I assume the article collection is already compressed), at slightly less cost for the media, and lot more for the drive; but with better reliability and even worse access times...

m-p-3 9y ago

Oh damn, this is way bigger than I imagined.