4 ms·

Do you mind expanding on that? What are you suggesting I search for on Google? I genuinely am trying to understand.

Skuzzzy 9y ago
Forgive my brevity, typos, and poor writing as I'm writing this on my phone. He is stating that you would never actually search on your primary storage mediums. You would build your search index off the primary storage medium such that when looking up an article you do not search over the primary medium and only search the index and then once the article is found you read from said primary medium.
- dredmorbius 9y ago
  This, yes.
- Redoubts 9y ago
  Well sure, but then you have to crawl though thousands of pieces of physical medium to load the file you want...
  - sli 9y ago
    Do search indexes typically not include an identifier for a given resource? Seems to me that an index would be almost useless without them, as the match itself is more often than not far less useful than the document that contains it. Unless the information you need happens to be in the surrounding blurb. So yeah, I'm not sure why you'd have to "crawl" through physical media when a search index ought to tell you right where your match is located. Is that not the entire point of indexing and searching?
  - dredmorbius 9y ago
    An index maps a query to a specific item. Say, a file reference within a storage hierarchy or record within a database. (The concepts are fundamentally identical.)
- snowpanda 9y ago
  Thank you for explaining :)
dredmorbius 9y ago
If I were to, say, build a document-management-system-as-a-filesystem,[1] I might approach it something like this: 1. Provide an independent storage tier for the actual content. Say, "stacks". This would store various formats of documents, uniquely identified (say, by a corresponding hash), and maintained independently of access. 2. I'd have an interface tier that would, simply put, be a search interface. The concept of "filesystem as document management" could present this in a familiar, apparently hierarchical fashion, but the hierarchy would actually be searches against various metadata. Say: author, publisher, title, publication date (as range), subject, or keywords. 3. These metadata would be comprised of one or more indices of the actual contents. The indices are what you want on fast storage, in memory if at all possible. Various forms of caching and distribution might provide for this. 4. In order to associate metadata with works and their attributes, you'd need an intake process. This means that as works are onboarded, the would be, effectively, catalogued and classified, with metadata fields supplied. It turns out that there are available metadata repositories for a large class of works (the US Library of Congress and OCLC are two organisations providing same), and it's possible that various distributed, crowdsourced, automated, AI, or similar mechanisms could be used to fill in metadata for works not otherwise categorised. 5. Further capabilities, access, reports, summaries, groupings, workflows, workgroups, projects, security layers, publication, editing, revision, editing, reference, exploration, etc., might be provided by filesystem analogues as possible, other means where not. If the overal system sounds a lot like a library and cataloguing system, that might be because it rather much is. Google's web index operates in much the same way. Primary storage is the origin server of a given URL. The index is maintained and accessed by Google within its own systems. Google typically returns a response in a few hundredths of a second. Retrieval of the referenced source typically takes a number of seconds, or roughly 2-3 orders of magnitude slower. ________________________________ Notes: 1. See: https://redd.it/6bgowu https://redd.it/6bgowu

Forgive my brevity, typos, and poor writing as I'm writing this on my phone. He is stating that you would never actually search on your primary storage mediums. You would build your search index off the primary storage medium such that when looking up an article you do not search over the primary medium and only search the index and then once the article is found you read from said primary medium.

dredmorbius 9y ago
This, yes.
Redoubts 9y ago
Well sure, but then you have to crawl though thousands of pieces of physical medium to load the file you want...
- sli 9y ago
  Do search indexes typically not include an identifier for a given resource? Seems to me that an index would be almost useless without them, as the match itself is more often than not far less useful than the document that contains it. Unless the information you need happens to be in the surrounding blurb. So yeah, I'm not sure why you'd have to "crawl" through physical media when a search index ought to tell you right where your match is located. Is that not the entire point of indexing and searching?
- dredmorbius 9y ago
  An index maps a query to a specific item. Say, a file reference within a storage hierarchy or record within a database. (The concepts are fundamentally identical.)
snowpanda 9y ago
Thank you for explaining :)

dredmorbius 9y ago

If I were to, say, build a document-management-system-as-a-filesystem,[1] I might approach it something like this: 1. Provide an independent storage tier for the actual content. Say, "stacks". This would store various formats of documents, uniquely identified (say, by a corresponding hash), and maintained independently of access. 2. I'd have an interface tier that would, simply put, be a search interface. The concept of "filesystem as document management" could present this in a familiar, apparently hierarchical fashion, but the hierarchy would actually be searches against various metadata. Say: author, publisher, title, publication date (as range), subject, or keywords. 3. These metadata would be comprised of one or more indices of the actual contents. The indices are what you want on fast storage, in memory if at all possible. Various forms of caching and distribution might provide for this. 4. In order to associate metadata with works and their attributes, you'd need an intake process. This means that as works are onboarded, the would be, effectively, catalogued and classified, with metadata fields supplied. It turns out that there are available metadata repositories for a large class of works (the US Library of Congress and OCLC are two organisations providing same), and it's possible that various distributed, crowdsourced, automated, AI, or similar mechanisms could be used to fill in metadata for works not otherwise categorised. 5. Further capabilities, access, reports, summaries, groupings, workflows, workgroups, projects, security layers, publication, editing, revision, editing, reference, exploration, etc., might be provided by filesystem analogues as possible, other means where not. If the overal system sounds a lot like a library and cataloguing system, that might be because it rather much is. Google's web index operates in much the same way. Primary storage is the origin server of a given URL. The index is maintained and accessed by Google within its own systems. Google typically returns a response in a few hundredths of a second. Retrieval of the referenced source typically takes a number of seconds, or roughly 2-3 orders of magnitude slower. ________________________________ Notes: 1. See: https://redd.it/6bgowu https://redd.it/6bgowu