3 ms·
It appears to be the case in a technical workflow sense, from the little I just read of Snowflake, but my proposal would be a much more open system than one und
by lambda_obrien 6y ago
It appears to be the case in a technical workflow sense, from the little I just read of Snowflake, but my proposal would be a much more open system than one under control of a single vendor. It'd be more like a set of standards for interoperation and a common data center so that the data is accessible under one roof. Maybe each entity could do specialized search as a service and the search aggregators would pay by providing some infrastructure to run the crawlers.
I don't personally think any system specification is impossible unless it goes against since mathematical law, so a really fast, distributed query system where there are a few hundred specialized providers for a single query is feasible. Imagine the aggregator does initial analysis to determine the category of search, like programming, news, or restaurant reviews, then sends the users query to a set of specialized providers that supply an index for that category, then fuse the results with some further analysis of the metadata returned. Then the user can also include or exclude the specialized providers at will.
You could also eliminate the aggregator as a service as simply make it a user application on the desktop, allowing for even more user control and maybe caching or something.