2 ms·
I don't disagree with the premise, but given that nobody can afford to make the actual 'engine' part of a search engine other than Microsoft, Google, Yandex, an
by ChuckMcM 9y ago
I don't disagree with the premise, but given that nobody can afford to make the actual 'engine' part of a search engine other than Microsoft, Google, Yandex, and Baidu you're pretty much stuck with having your queries ending up at one of the four anyway.
Masking front ends are ok but not any more effective than say using Tor to route your search requests.
Today this suggestion, you should use a private search engine, is equivalent in my mind to you should only use cash to buy things. When you have a panoptic view of both, digital and commercial activity the ability to create a reconstruction of what you have been doing or thinking through meta-data analysis becomes quite difficult to avoid.
At this point having your browser spam with other searches as a fuzzing technique would be effective.
I suggested (only half joking) that a fundable startup might be a system where subscribers send in their license plate and for $10/month or something every month they will get an envelope with one, two, or three other license plate symbols printed on magnetic material to stick to your car. It doesn't have to look like a license plate to you and me, only to an automated license plate reader which leaves a lot of room for avoiding state laws about having multiple plates.
The idea being that members plates will be thoroughly mixed with a bunch of false hits in license plate reader data bases everywhere making use of that data impractical for your enemies.
- greglindahl 9y agoOnly paying cash for privacy reasons is a good metaphor: not only is it kind of a pain to carry cash & not run out, but some stores have stopped taking cash. For example, my favorite coffee shop on University Ave in Palo Alto :-( BTW the license plate thing has been done by my friends with store loyalty cards, but after a while everyone got bored doing it.
- hux_ 9y agoOffline search is the obvious alternative. Every cell in your body has its own data and search engine. It doesn't need a Google, Microsoft, Yandex, and Baidu. How come? Downloading a dump of Wikipedia or Stackoverflow and indexing it however you like on mid range hardware is trivial today. Look at what Kiwix or Zeal(offline doc browser) do today. It's just a matter of time before people start packaging up data and indexes customized to your info needs downloadable like a song from iTunes.
- ChuckMcM 9y agoHaving built a search engine I can tell you that your minimum index size is on the order of 5 billion documents. You can back fill long tail searches with one of the big players. For minimum hits on the backfill 10 billion documents are better. To get 10 billion high quality documents you will want a “crawl frontier” of probably 500 billion URIs, which, given the power log rule of data you will need to actually crawl about 20% of those every 7 days. Of course you can hold much fewer pages in your index if you are not particularly broad in your searches. The gotcha there is that when you need that thing you don’t normally need, you either have to go online or go without. Not impossible of course, just that the scale may be larger than you expect.
- hux_ 9y agoThe value of the long tail has been oversold I feel. My estimate based on my own usage for the past year and a half or so, of curating my own local indexes is about 30-40 million docs. This is the equivalent of having your own personal Library of Alexandria (text and images no video). I don't have numbers but I work offline a lot and my guess is 70-80% of my queries are probably getting satisfied by my local dumps.
- ChuckMcM 9y agoNot disagreeing with you, I was just trying to connect that the index size is a function of the breadth of information. For example, I have digitized roughly 300 volumes (books) which collectively represent about 100,000 pages of information. That collection which represents a big chunk of my originally print reference library creates a relatively small n-gram index of about 22 GB. But it is a small sample of generally available reference material and doesn't include the 22 years of digitized Scientific American articles (much harder to parse out for indexing when starting with the PDF form). But it still answers a lot of reference queries quickly and accurately for things I am interested in. For things that I become interested in and have yet to have started curating a set of references for, its worthless. As a result my experience is that the closer I get to my long term interests the more likely I am to find something in my library to answer the question, things that are more temporal (news, new research) are not there at all generally, and things that are only now of interest are similarly not represented. The thing that Search engines do so remarkably well is that they cover a very wide swath of interesting material preemptively. To host that locally would be a more significant effort for me.