5 ms·

Here is an unsolicited email I sent last night. I sent it to spinn3r, but any company doing "competitive intelligience", market research, big data, ect can be s

Here is an unsolicited email I sent last night. I sent it to spinn3r, but any company doing "competitive intelligience", market research, big data, ect can be subbed in here for their name, because the browser and search engine wars are starting. The email: Just read the blog post on optimizing the elasticsearch cluster. It sounds like you have a pretty awesome deployment setup. Now, I know competitive intelligience is "hot" and "trendy", I mean, for sure it isn't mobile-first omni-channel blockchain bitcoin disruptive "trendy", but it's prety close. Now, I jest, but what I am about to say next I am dead serious about: your product is a search engine, just let people use it as a goddamn search engine. Now, let me explain. For like probably over a year now I have been making the case that between google, elastic and apache there is just a goddamn open source version of google sitting out there. Would a consumer facing pay for search engine be profitable? idk, is your product profitable? I suspect there is a lot of people who would pay a $9.99 a month to upload and control page rank and not have google own them. Lets assume you were able to capture even 0.01% of googles customers e.g. 1 billion people and got them to pay you like $9.99, then you'd have just shy of $1M MRR, and $11.98M annually. Obviously, you could do some nice caching of resources like stack overflow, reddit, hacker news, and wikipedia and simply store peoples preferences (sort of a heirarchical git like system) and then you would have some pretty swiggity sweet optimizations. You know what might really add value? Having NLP on your own custom tailored information database. I bet people (who search and use the internet daily) would probably see much higher leverage from this than the sentiment analytics you can give them on a competitor. Also, as an aside, you should be mining the competitors customer support channels not only for sentiment but for their inabiliy to service feature requests/complaints. Proabably one of the biggest predictors of why wealthfront lost MS to betterment. Regardless though, if you guys are looking for an expansion strategy or a pivot, this is it. tl;dr. Peter Theil was wrong (no, not just about Clinkle) but it is true that: The next larry page & sergei brin will build a search engine.

ChuckMcM 10y ago
Having "been there, done that, got the t-shirt" I can tell you that no, there currently is no market for an unbiased search engine. We (Blekko) couldn't even get enough people to give up an email address in exchange for more control over their search experience.
- vonklaus 10y ago
  I respect that. The site just has "acquired by IBM watson team" so I don't know. However, it is possible you were too early. Also, it is impertive that the expereince is built inside a browser. Not 100% sure whate happened to blekko (although looks like you guys landed on your feet) however, if anyone is building a search engine (or you yourself are) I will write up a full marketing plan and strategy and product playbook for free, because it is just so goddamn important.
  - greglindahl 10y ago
    I think the only Blekko team member stupid enough to still be interested in building a search engine is me, building search for the Wayback Machine at the Internet Archive. And we have a business plan, we're a charity! :-)
- jdp23 10y ago
  Thanks for weighing in. Disappointing that Blekko didn't work out better ... but hey, at least you have the t-shirt! Do you think that Duck Duck Go's (relative) success shows that there are still possibilities here?
  - ChuckMcM 10y ago
    I liked DuckDuckGo's approach, they have really benefited from a user first strategy. That said, it has been interesting how the cost of compute and storage going down has led to such a large number of non-pages on the web. I was reading a paper which estimated that there were probably 10 trillion web documents reachable on the web, and fewer than 10 billion of them were "meaningful" (meaning they had accurate and unique information). If their estimates are correct, literally 99.9% of the web is crap pages. That is a huge discrimination problem.
    - vonklaus 10y ago
      do you have a link? this fits with my hypothesis, and would be interested to read it.
nostrademons 10y ago
FWIW, having worked in Google Search before and now working with the open-source/IaaS big data equivalents, there's no comparison. Elasticsearch is light years behind where Google ranking is, and the way the APIs are setup and the service is architected, it's doubtful you could ever use it to build a system that rivals Google. Some of the key signals that Google uses just require a different data organization than all the open-source tools support out of the box.
- vonklaus 10y ago
  You have to understand that google is solving a much harder problem, but as a user, a personal solution is much easier to develop for and has higher leverage. If you gave me the ability to curate resources and browse sites I was interested in and upload pools of information as seed data, I would be able to tell (and build) the initial training set much much easier than google. I only have to be correct for my own use case (with nearly 28 years experience being myself) and google has to be correct for everyone with (at best) 18 years of search results, of which you have to back out what the user is searching for. e.g. If I look for George Washingtons birthday I could look for information about the person george washington, the actual query george washington birthday, where I might find the data wikipedia:george washington, or an event that could contain it preseidents day. I know both what my query is, and what a successful answer to my query is (at least the format of an acceptable return). Google has to infer that a query like presidents day and then a query like george washington are related and a successful result was the resource: wikipedia, because it contained george washington's birthday. The point is: * these examples are contrived. However, a person can be given the tooling to organize their own data & potential data. * a person will have insight into what they want, almost certainly better than google. * if they are wrong/the resources hasn't been cached for them, well, the personalized engine can just simply default to google if no results exist and return them * privacy. edit: not only this but having every site I visit (if I wanted this and private & personally encrypted) available and searchable is hugely valuable. Google receives takedown requests, resources move, or I forgot how I received them. Caching/downloading/storing video, images and web results and having them remain private and searchable is pretty much reason enough for me to tell you to take my money.,
yongjik 10y ago
> Lets assume you were able to capture even 0.01% of googles customers ... Reminds me of this story: http://www.musicthinktank.com/blog/and-if-only-1-of-those-people.html http://www.musicthinktank.com/blog/and-if-only-1-of-those-pe...
- vonklaus 10y ago
  you aren't wrong. It was an off the cuff email I sent at 3am, it was purely a rediculous top down analysis comletely devoid of research, simply to ilustrate a point. The point is, if you were able to get a million people to use your service and pay you $9.99, you would be making decent money. This is important, because there is a strong revenue stream tied to a service you provide whereas many companies struggle to get to 1 million users which if achieved, will complicate engineering and increase costs. After those challeneges are faced, well, then they need to convert those users into some sort of payment scheme...

ChuckMcM 10y ago

Having "been there, done that, got the t-shirt" I can tell you that no, there currently is no market for an unbiased search engine. We (Blekko) couldn't even get enough people to give up an email address in exchange for more control over their search experience.

vonklaus 10y ago
I respect that. The site just has "acquired by IBM watson team" so I don't know. However, it is possible you were too early. Also, it is impertive that the expereince is built inside a browser. Not 100% sure whate happened to blekko (although looks like you guys landed on your feet) however, if anyone is building a search engine (or you yourself are) I will write up a full marketing plan and strategy and product playbook for free, because it is just so goddamn important.
- greglindahl 10y ago
  I think the only Blekko team member stupid enough to still be interested in building a search engine is me, building search for the Wayback Machine at the Internet Archive. And we have a business plan, we're a charity! :-)
jdp23 10y ago
Thanks for weighing in. Disappointing that Blekko didn't work out better ... but hey, at least you have the t-shirt! Do you think that Duck Duck Go's (relative) success shows that there are still possibilities here?
- ChuckMcM 10y ago
  I liked DuckDuckGo's approach, they have really benefited from a user first strategy. That said, it has been interesting how the cost of compute and storage going down has led to such a large number of non-pages on the web. I was reading a paper which estimated that there were probably 10 trillion web documents reachable on the web, and fewer than 10 billion of them were "meaningful" (meaning they had accurate and unique information). If their estimates are correct, literally 99.9% of the web is crap pages. That is a huge discrimination problem.
  - vonklaus 10y ago
    do you have a link? this fits with my hypothesis, and would be interested to read it.

nostrademons 10y ago

FWIW, having worked in Google Search before and now working with the open-source/IaaS big data equivalents, there's no comparison. Elasticsearch is light years behind where Google ranking is, and the way the APIs are setup and the service is architected, it's doubtful you could ever use it to build a system that rivals Google. Some of the key signals that Google uses just require a different data organization than all the open-source tools support out of the box.

vonklaus 10y ago
You have to understand that google is solving a much harder problem, but as a user, a personal solution is much easier to develop for and has higher leverage. If you gave me the ability to curate resources and browse sites I was interested in and upload pools of information as seed data, I would be able to tell (and build) the initial training set much much easier than google. I only have to be correct for my own use case (with nearly 28 years experience being myself) and google has to be correct for everyone with (at best) 18 years of search results, of which you have to back out what the user is searching for. e.g. If I look for George Washingtons birthday I could look for information about the person george washington, the actual query george washington birthday, where I might find the data wikipedia:george washington, or an event that could contain it preseidents day. I know both what my query is, and what a successful answer to my query is (at least the format of an acceptable return). Google has to infer that a query like presidents day and then a query like george washington are related and a successful result was the resource: wikipedia, because it contained george washington's birthday. The point is: * these examples are contrived. However, a person can be given the tooling to organize their own data & potential data. * a person will have insight into what they want, almost certainly better than google. * if they are wrong/the resources hasn't been cached for them, well, the personalized engine can just simply default to google if no results exist and return them * privacy. edit: not only this but having every site I visit (if I wanted this and private & personally encrypted) available and searchable is hugely valuable. Google receives takedown requests, resources move, or I forgot how I received them. Caching/downloading/storing video, images and web results and having them remain private and searchable is pretty much reason enough for me to tell you to take my money.,

yongjik 10y ago

> Lets assume you were able to capture even 0.01% of googles customers ... Reminds me of this story: http://www.musicthinktank.com/blog/and-if-only-1-of-those-people.html http://www.musicthinktank.com/blog/and-if-only-1-of-those-pe...

vonklaus 10y ago
you aren't wrong. It was an off the cuff email I sent at 3am, it was purely a rediculous top down analysis comletely devoid of research, simply to ilustrate a point. The point is, if you were able to get a million people to use your service and pay you $9.99, you would be making decent money. This is important, because there is a strong revenue stream tied to a service you provide whereas many companies struggle to get to 1 million users which if achieved, will complicate engineering and increase costs. After those challeneges are faced, well, then they need to convert those users into some sort of payment scheme...