3 ms·

Search is a gold mine, and I don't understand why there aren't more people diving in to building niche search engines. Sure you can't really compete with Googl

Search is a gold mine, and I don't understand why there aren't more people diving in to building niche search engines. Sure you can't really compete with Google on size, but there's a lot of nooks and crannies online where you can pick up valuable search traffic around the edges. At least, that's why I'm working on a search engine for financial news at http://Newsley.com/search http://Newsley.com/search. (We're focused on building the crawlers and the index right now. Search is _very_ alpha). After reading this article, I feel validated for a bunch of the decisions that I've been making. I've been running on EC2, but their disk IO is slow as molasses. So, I'm starting to build servers and throw them in my garage. I'll be migrating to garage servers in the next few months. Pretty much everyone I talk to thinks running servers in your garage is a terrible idea, but I can't think of any way else to do this cheaper and still have control over my hardware. It's nice to read that I'm not crazy for thinking this. It was also great to read that on early search engines, the bulk of the work is done by small teams. Being the only dev, at times I think I'm a bit crazy for trying to boostrap a search startup. Again, it was nice to read that it's not all that crazy to try and do it on my own.

JusticeJones 16y ago
I don't mean to knock your project, I think it's cool. However, if you look at the improvements done to the existing search giants it's all about solutions to social problems. Stuff like checking the latest score of a game, finding movie times, the weather, etc. Which to me suggests that the goldmine for modern search development is in creating a glanceable results. Just to be clear when I say 'social' I am referring to a general community, so social problems are really just annoyances that the majority of community members have to deal with. Sorry if that was a bit confusing; I'm not the most articulate person in the world.
ljlolel 16y ago
She specifically says to go with slow disk I/O (not in-memory indices) because the most important thing is not to have to deal with failing servers. This was written in 2004, and the equivalent statement would be that you should not deal with maintaining physical servers but have the cloud handle it so you can focus on algorithms/parallelism/runtime. Also, unrelated to what you wrote, this is 2004, and statements like "people search for words not phrases" are frankly no longer true. Average query length is way up (even before google suggest and instant) and people have been searching for phrases more and more since 2004. She's completely right, but if she were to rewrite this today, in 2010, it would be an even longer article. A single-word search engine would not return acceptable results.
- iamelgringo 16y ago
  re: Cloud Srorage vs Garage data center Elastic Block is NAS . This is why Elastic Block storage is two to three orders of magnitude slower than SSD attached to a server in your garage. 1 EC2 compute unit is analagous to a 1.6 Ghz 2005 Opteron, or performance wise it's similar to a fast Atom processor. I can build a 4 core server with 8 Gigs of RAM and 120GB of SSD and 2 TB of spinning disk for around $600. SSD latencies are low enough that they can be thought of as slow local memory. So, for the cost of a 3 to 4 months of medium / large instance EC2 instances, you can have roughly 8 times the processing power and 2 orders of magnitude more local memory/SSD. By setting up a couple of cheap servers in my garage, I can get as much processing power as 20 to 30 EC2 instances. That means I can spend a lot more time worrying about coding and less worrying about SysOps. re: single word search You are absolutely right. As I said, our search is very much in alpha right now, and I really haven't spent much time at all on it.
Yrlec 16y ago
Have you looked into 80legs.com? It seems like a good service to use if you want to start your own niche search engine (I'm not affiliated with them, just curious).
- iamelgringo 16y ago
  I have. They provide crawlers . We have a curated set of news feeds that we scrape and index. It doesn't really fit what we want to do.

JusticeJones 16y ago

I don't mean to knock your project, I think it's cool. However, if you look at the improvements done to the existing search giants it's all about solutions to social problems. Stuff like checking the latest score of a game, finding movie times, the weather, etc. Which to me suggests that the goldmine for modern search development is in creating a glanceable results. Just to be clear when I say 'social' I am referring to a general community, so social problems are really just annoyances that the majority of community members have to deal with. Sorry if that was a bit confusing; I'm not the most articulate person in the world.

ljlolel 16y ago

She specifically says to go with slow disk I/O (not in-memory indices) because the most important thing is not to have to deal with failing servers. This was written in 2004, and the equivalent statement would be that you should not deal with maintaining physical servers but have the cloud handle it so you can focus on algorithms/parallelism/runtime. Also, unrelated to what you wrote, this is 2004, and statements like "people search for words not phrases" are frankly no longer true. Average query length is way up (even before google suggest and instant) and people have been searching for phrases more and more since 2004. She's completely right, but if she were to rewrite this today, in 2010, it would be an even longer article. A single-word search engine would not return acceptable results.

iamelgringo 16y ago
re: Cloud Srorage vs Garage data center Elastic Block is NAS . This is why Elastic Block storage is two to three orders of magnitude slower than SSD attached to a server in your garage. 1 EC2 compute unit is analagous to a 1.6 Ghz 2005 Opteron, or performance wise it's similar to a fast Atom processor. I can build a 4 core server with 8 Gigs of RAM and 120GB of SSD and 2 TB of spinning disk for around $600. SSD latencies are low enough that they can be thought of as slow local memory. So, for the cost of a 3 to 4 months of medium / large instance EC2 instances, you can have roughly 8 times the processing power and 2 orders of magnitude more local memory/SSD. By setting up a couple of cheap servers in my garage, I can get as much processing power as 20 to 30 EC2 instances. That means I can spend a lot more time worrying about coding and less worrying about SysOps. re: single word search You are absolutely right. As I said, our search is very much in alpha right now, and I really haven't spent much time at all on it.

Yrlec 16y ago

Have you looked into 80legs.com? It seems like a good service to use if you want to start your own niche search engine (I'm not affiliated with them, just curious).

iamelgringo 16y ago
I have. They provide crawlers . We have a curated set of news feeds that we scrape and index. It doesn't really fit what we want to do.