4 ms·

Eh, it's not so bad really: we've also entered an age where any programmer, if they choose to do so, can roll their own search engine and build something pretty

Eh, it's not so bad really: we've also entered an age where any programmer, if they choose to do so, can roll their own search engine and build something pretty OK in a reasonable amount of time. That's amazing if you think about it; back in 2000, I could not have imagined building my own Excite. The kind of parsing that Facebook is doing (or planning to do -- I don't use Facebook, so I don't know if it's implemented yet or not) isn't all that hard. I do it with my own toy search engine. Comments like http://news.ycombinator.com/item?id=5099787 http://news.ycombinator.com/item?id=5099787 are compiled from results for searches like, "top submissions with 'hackny'", or "recent submissions from qz.com". I'm currently compiling a huge data set that should let me do neat things like, "front page threads with comments from [user] and [user] from last week" -- or just about any other logical query you can think of. Technology is really damn cool sometimes.

rpm4321 14y ago
Just curious, when you say that it's easy to roll your own web search engine, are you referring to things like Yahoo BOSS and Common Crawl, or are their other OSS projects / data sources out there I'm not aware of yet? It's relevant to an important side project that I'm working on, but I'm new to search, so any links or other info you might have would be greatly appreciated. Thanks in advance.
- oscilloscope 14y ago
  Check out ElasticSearch and Solr. For graph queries like Facebook, check out the query language Gremlin and graph database Titan. These are just starting points. I'm not an expert.
- thaumaturgy 14y ago
  In my case, I initially just built an interface for ThriftDB's HN data; I've recently started building my own data set in MySQL (partly because I wanted to learn how to tune MySQL, partly because I'm starting to expand beyond what's available in ThriftDB). I have a little custom crawler that regularly retrieves the stuff I'm currently after. I've been looking at using Sphinx (http://sphinxsearch.com/docs/current.html#about http://sphinxsearch.com/docs/current.html#about) for full text search capability on that data set, but it's not set up yet. It seems to be the most recommended way to handle text search with a MySQL data store. I wanted to have something that can do a good job of answering questions like, "popular articles about that space mining company from last week" (because that's how my brain works). I probably can't offer any useful information for what you're looking for. edit: Sorry, to more directly answer your question: "Just curious, when you say that it's easy to roll your own web search engine, are you referring to things like Yahoo BOSS and Common Crawl, or are their other OSS projects / data sources out there I'm not aware of yet?" Sometime around 2000, full text search of just the contents of your hard drive was still a really icky problem. Programmers spent a ton of time coming up with reasonable ways to do that. It blows my mind sometimes that now we can fairly easily set up our very own SQL database with just a few commands and a little bit of forethought, and then use any one of a number of programming languages to interface with a standard communication protocol to connect to other servers around the world and then download the data, use fairly decent libraries to parse it all, shove it into our own database, and then quickly search the database. I wasn't really thinking about a particular API or service.

rpm4321 14y ago

Just curious, when you say that it's easy to roll your own web search engine, are you referring to things like Yahoo BOSS and Common Crawl, or are their other OSS projects / data sources out there I'm not aware of yet? It's relevant to an important side project that I'm working on, but I'm new to search, so any links or other info you might have would be greatly appreciated. Thanks in advance.

oscilloscope 14y ago
Check out ElasticSearch and Solr. For graph queries like Facebook, check out the query language Gremlin and graph database Titan. These are just starting points. I'm not an expert.
thaumaturgy 14y ago
In my case, I initially just built an interface for ThriftDB's HN data; I've recently started building my own data set in MySQL (partly because I wanted to learn how to tune MySQL, partly because I'm starting to expand beyond what's available in ThriftDB). I have a little custom crawler that regularly retrieves the stuff I'm currently after. I've been looking at using Sphinx (http://sphinxsearch.com/docs/current.html#about http://sphinxsearch.com/docs/current.html#about) for full text search capability on that data set, but it's not set up yet. It seems to be the most recommended way to handle text search with a MySQL data store. I wanted to have something that can do a good job of answering questions like, "popular articles about that space mining company from last week" (because that's how my brain works). I probably can't offer any useful information for what you're looking for. edit: Sorry, to more directly answer your question: "Just curious, when you say that it's easy to roll your own web search engine, are you referring to things like Yahoo BOSS and Common Crawl, or are their other OSS projects / data sources out there I'm not aware of yet?" Sometime around 2000, full text search of just the contents of your hard drive was still a really icky problem. Programmers spent a ton of time coming up with reasonable ways to do that. It blows my mind sometimes that now we can fairly easily set up our very own SQL database with just a few commands and a little bit of forethought, and then use any one of a number of programming languages to interface with a standard communication protocol to connect to other servers around the world and then download the data, use fairly decent libraries to parse it all, shove it into our own database, and then quickly search the database. I wasn't really thinking about a particular API or service.