If you are looking to build something similar, check out the open source elasticsearch (http://www.elasticsearch.com http://www.elasticsearch.com). It was built

If you are looking to build something similar, check out the open source elasticsearch (http://www.elasticsearch.com http://www.elasticsearch.com). It was built with the mentioned architecture in mind (sorry for the "plug", I am the creator of elasticsearch).