I have started using Impala/Parquet in combination with Solr for real-time/batch analytics. Impala is very fast - it basically bypasses the entire Hadoop stack

I have started using Impala/Parquet in combination with Solr for real-time/batch analytics. Impala is very fast - it basically bypasses the entire Hadoop stack to perform its own distributed processing and IO, and Parquet is a great data-at-rest format. I have found the best way to update data holdings is by landing data in Avro format (which is a great streaming format as it carries its schema around with it), referencing it as an external table, then scheduling an Impala query to convert the current Avro bucket into Parquet. I am also looking at using Spark Streaming for real-time map/reduce style processing to value add to the data flowing into Solr/Impala. Even with a moderate sized cluster I feel like I have a lot of processing power at my fingertips, and it all just works. Hadoop is really starting to shine.