The article discusses the challenge of supplanting entrenched software such as Hadoop. I am actually a bit more optimistic about the ability to swap out Hadoop

The article discusses the challenge of supplanting entrenched software such as Hadoop. I am actually a bit more optimistic about the ability to swap out Hadoop with technologies like Spark. At Twitter we don't program using Hadoop directly; we mostly use either Scalding or Pig, languages that compile down to Hadoop code. https://dev.twitter.com/blog/scalding https://dev.twitter.com/blog/scalding http://www.slideshare.net/kevinweil/hadoop-pig-and-twitter-nosql-east-2009 http://www.slideshare.net/kevinweil/hadoop-pig-and-twitter-n... I believe this is how many other companies use Hadoop as well. The benefit here is that it's possible to write new backends for Pig and Scalding that compile down to Spark or anything else. And then you have backwards compatibility with all your old big-data code.