I think what the parent meant is that there are now tools available to implement the algorithms used by Amazon et al for recommendations. It isn't about having

I think what the parent meant is that there are now tools available to implement the algorithms used by Amazon et al for recommendations. It isn't about having more users, it's about being able to keep track of great deal more data and use that data at great deal more points in the applications. That's what Hadoop, NoSQL et al allow you to do (e.g. Voldemort's and HBase's ability efficiently load and serve data prepared in Hadoop; afaik Cassandra has Hadoop integration now as well). Now even if you are okay with low throughput (you don't see yourself become a major Internet destination), you can still benefit from non-relational approaches. Implementing a recommender system or PageRank on top of a relational database is simply painful and requires expertise in that system's SQL dialect (someone who's done this on top of Oracle won't be able to do this -- in the same amount of time -- on top of MySQL or Postgres). With Hadoop you can start with a Perl or Python script (and make use of libraries like scipy or pysvm) and then gradually re-write the performance critical parts in Java. Even if you aren't and will never get to Facebook's or Amazon's scale, you're now able to build intelligent applications on top of your data. You can also sleep sound at night (i.e., not in front of a terminal tailing your logs or watching ganglia) knowing that a load spike (which could be start of a new financial quarter, holiday shopping season; something as positive as a NYTimes article or -- much more likely -- as negative as a DDOS attack) won't bring you to an outage. The algorithms themselves, however, are very difficult and require a background in machine learning; that is even to use an existing implementation of these algorithms (e.g. Apache Mahout). I am not an expert in these topics. I took a graduate course in machine learning, but that only demonstrated to me how little I know and how much more I have left to learn. It's this expertise that coupled with ability to implement a scalable infrastructure with open source/in-house developed software (instead of relying on commercial vendors) that makes your a business idea (you know, the one that the "business guys" think they can hire bunch of MSCEs to build in ASP.NET/J2EE/PHP/RoR) defensible from some "kid" in Harvard dorms (especially if that kid declined a $1mm starting bonus to work for Microsoft instead of going to Harvard). Don't get me wrong. SQL is an amazingly example of a domain specific language; RDBMs are wonderful works of software engineering. SQL RDBMs are great for many applications (e.g., ERP systems and their functional equivalents like the university registration system patio11 mentioned). Yet they're not made to solve the kinds of problems/create the kind of applications that simply didn't exist when they were created. The fact that they can at times be used for this doesn't mean they should: I can write Russian with Latin instead of Cyrillic Alphabet (or write Lisp in vi instead of emacs), but it's a lot better to use the correct tool.