I didn't necessarily mean scalability in terms of just compute throughput; I was also talking about sustained productivity, though the two are related. Abstrac

I didn't necessarily mean scalability in terms of just compute throughput; I was also talking about sustained productivity, though the two are related. Abstraction does let you build faster if done correctly, but in my experience I've never seen a project turn out a good framework that started framework first. Rails was abstracted out of Basecamp. MapReduce was abstracted out of Google's experience with a huge mess of scripts for running batch jobs, with some theory on top. I'm working with a similar issue (a huge amount of base data, relatively independent of users) and so have experienced the big-data-from-day-one problem. I started by writing a bunch of stuff to make the "heavy lifting" easier. I came out of that experience with the lesson that when the data is big enough, abstractions break down because you do care about the properties of the underlying data stores. On InnoDB/XtraDB, DELETEs cause an in-memory copy of the table to be created for MVCC, and if any INSERTs commit in that interval you'll get sporadic, hard-to-debug timeouts on 30GB tables, even if the machine has 96G memory. If you're making recommendations online and need to do set intersections to apply a topic model, you absolutely do care about the underlying data store. Code abstractions obscure this, are probably not the right ones anyway, and therefore end up being worked around when you're trying to debug these things.