"he made the claim it would be easier to gleen useful data from what we've collected" He's probably right. "Would this fall into the "premature optimization"

"he made the claim it would be easier to gleen useful data from what we've collected" He's probably right. "Would this fall into the "premature optimization" category of common mistakes?" No. I don't believe so. You don't have to optimize your data. In CouchDB you need to convert it to JSON but that's a pretty straight forward process. I wrote a perl script to do that for data I was working it. You can create databases that work on a specific dataset if you want, or create a single database and load it with everything you've got. From there you design "Views" that work with data in the "files" you store data in. When you run the view the first time CouchDB creates a B-tree that is then stored and used to find the data you're working with and return the results based on your criteria. That B-tree is where your data is "optimized" and CouchDB does that for you. The B-tree is updated whenever you add or make a change to your dataset, so you don't have to do anything to "re-optimize" it. This is all fairly easy to do and the learning curve is fairly short. If this is being run in-house I'd recommend developing your app on a Raspberry Pi. You may find that's all you really need. You can setup a CouchDB on a Raspberry Pi with over 250 GB of storage for around $100. (http://wdlabs.wd.com/products/pidrive-compute-centre/ http://wdlabs.wd.com/products/pidrive-compute-centre/). This actually provides a pretty great way to distribute your app and data too, and gives those who have access to modify it themselves. You can develop your app on a Pi and from there if you need more power you can replicate everything you've got to a Cloudant DB, or install CouchDB on Google Cloud or Amazon Cloud or a spin up a DigitalOcean VPS and install CouchDB on it. You've got several great options there. CouchDB has a built-in tool to replicate your data to/from other CouchDBs so this is incredibly easy to do. Pretty much push a button easy. And don't underestimate what you can do with the Raspberry Pi. They run CouchDB just fine. I'd be glad to send you step-by-step install instructions for CouchDB 2.0 (latest version) to help get you started with that.