4 ms·

It was recommended by our data scientist as he made the claim it would be easier to gleen useful data from what we've collected. I know nothing about this, so I

It was recommended by our data scientist as he made the claim it would be easier to gleen useful data from what we've collected. I know nothing about this, so I have no opinion on it at all. It was also proposed to prevent increased server load as a result of not denormalizing any data in our schema. Granted, we haven't started building yet, as this is all preliminary. Would this fall into the "premature optimization" category of common mistakes?

oblib 10y ago
Go take a look at IMB's Cloudant. (https://cloudant.com https://cloudant.com) They have some examples there demonstrate how fast it is at digesting data from 100s of 1000s of records. It has to do with how the data from searches are stored in a B-tree. I won't pretend to know the science behind that but I can tell you it's impressively fast and easy to achieve. And you can use PouchDB to create the client side interface and plug it into a CouchDB server you set up or IBM's Cloudant, or both for that matter, with just a line or two of code. So you really don't even need to setup a database server to get started on you application code. You can create a Cloudant DB and load it with data and get right to work on your client.
itamarst 10y ago
Definitely seems like premature optimization. Unless you know you're scaling up a lot worry about it later... and then you can separate the different kinds of data into different databases for different uses.
oblib 10y ago
"he made the claim it would be easier to gleen useful data from what we've collected" He's probably right. "Would this fall into the "premature optimization" category of common mistakes?" No. I don't believe so. You don't have to optimize your data. In CouchDB you need to convert it to JSON but that's a pretty straight forward process. I wrote a perl script to do that for data I was working it. You can create databases that work on a specific dataset if you want, or create a single database and load it with everything you've got. From there you design "Views" that work with data in the "files" you store data in. When you run the view the first time CouchDB creates a B-tree that is then stored and used to find the data you're working with and return the results based on your criteria. That B-tree is where your data is "optimized" and CouchDB does that for you. The B-tree is updated whenever you add or make a change to your dataset, so you don't have to do anything to "re-optimize" it. This is all fairly easy to do and the learning curve is fairly short. If this is being run in-house I'd recommend developing your app on a Raspberry Pi. You may find that's all you really need. You can setup a CouchDB on a Raspberry Pi with over 250 GB of storage for around $100. (http://wdlabs.wd.com/products/pidrive-compute-centre/ http://wdlabs.wd.com/products/pidrive-compute-centre/). This actually provides a pretty great way to distribute your app and data too, and gives those who have access to modify it themselves. You can develop your app on a Pi and from there if you need more power you can replicate everything you've got to a Cloudant DB, or install CouchDB on Google Cloud or Amazon Cloud or a spin up a DigitalOcean VPS and install CouchDB on it. You've got several great options there. CouchDB has a built-in tool to replicate your data to/from other CouchDBs so this is incredibly easy to do. Pretty much push a button easy. And don't underestimate what you can do with the Raspberry Pi. They run CouchDB just fine. I'd be glad to send you step-by-step install instructions for CouchDB 2.0 (latest version) to help get you started with that.

oblib 10y ago

Go take a look at IMB's Cloudant. (https://cloudant.com https://cloudant.com) They have some examples there demonstrate how fast it is at digesting data from 100s of 1000s of records. It has to do with how the data from searches are stored in a B-tree. I won't pretend to know the science behind that but I can tell you it's impressively fast and easy to achieve. And you can use PouchDB to create the client side interface and plug it into a CouchDB server you set up or IBM's Cloudant, or both for that matter, with just a line or two of code. So you really don't even need to setup a database server to get started on you application code. You can create a Cloudant DB and load it with data and get right to work on your client.

itamarst 10y ago

Definitely seems like premature optimization. Unless you know you're scaling up a lot worry about it later... and then you can separate the different kinds of data into different databases for different uses.

oblib 10y ago

"he made the claim it would be easier to gleen useful data from what we've collected" He's probably right. "Would this fall into the "premature optimization" category of common mistakes?" No. I don't believe so. You don't have to optimize your data. In CouchDB you need to convert it to JSON but that's a pretty straight forward process. I wrote a perl script to do that for data I was working it. You can create databases that work on a specific dataset if you want, or create a single database and load it with everything you've got. From there you design "Views" that work with data in the "files" you store data in. When you run the view the first time CouchDB creates a B-tree that is then stored and used to find the data you're working with and return the results based on your criteria. That B-tree is where your data is "optimized" and CouchDB does that for you. The B-tree is updated whenever you add or make a change to your dataset, so you don't have to do anything to "re-optimize" it. This is all fairly easy to do and the learning curve is fairly short. If this is being run in-house I'd recommend developing your app on a Raspberry Pi. You may find that's all you really need. You can setup a CouchDB on a Raspberry Pi with over 250 GB of storage for around $100. (http://wdlabs.wd.com/products/pidrive-compute-centre/ http://wdlabs.wd.com/products/pidrive-compute-centre/). This actually provides a pretty great way to distribute your app and data too, and gives those who have access to modify it themselves. You can develop your app on a Pi and from there if you need more power you can replicate everything you've got to a Cloudant DB, or install CouchDB on Google Cloud or Amazon Cloud or a spin up a DigitalOcean VPS and install CouchDB on it. You've got several great options there. CouchDB has a built-in tool to replicate your data to/from other CouchDBs so this is incredibly easy to do. Pretty much push a button easy. And don't underestimate what you can do with the Raspberry Pi. They run CouchDB just fine. I'd be glad to send you step-by-step install instructions for CouchDB 2.0 (latest version) to help get you started with that.