4 ms·

I like this idea but my first impression is it sounds like a lot of work to setup and scale. Am I wrong? Secondarily, if this were a service, wow!

by blaines 14y ago

siganakis 14y ago
It does seem like a lot of work to me. One of the problems is servicing two workloads, one for "transactional" processing and another for analysis. For transactional systems you need to be able to change things quickly and consistently. For analysis you need to be able to query lots of data quickly. For decades people have realized that these are two separate workloads, so have built 2 systems, a transactional system (on an RDBMS) and a data warehouse (generally on an RDBMS). Data is then shipped between the 2 in batch jobs. The transactional system is normalized, and the data warehouse is normalized. Within the data warehouse you make denormalized copies of the data that fit the reporting workload required so that as much of the workload is pre-computed as possible. The problem is that as reporting requirements change, you need to modify these pre-computed stores, as they are very heavily tuned for the particular reporting requirement. Building pre-computed stores is generally done in SQL and can be challenging as you are generally shifting a lot of data and you are trusting the RDBMS to get its optimizations right. There is a trend now to use Hadoop for the building of these pre-computed stores (and even to use Hadoop for the entire data warehouse). However, writing map-reduce jobs for queries is cumbersome compared to SQL so your productivity suffers. But you don't have to pay Oracle or IBM for licenses. The key problem is that you need to pre-compute stuff to do fast aggregation, but you can't pre-compute everything. So what you pre-compute is dependent on what your users want, and that changes all the time. So what you want is a system that lets you change what is pre-computed easily and efficiently.

I like this idea but my first impression is it sounds like a lot of work to setup and scale. Am I wrong? Secondarily, if this were a service, wow!

by blaines 14y ago

I like this idea but my first impression is it sounds like a lot of work to setup and scale. Am I wrong? Secondarily, if this were a service, wow!

siganakis 14y ago

It does seem like a lot of work to me. One of the problems is servicing two workloads, one for "transactional" processing and another for analysis. For transactional systems you need to be able to change things quickly and consistently. For analysis you need to be able to query lots of data quickly. For decades people have realized that these are two separate workloads, so have built 2 systems, a transactional system (on an RDBMS) and a data warehouse (generally on an RDBMS). Data is then shipped between the 2 in batch jobs. The transactional system is normalized, and the data warehouse is normalized. Within the data warehouse you make denormalized copies of the data that fit the reporting workload required so that as much of the workload is pre-computed as possible. The problem is that as reporting requirements change, you need to modify these pre-computed stores, as they are very heavily tuned for the particular reporting requirement. Building pre-computed stores is generally done in SQL and can be challenging as you are generally shifting a lot of data and you are trusting the RDBMS to get its optimizations right. There is a trend now to use Hadoop for the building of these pre-computed stores (and even to use Hadoop for the entire data warehouse). However, writing map-reduce jobs for queries is cumbersome compared to SQL so your productivity suffers. But you don't have to pay Oracle or IBM for licenses. The key problem is that you need to pre-compute stuff to do fast aggregation, but you can't pre-compute everything. So what you pre-compute is dependent on what your users want, and that changes all the time. So what you want is a system that lets you change what is pre-computed easily and efficiently.