3 ms·

I completely agree. However I think there is also plenty of room for improvement in the ETL process that doesn't necessarily mean a one-stop solution. For exam

by rpedela 12y ago

I completely agree. However I think there is also plenty of room for improvement in the ETL process that doesn't necessarily mean a one-stop solution. For example, the common denominator data format for many use cases (not all) is a relational database. Why isn't there something that can grab data from anywhere in any common format and import it into a relational database with only a couple commands or mouse clicks? Right now the situation is that you either have to pay a lot of money for such a tool which usually doesn't work very well or you piece together the various data conversion tools with scripts. There is of course a lot of other steps in the ETL process, but that alone would save a lot of time.

kiyoto 12y ago
>However I think there is also plenty of room for improvement in the ETL process that doesn't necessarily mean a one-stop solution. I agree with you here as well. I recently wrote a blog article about this, at least for log data > http://www.fluentd.org/blog/unified-logging-layer http://www.fluentd.org/blog/unified-logging-layer
- rpedela 12y ago
  Very cool! I guess my particular ETL complaint above is solved for log data. :)
akbar501 12y ago
> Why isn't there something that can grab data from anywhere in any common format and import it into a relational database IME, the issue is not tooling so much as dirty data. Even if a tool is a perfect solution, the data is (in the majority of companies) so dirty and inconsistently sparse that automated tooling breaks down.
- rpedela 12y ago
  Absolutely. I was trying to make a larger point that often the tooling for the stuff that can be easily automated or semi-automated is currently sub-par. If we fixed that tooling as a community then that would reduce the majority of the "janitorial work" to fixing dirty data which is definitely a harder problem. In other words, I think there is still a lot of low-hanging fruit in the ETL process.

by rpedela 12y ago

kiyoto 12y ago

>However I think there is also plenty of room for improvement in the ETL process that doesn't necessarily mean a one-stop solution. I agree with you here as well. I recently wrote a blog article about this, at least for log data > http://www.fluentd.org/blog/unified-logging-layer http://www.fluentd.org/blog/unified-logging-layer

rpedela 12y ago
Very cool! I guess my particular ETL complaint above is solved for log data. :)

akbar501 12y ago

> Why isn't there something that can grab data from anywhere in any common format and import it into a relational database IME, the issue is not tooling so much as dirty data. Even if a tool is a perfect solution, the data is (in the majority of companies) so dirty and inconsistently sparse that automated tooling breaks down.

rpedela 12y ago
Absolutely. I was trying to make a larger point that often the tooling for the stuff that can be easily automated or semi-automated is currently sub-par. If we fixed that tooling as a community then that would reduce the majority of the "janitorial work" to fixing dirty data which is definitely a harder problem. In other words, I think there is still a lot of low-hanging fruit in the ETL process.