3 ms·

In almost all applications using Postgres (or virtually any other decent RDBMS system) is the right technical choice. But (and this may be cultural) wearing my

by sgt101 11y ago

In almost all applications using Postgres (or virtually any other decent RDBMS system) is the right technical choice. But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. The why is this - data and access to data is now trumping applications for business value. Where as I can run almost any of my applications on Postgres (almost) and many of them can co-habit on one server relatively happily, I cannot run all of them on it. This means that I will end up moving data from one of them to the others, but which? Alternatively I will shift the data from the RDBMS to Hadoop (or another data warehouse solution if I am working in oil-and-gas-three-years-ago or a hedge-fund-or-something now) where the insight that I am being kicked for daily can be extracted by combining the data product of 20 systems. Or we can use Hadoop as the core of the data system and feed RDBMS slaves (or use Impala sometimes). This has the benefit that the data is managed and clean (ho ho ho) on the core system and we can move at the speed of security checking and data understanding rather than the speed of budgets for writing and running data extracts.

jldugger 11y ago
> But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. Google's F1 paper makes it clear that they were running Adwords, their most (maybe only) important product, on MySQL on Jan 1 2012. GFS and Big Table were not enough, and until Hadoop provides a robust ACID compliant data store, you should probably keep your pgsql books around.
- sgt101 11y ago
  Agree - there's no doubt that where transactions are needed you need something that can manage transactions (in the particular way you need them managed). I actually think that transactions on Hadoop should be strictly limited to data import a-la HIVE but my arguement is that the enterprise data master is best on Hadoop rather in the slave systems.
IndianAstronaut 11y ago
>But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop This has been my experience as well. Worked on a data warehousing project that eventually had to get moved to Hadoop because management underestimated the scale of the data. We could have saved a lot of pain if we just started on Hadoop.

by sgt101 11y ago

jldugger 11y ago

> But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop. Google's F1 paper makes it clear that they were running Adwords, their most (maybe only) important product, on MySQL on Jan 1 2012. GFS and Big Table were not enough, and until Hadoop provides a robust ACID compliant data store, you should probably keep your pgsql books around.

sgt101 11y ago
Agree - there's no doubt that where transactions are needed you need something that can manage transactions (in the particular way you need them managed). I actually think that transactions on Hadoop should be strictly limited to data import a-la HIVE but my arguement is that the enterprise data master is best on Hadoop rather in the slave systems.

IndianAstronaut 11y ago

>But (and this may be cultural) wearing my "where are we in 2 years" hat makes me push for Hadoop This has been my experience as well. Worked on a data warehousing project that eventually had to get moved to Hadoop because management underestimated the scale of the data. We could have saved a lot of pain if we just started on Hadoop.