4 ms·

Have problems with PostgreSQL? Try using Hive (2015)

9 points by aargh_aargh 5y ago

debarshri 5y ago
This makes me nostalgic of time we were on hadoop stack back in the day. Eco-system was thriving, hadoop, pig, hbase, hive, mahout. Everyday there was a new tool with an animal name released. Consultancies and VCs were bullish on the data tools. Even cloudera had a tagline that said - "Data is the new bacon". There were "BigData" experts in every organisation. Hortonworks was relatively new and was becoming popular. Somehow, all of them just disappeared in a span of 5 years. The market just shrunk or rather disappeared. Few survived - confluence, databricks and datastax. Today, I see the same trend in kubernetes space. The eco-system is growing even faster than hadoop's eco-system. I have a feeling that at one point there will be realization that not everyone will need kubernetes and the market will start contracting and consolidating.
- Uberphallus 5y ago
  Hi, working for major telco here. I think the realisation is that most data workloads aren't really big data, which people already knew back in 2015, but the same people will overengineer things just to learn stuff. The fad moved onto something else, I agree. But my department is a giant, just-ETL, everything-requires-a-cluster kind of department. More than half of our business runs on Hadoop, for a reason. The Hadoop ecosystem is a bit like K8s. If it feels too complicated for your purpose, it probably is. But with all its warts, when your data is big enough, either you use the Hadoop ecosystem or you reinvent significant parts of the wheel.
  - debarshri 5y ago
    What tools and stack do you use in your organization?
    - Uberphallus 5y ago
      From major technologies around Hadoop we use Hive, Oozie, Spark... it's easier to say the ones we don't use: HBase.
jstx1 5y ago
> So I had this PostgreSQL database that was getting a bit too big At what scale and for what kinds of applications and use cases does this really happen?
- debarshri 5y ago
  Back in the day, clustering postgres was pain. The focus is mostly on the batch processes here. Use case primarily is ETL processes and data warehousing
mypastself 5y ago
The only problem I have with PostgreSQL is the lack of really good client software, now that pgAdmin is getting less usable with each new update. DBeaver and HeidiSQL are acceptable replacements, but they’re not specifically designed with Postgres in mind. I feel like the pgAdmin3 desktop application was the peak version of the tool, and it’s been on the decline ever since.
- zertrin 5y ago
  I also feel the same. Unfortunately there's no way back for pgAdmin. But also no way forward to look for. I am convinced the Web based version won't be able to match pgAdmin3 usability ever. I just forget about it and use DBeaver instead.
- cosmotic 5y ago
  IDEA and company have pretty decent DB tools.

9 points by aargh_aargh 5y ago

This makes me nostalgic of time we were on hadoop stack back in the day. Eco-system was thriving, hadoop, pig, hbase, hive, mahout. Everyday there was a new tool with an animal name released. Consultancies and VCs were bullish on the data tools. Even cloudera had a tagline that said - "Data is the new bacon". There were "BigData" experts in every organisation. Hortonworks was relatively new and was becoming popular. Somehow, all of them just disappeared in a span of 5 years. The market just shrunk or rather disappeared. Few survived - confluence, databricks and datastax. Today, I see the same trend in kubernetes space. The eco-system is growing even faster than hadoop's eco-system. I have a feeling that at one point there will be realization that not everyone will need kubernetes and the market will start contracting and consolidating.

Uberphallus 5y ago
Hi, working for major telco here. I think the realisation is that most data workloads aren't really big data, which people already knew back in 2015, but the same people will overengineer things just to learn stuff. The fad moved onto something else, I agree. But my department is a giant, just-ETL, everything-requires-a-cluster kind of department. More than half of our business runs on Hadoop, for a reason. The Hadoop ecosystem is a bit like K8s. If it feels too complicated for your purpose, it probably is. But with all its warts, when your data is big enough, either you use the Hadoop ecosystem or you reinvent significant parts of the wheel.
- debarshri 5y ago
  What tools and stack do you use in your organization?
  - Uberphallus 5y ago
    From major technologies around Hadoop we use Hive, Oozie, Spark... it's easier to say the ones we don't use: HBase.

jstx1 5y ago

> So I had this PostgreSQL database that was getting a bit too big At what scale and for what kinds of applications and use cases does this really happen?

debarshri 5y ago
Back in the day, clustering postgres was pain. The focus is mostly on the batch processes here. Use case primarily is ETL processes and data warehousing

mypastself 5y ago

The only problem I have with PostgreSQL is the lack of really good client software, now that pgAdmin is getting less usable with each new update. DBeaver and HeidiSQL are acceptable replacements, but they’re not specifically designed with Postgres in mind. I feel like the pgAdmin3 desktop application was the peak version of the tool, and it’s been on the decline ever since.

zertrin 5y ago
I also feel the same. Unfortunately there's no way back for pgAdmin. But also no way forward to look for. I am convinced the Web based version won't be able to match pgAdmin3 usability ever. I just forget about it and use DBeaver instead.
cosmotic 5y ago
IDEA and company have pretty decent DB tools.