4 ms·

Hi HN, we’re Joe and JD from Hydra (https://hydra.so/ https://hydra.so/). We're excited to announce our team has added parallel query execution and vectorizatio

by coatue 4y ago

Hi HN, we’re Joe and JD from Hydra (https://hydra.so/ https://hydra.so/). We're excited to announce our team has added parallel query execution and vectorization to columnar storage on Postgres. In our blog you can review the clickbench benchmarks and the method of how we built Hydra. Hydra open source (https://github.com/HydrasDB/hydra https://github.com/HydrasDB/hydra) Starting today, we are now offering 14-day free trials of our cloud managed Hydra databases. Click the 'Get Started for Free" button on (https://hydra.so/ https://hydra.so/) to get one. Power to the Postgres people!

senttoschool 4y ago
Just curious, can this be used together with TimescaleDB?
hartator 4y ago
I am a little confused by what you mean by columnar storage and "in fact, it doesn’t use indexes at all". Does that mean every columns (lines) are auto-indexed? Can you use that strategy for storing things like users?
- wuputah 4y ago
  The metadata can act as a basic form of indexing (or sometimes caching, though Hydra doesn't use metadata to calculate results yet), but it's not an index in the traditional sense. It's used to eliminate stripes and blocks from consideration during a scan. Columnar is not ideal for a `users` table where you want to select and update specific rows, often in very small, quick transactions (OLTP). You would want to continue to use a traditional (heap) table in that case. That's certainly something you can still do with Hydra, and combining both kinds of tables is considered HTAP, and something that is a unique use case of our product. To contrast, columnar is best for "fact" tables -- data about something that happened (thus it does not change) that will be analyzed in an aggregate way. Those might be logs, events, transactions, etc.
  - winrid 4y ago
    Clickhouse works the same way right?
pella 4y ago
Any info for "Columnar storage" for geo data? or PostGIS support?
- wuputah 4y ago
  Yes, Hydra Columnar and PostGIS get along just fine. We've not looked into any PostGIS-specific optimizations yet, but if users run into issues, we'll be happy to investigate.
  - pella 4y ago
    based on https://github.com/HydrasDB/hydra/blob/main/columnar/src/backend/columnar/README.md https://github.com/HydrasDB/hydra/blob/main/columnar/src/bac... the columnar "does not support gist, gin, spgist and brin indexes." :-( and "gist" is important for spatial indexing : http://postgis.net/workshops/postgis-intro/indexing.html http://postgis.net/workshops/postgis-intro/indexing.html
blixtra 4y ago
The project looks very interesting. But I had a look at the license of the Citus source code and it appears to be under an AGPL license and I didn't see an exception for the part of the code that you're including in Hydra. FWIU, AGPL code is not compatible with including in Apache code, although the opposite is compatible. So, I'd be interested to know if I'm understanding this wrong or if there is some license exception I'm not seeing.
- blixtra 4y ago
  I'll answer my own question after doing some more digging. It seems that as the columnar code is a self-contained PostgreSQL extension, you're only using the API which would be fine as there is no linking involved.

by coatue 4y ago

senttoschool 4y ago

Just curious, can this be used together with TimescaleDB?

hartator 4y ago

I am a little confused by what you mean by columnar storage and "in fact, it doesn’t use indexes at all". Does that mean every columns (lines) are auto-indexed? Can you use that strategy for storing things like users?

wuputah 4y ago
The metadata can act as a basic form of indexing (or sometimes caching, though Hydra doesn't use metadata to calculate results yet), but it's not an index in the traditional sense. It's used to eliminate stripes and blocks from consideration during a scan. Columnar is not ideal for a `users` table where you want to select and update specific rows, often in very small, quick transactions (OLTP). You would want to continue to use a traditional (heap) table in that case. That's certainly something you can still do with Hydra, and combining both kinds of tables is considered HTAP, and something that is a unique use case of our product. To contrast, columnar is best for "fact" tables -- data about something that happened (thus it does not change) that will be analyzed in an aggregate way. Those might be logs, events, transactions, etc.
- winrid 4y ago
  Clickhouse works the same way right?

pella 4y ago

Any info for "Columnar storage" for geo data? or PostGIS support?

wuputah 4y ago
Yes, Hydra Columnar and PostGIS get along just fine. We've not looked into any PostGIS-specific optimizations yet, but if users run into issues, we'll be happy to investigate.
- pella 4y ago
  based on https://github.com/HydrasDB/hydra/blob/main/columnar/src/backend/columnar/README.md https://github.com/HydrasDB/hydra/blob/main/columnar/src/bac... the columnar "does not support gist, gin, spgist and brin indexes." :-( and "gist" is important for spatial indexing : http://postgis.net/workshops/postgis-intro/indexing.html http://postgis.net/workshops/postgis-intro/indexing.html

blixtra 4y ago

The project looks very interesting. But I had a look at the license of the Citus source code and it appears to be under an AGPL license and I didn't see an exception for the part of the code that you're including in Hydra. FWIU, AGPL code is not compatible with including in Apache code, although the opposite is compatible. So, I'd be interested to know if I'm understanding this wrong or if there is some license exception I'm not seeing.

blixtra 4y ago
I'll answer my own question after doing some more digging. It seems that as the columnar code is a self-contained PostgreSQL extension, you're only using the API which would be fine as there is no linking involved.