9 ms·
Show HN: Hydra 1.0 – open-source column-oriented Postgres
hi hn, hydra ceo here
hydra is an open source, column-oriented postgres. you can set up remarkably fast aggregates on your project in minutes to query billions of rows instantly.
postgres is great, but aggregates can take minutes to hours to return results on large data sets. long-running analytical queries hog database resources and degrade performance. use hydra to run much faster analytics on postgres without making code changes. data is automatically loaded into columnar format and compressed. connect to hydra with your preferred postgres client (psql, dbeaver, etc).
following 4 months of development on hydra v0.3.0-alpha, our team is proud to share our first major version release. hydra 1.0 is under active development, but ready for use and feedback. we’re aiming to release 1.0 into general availability (ga) soon.
for testing, try the hydra free tier to create a column-oriented postgres instance on the cloud. https://dashboard.hydra.so/signup https://dashboard.hydra.so/signup
- stolt45 3y ago[dead]
- wrowan33 3y agoCongrats on the success!
- s-mon 3y agoThis looks wild! Been looking for a good event based logs DB and didn’t want to go full clickhouse. This will do!
- mlenhard 3y agoCongrats on the launch! For those who have not experimented with columnar based databases, I would highly recommend toying around with them. The performance improvements can be substantial. Obviously there are drawbacks involved with integrating a new database into your infrastructure, so it is exciting to see columnar format introduced to Postgres. Removes the hurdle of learning, deploying and monitoring another database.
- carlod 3y agoCongrats guys!
- therealwardo 3y agohow does Hydra compare to Citus? https://www.citusdata.com https://www.citusdata.com
- jerrysievert 3y agogenerally faster across the board, a lot of work was done to expand and speed it up, plus updates, deletes, and vacuuming. https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIChwYXJ0aXRpb25lZCkiOmZhbHNlLCJBdGhlbmEgKHNpbmdsZSkiOmZhbHNlLCJBdXJvcmEgZm9yIE15U1FMIjpmYWxzZSwiQXVyb3JhIGZvciBQb3N0Z3JlU1FMIjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiQ2l0dXMiOnRydWUsImNsaWNraG91c2UtbG9jYWwgKHBhcnRpdGlvbmVkKSI6ZmFsc2UsImNsaWNraG91c2UtbG9jYWwgKHNpbmdsZSkiOmZhbHNlLCJDbGlja0hvdXNlICh3ZWIpIjpmYWxzZSwiQ2xpY2tIb3VzZSI6ZmFsc2UsIkNsaWNrSG91c2UgKHR1bmVkKSI6ZmFsc2UsIkNsaWNrSG91c2UgKHpzdGQpIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCI6ZmFsc2UsIkNyYXRlREIiOmZhbHNlLCJEYXRhYmVuZCI6ZmFsc2UsIkRhdGFGdXNpb24gKHNpbmdsZSkiOmZhbHNlLCJBcGFjaGUgRG9yaXMiOmZhbHNlLCJEcnVpZCI6ZmFsc2UsIkR1Y2tEQiAoUGFycXVldCkiOmZhbHNlLCJEdWNrREIiOmZhbHNlLCJFbGFzdGljc2VhcmNoIjpmYWxzZSwiRWxhc3RpY3NlYXJjaCAodHVuZWQpIjpmYWxzZSwiR3JlZW5wbHVtIjpmYWxzZSwiSGVhdnlBSSI6ZmFsc2UsIkh5ZHJhIjp0cnVlLCJJbmZvYnJpZ2h0IjpmYWxzZSwiS2luZXRpY2EiOmZhbHNlLCJNYXJpYURCIENvbHVtblN0b3JlIjpmYWxzZSwiTWFyaWFEQiI6ZmFsc2UsIk1vbmV0REIiOmZhbHNlLCJNb25nb0RCIjpmYWxzZSwiTXlTUUwgKE15SVNBTSkiOmZhbHNlLCJNeVNRTCI6ZmFsc2UsIlBpbm90IjpmYWxzZSwiUG9zdGdyZVNRTCAodHVuZWQpIjpmYWxzZSwiUG9zdGdyZVNRTCI6dHJ1ZSwiUXVlc3REQiAocGFydGl0aW9uZWQpIjpmYWxzZSwiUXVlc3REQiI6ZmFsc2UsIlJlZHNoaWZ0IjpmYWxzZSwiU2VsZWN0REIiOmZhbHNlLCJTaW5nbGVTdG9yZSI6ZmFsc2UsIlNub3dmbGFrZSI6ZmFsc2UsIlNRTGl0ZSI6ZmFsc2UsIlN0YXJSb2NrcyI6ZmFsc2UsIlRpbWVzY2FsZURCIChjb21wcmVzc2lvbikiOmZhbHNlLCJUaW1lc2NhbGVEQiI6ZmFsc2V9LCJ0eXBlIjp7InN0YXRlbGVzcyI6dHJ1ZSwibWFuYWdlZCI6dHJ1ZSwiSmF2YSI6dHJ1ZSwiY29sdW1uLW9yaWVudGVkIjp0cnVlLCJDKysiOnRydWUsIk15U1FMIGNvbXBhdGlibGUiOnRydWUsInJvdy1vcmllbnRlZCI6dHJ1ZSwiQyI6dHJ1ZSwiUG9zdGdyZVNRTCBjb21wYXRpYmxlIjp0cnVlLCJDbGlja0hvdXNlIGRlcml2YXRpdmUiOnRydWUsImVtYmVkZGVkIjp0cnVlLCJzZXJ2ZXJsZXNzIjp0cnVlLCJSdXN0Ijp0cnVlLCJzZWFyY2giOnRydWUsImRvY3VtZW50Ijp0cnVlLCJ0aW1lLXNlcmllcyI6dHJ1ZX0sIm1hY2hpbmUiOnsic2VydmVybGVzcyI6ZmFsc2UsIjE2YWN1IjpmYWxzZSwiTCI6ZmFsc2UsIk0iOmZhbHNlLCJTIjpmYWxzZSwiWFMiOnRydWUsImM2YS40eGxhcmdlLCA1MDBnYiBncDIiOnRydWUsImM1bi40eGxhcmdlLCAyMDBnYiBncDIiOmZhbHNlLCJjNS40eGxhcmdlLCA1MDBnYiBncDIiOmZhbHNlLCJjNmEubWV0YWwsIDUwMGdiIGdwMiI6ZmFsc2UsIjE2IHRocmVhZHMiOmZhbHNlLCIyMCB0aHJlYWRzIjpmYWxzZSwiMjQgdGhyZWFkcyI6ZmFsc2UsIjI4IHRocmVhZHMiOmZhbHNlLCIzMCB0aHJlYWRzIjpmYWxzZSwiNDggdGhyZWFkcyI6ZmFsc2UsIjYwIHRocmVhZHMiOmZhbHNlLCJtNWQuMjR4bGFyZ2UiOmZhbHNlLCJjNmEuNHhsYXJnZSwgMTUwMGdiIGdwMiI6ZmFsc2UsInJhMy4xNnhsYXJnZSI6ZmFsc2UsInJhMy40eGxhcmdlIjpmYWxzZSwicmEzLnhscGx1cyI6ZmFsc2UsIlMyIjpmYWxzZSwiUzI0IjpmYWxzZSwiMlhMIjpmYWxzZSwiM1hMIjpmYWxzZSwiNFhMIjpmYWxzZSwiWEwiOmZhbHNlfSwiY2x1c3Rlcl9zaXplIjp7IjEiOnRydWUsIjIiOnRydWUsIjQiOnRydWUsIjgiOnRydWUsIjE2Ijp0cnVlLCIzMiI6dHJ1ZSwiNjQiOnRydWUsIjEyOCI6dHJ1ZSwic2VydmVybGVzcyI6dHJ1ZSwidW5kZWZpbmVkIjp0cnVlfSwibWV0cmljIjoiaG90IiwicXVlcmllcyI6W3RydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWVdfQ== https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIC...
- ushakov 3y agoAre you funded?
- coatue 3y agoCheck out the "about us" section on the page for more details! "We are excited to share that Hydra raised a $3.1M seed round to drive development of columnar Postgres. We remain committed to sharing our upcoming releases to open source."
- zhendlin 3y agoawesome project - and we tested Zing Data ( http://www.zingdata.com http://www.zingdata.com ) with Hydra to make really fast analytical queries on postgres scale to analytics users on mobile and so far have seen great results.
- mitchpatin 3y agosuper impressive performance improvements! do most of your customers replicate their postgres database to Hydra for analytics jobs, or what's the typical set up?
- mdaniel 3y agoI wanted to say thank you for using actual Open Source licenses. It's gotten to where I treat any recent "Launch HN" or "Show HN" containing "open source" in the title as "well, I wonder which crazy license this project is using"
- dang 3y agoCan you point me to examples of Launch HNs using funky licenses? Show HNs are free-form but Launch HNs are curated by us, and I'd like to know what red flags to watch for. (As this is offtopic for the current Show HN, it might be better for to email hn@ycombinator.com if you, or anyone, would be willing to share that way.)
- mdaniel 3y agoI sent email to avoid being a distraction, but I did want to follow up publicly and say that I apologize for lumping Launch HN into the same bucket as Show HN. For the most part the Launch ones are really Open Source and I apologize for the over generalization :-(
- Iwan-Zotow 3y ago> to query billions of rows instantly Rows? Rows?!? What's the point to have columnar DB to query rows?
- ithkuil 3y agoColumnar DBs often allow you to have tables consisting of multiple columns where values in a column are correlated with values in the other columns. All such correlated values belonging to different columns are commonly called a "row" despite not being stored contiguously. Generally what you do is to scan a column and evaluate a predicate in each value you encounter during the scan (possibly in parallel). For each value of that column that matches the predicate you then keep track of the "position" of the value in the column (a common technique is sparse set data structure such as for example a roaring bitmap). Then you scan through another column and select values for the saved "positions". As you can see, it's not a stretch to view values from different columns that belong to the same "position" as belonging to the same "row" and the "position" to be the "row index" or "row id"
- Iwan-Zotow 3y agoSure, you could think of a row as "tuple of values returned from all columns at the same index". But why it was that important to measure and use and present for primarily columnar store? What's the point to measure and count row access? Isn't existing row DBs working for such case?
- joshgray 3y agoCongrats to the entire Hydra team on the launch! We (Artemis - https://www.artemisdata.io/ https://www.artemisdata.io/) are stoked to be build with you as a partner to help data teams analyze data even faster!
- hgimenez 3y agoAlways awesome to see folks moving Postgres forward. Congrats on the launch!
- burcs 3y agoThis is really cool, just played around with it a bit but excited to do a deeper dive later. Nice work guys!
- coatue 3y agoThanks! feel free to DM me in the hydra discord (or email) anytime
- zrizavi17 3y agoSuch a game changer and useful alternative to legacy databases!
- chrisjc 3y agoVery out of touch with Postgres, but is there a native column oriented table type option in Postgres so that you choose either row-based or columnar in the CREATE TABLE DDL?
- btown 3y agoI don't believe Postgres has this natively, but an alternative to OP is Citus, a Postgres extension which allows this kind of syntax. https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-compression-for-postgres/ https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-... EDIT: per another comment, OP is a fork of Citus Columnar!
- techwizrd 3y agoThis looks really impressive, and I'm excited to see how it performs on our data! P.S., I think the name conflicts with Hydra, the configuration management library: https://hydra.cc/ https://hydra.cc/
- entuno 3y agoAnd also the password bruteforcing tool by THC.
- notpushkin 3y agoAlso with the OIDC server by Ory [1] and a certain defunct Russian darknet marketplace [2]. [1]: https://www.ory.sh/hydra/ https://www.ory.sh/hydra/ [2]: Not today, tovarisch mayor.
- quickthrower2 3y agoThe many heads of Hydra
- say_it_as_it_is 3y agoHydra Columnar is a fork from Citus Columnar c. April, 2022.
- thih9 3y agoCitus: https://github.com/citusdata/citus https://github.com/citusdata/citus BTW, Citus license is GNU Affero General Public License (github lists “conditions: same license”) and hydra is Apache. How is that possible if the latter is a fork? There’s probably something about these licenses I’m not aware of and I’m curious.
- jerrysievert 3y agohydra columnar inherits its license from citus: https://github.com/hydradatabase/hydra/blob/main/columnar/LICENSE https://github.com/hydradatabase/hydra/blob/main/columnar/LI... but, hydra itself is more than just the columnar extension.
- thih9 3y agoThanks for explaining. This is confusing to me as a github user, i.e. if I saw a license in the project’s description, I wouldn’t expect another license in a subdirectory. Github now has UI for repos with multiple licenses: https://github.blog/changelog/2022-05-26-easily-discover-and-navigate-to-multiple-licenses-in-repositories/ https://github.blog/changelog/2022-05-26-easily-discover-and... , that would have been clearer for me.
- mdaniel 3y agothey must not look at anything other than the top directory, then, because this repo's sidebar only says "Apache-2.0 license" which I grant is clarified down at the _bottom_ of the README but if that multi-license thing really wants to be helpful it should at least look one subdirectory down
- pella 3y agoCongratulations! Can we expect support for gist, gin, spgist, and brin indexes sometime in the near future? Based on the source code, it appears that they are not supported: https://github.com/hydradatabase/hydra/blob/96056312e7c0f4135e31ce5380f4f7b4cfb7b42f/columnar/src/backend/columnar/README.md?plain=1#L190 https://github.com/hydradatabase/hydra/blob/96056312e7c0f413... "... Columnar supports `btree` and `hash `indexes (and the constraints requiring them) but does not support `gist`, `gin`, `spgist` and `brin` indexes."
- nerdponx 3y agoDo different kinds of indexes work better for columnar storage? Or is it the same principles for both?
- hodgesrm 3y agoDifference principles of indexing, as least based on my experience with ClickHouse. * Column-based stores have really fast scans due to compression and vectorization, so you'll generally always read down the column. The way to speed it up is to have "skip indexes" that allow you to skip blocks, e.g., don't even bother to read/decompress them. * Commonly used indexes need to be very sparse, so they fit in memory even when tables run to hundreds of billions of rows. * Finally highly compressed columns can be used as indexes to filter data rapidly. ClickHouse calls this PREWHERE processing. Edit: clarify skip indexes
- pella 3y agoWe need a spatial index for spatial (columnar) data! - https://www.crunchydata.com/blog/the-many-spatial-indexes-of-postgis https://www.crunchydata.com/blog/the-many-spatial-indexes-of... - http://postgis.net/workshops/postgis-intro/indexing.html http://postgis.net/workshops/postgis-intro/indexing.html - Spatial indexes for OSM in PostGIS (PDF) : https://pretalx.com/media/sotm2019/submissions/CAD93S/resources/Kunde_spatial_indexes_for_osm_in_postgis.pdf https://pretalx.com/media/sotm2019/submissions/CAD93S/resour...
- edublancas 3y agoCongrats on the 1.0 milestone! A few months ago, we worked with the team to bring Hydra to Jupyter, you can check out the tutorial here: https://docs.hydra.so/analyze/jupyter https://docs.hydra.so/analyze/jupyter JupySQL's GitHub: https://github.com/ploomber/jupysql https://github.com/ploomber/jupysql
- coatue 3y agoNice and thanks @edublancas, that was a useful tutorial and it's nice to be able to query hydra with SQL via Jupyter. fan of your project
- ryanb_wise 3y agoTalked to OP last night and played around with it this morning. This is something I've wanted to see added to postgres for a long time, and couldn't have been done by a nicer and more accommodating founder. Very excited.
- pajep 3y agocan I ask if you guys take contributors?
- wuputah 3y agoof course :) drop by our Discord if there's something you'd like to contribute and want to chat about it beforehand, need help/have questions getting started, etc. https://hydra.so/discord https://hydra.so/discord
- florianherrengt 3y agoCan you add the extension to an existing database?
- ahmedfromtunis 3y agoThat's also my question. Couldn't find anything for it in the docs provided. Also, how to migrate data from an existing database? (Is it the usual pg_dump/psql combo?)
- d4rkp4ttern 3y agoIndeed I am also wondering — if I have an existing Postgres DB , how do I leverage this?
- qeternity 3y agoOk so this is a Citus fork. Where can I read about what the differences / trade offs are? I don’t see anything in the docs.
- deleted 3y ago[deleted]
- quadrature 3y agoHow are updates handled, is it doing a merge on read ?
- wuputah 3y agoFirst we added a bitmask to mark rows as deleted - these rows are filtered out on read. Then updates are implemented as deletions + inserts. We have also added vacuum functions to remove/rewrite stripes that have >20% of deleted rows in order to reclaim space and optimize those stripes.
- afeiszli 3y agoReally cool!
- dikei 3y agoFor a Postgres extension, there's a strangely lack of documentation on how to add Hydra to an existing PG installation.
- vicsomething 3y ago[dead]
- jhu2002 3y ago[dead]
- mch82 3y agoIs hydra compatible with the PostGIS extension?
- KingOfCoders 3y agoHow does it compare to Timescaledb? Performance? Features? (TSDB is my current Postgres data lake setup)
- anagri 3y agoCongrats on the launch. Curious, why not contribute to Citus? Why create a fork and have duplicated effort?
- cosmojg 3y agoBecause then it would be much harder to bait investors. By forking, they can more confidently say things like, "We built Hydra to solve real problems affecting real people who will part with oodles of money to access our totally unique solution."
- samgaw 3y agoFrom my initial 5 minutes of getting setup with an existing install, it's all fairly easy. # Requirements - lz4 - zstd # Building git clone https://github.com/hydradatabase/hydra cd hydra/columnar ./configure make make install # Install CREATE EXTENSION IF NOT EXISTS columnar; The actual extension docs are at https://docs.hydra.so/concepts/using-hydra-columnar https://docs.hydra.so/concepts/using-hydra-columnar
- kevincox 3y agoI would love to have the ability to define "column groups" in regular Postgres. For example most of the time I just leave the table row-oriented, but when trying to optimize performance of specific tables and queries I could tell postgres to store one or a handful of columns separately similar to a column oriented database. This would allow getting quite a bit of performance out of my database for the few cases in OLTP where I need to do some scans or searches. Even just breaking out some rarely used columns into a separate group could be quite nice for keeping the caches hot.