15 ms·

Citus Unforks from PostgreSQL, Goes Open Source

763 points by jamesheroku 11y ago

gtrubetskoy 11y ago
If anyone from Citus is reading this: how does this affect your business model? I remember when I asked at Strata conf a couple of years ago why isn't your stuff Open Source, the answer then was "because revenue". So what changed since then?
- atonse 11y ago
  My hunch is that the two are not really related. Companies of any appreciable size will be happy to pay for support if they choose to make Citus a part of their critical infrastructure. And the industry reached an inflection point where there are enough companies want as much of their infrastructure to be open source as possible, that you can run a company where most of your stuff is open source, while still making a ton of money (like RedHat, CoreOS, Docker, etc)
  - SwellJoe 11y ago
    I know Red Hat is making a ton of money. But, CoreOS and Docker, are they at the "making a ton of money" stage, or merely well-funded by investors?
    - atonse 11y ago
      Good point. I wondered if I should edit that specific part, but kept it anyway. But a sign of getting investors is also that they see that there's still potential of making a lot of money, in spite of being open source. That's true here too. Citus mentioned they spoke to their board (and presumably their investors too) about this change.
- ascetone 11y ago
  It doesn't. AGPL means the only people using it have to licensed the same.
  - jgreen10 11y ago
    Well, people use MongoDB... AGPL only requires open sourcing any modifications you make to the software when you give users direct access to a server running the software, which seems like something you would never want to do in case of a database. You can use database servers running AGPL software in a closed source SaaS: http://www.gnu.org/licenses/why-affero-gpl.en.html http://www.gnu.org/licenses/why-affero-gpl.en.html
    - reitanqild 11y ago
      MongoDB used to distribute drivers under a more permissive license for this exact reason IIRC.
      - jgreen10 11y ago
        Well, applications that want to use Citus can just use whichever PostgreSQL client is available.
        reitanqild 11y ago
        Thanks, twists my head but as far as I see it could work.- IANAL
- craigkerstiens 11y ago
  Hi, Craig from Citus here. We have some premium features in our Enterprise edition. Many of these are features that larger enterprises will want to pay for such as security features around roles, a tool for automated cluster resizing, and enhanced load balancing tools, and of course support. Beyond that, we have a few other things in the works for the future that will cover other revenue models.
- umur 11y ago
  Umur from Citus here. Adding to Craig's comments: Several things have changed over the last two years that allowed us to make this happen: Most importantly, we've continued building out the product for a more broad user base, grown with more customers and users, received further funding as validation, and expanded both our team and product to offer additional revenue generating services. All put together, open sourcing Citus is something we've always wanted to do, and we are excited to continue building on it for many years to come, with the help of the community and our enterprise customers.
rkrzr 11y ago
This is fantastic news! Postgres does not have a terribly strong High Availability story so far and of course it also does not scale out vertically. I have looked at CitusDB in the past, but was always put off by its closed-source nature. Opening it up seems like a great move for them and for all Postgres users. I can imagine that a very active open-source community will develop around it.
- hblanks 11y ago
  We've been running CitusDB for a couple years now at CloudFlare for serving aggregated analytics to customers (cf. https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...). It's a good product, and it was even fairly easy to do a major version upgrade / cluster relocation. At least as easy as such a thing can be. :-)
  - rkrzr 11y ago
    Are there any limitations you have run into? E.g. can you still use all index types that Postgres offers or are there any special distributed index types that CitusDB adds perhaps?
    - makr17 11y ago
      My recollection from the last time I played with it, some bits from core postgres are unsupported; things like sequences and recursive CTEs. Maybe all CTEs?
    - hblanks 11y ago
      Correct. CTEs are pretty much a no-go, and you also can't do things like join between distributed and non-distributed tables. Your queries tend to break down into ones where you're hitting a small number of shards (such as when we serve data for the analytics page), or else ones where you tend to aggregate up into temporary, non-distributed tables for further analysis (such as you'd do for infrequent business reporting). All told, it's been one of the easier tools for us to operationalize, largely thanks to the fact that it's "just PostgreSQL." The one thing I wish we had was better documentation of what kind of consistency guarantees to expect, although for an append-only store like our current use case, that's less of a concern. And to be fair, Ozgun and the other guys at Citus have always been really happy to answer any questions we have. With this code now going open source, it should be pretty easy to look into these sorts of internals.
TY 11y ago
This is awesome! Tebrikler (congrats) on the release of 5.0 and going OS, definitely great news. Can you publish competitive positioning of Citus vs Actian Matrix (nee ParAccel) and Vertica? I'd love to compare them side by side - even if it's just from your point of view :-)
- peatmoss 11y ago
  Second the request for comparison to Vertica. I've recentishly become a user at work, and I wonder how this compares. A quick googling didn't yield anything too informative.
- flavor8 11y ago
  ...and Redshift. I love what Amazon provide, but it gets expensive.
  - umur 11y ago
    Umur from Citus here. For purposes of this question, I’ll bucket traditional data warehousing (DWH) solutions like Redshift, Vertica, Greenplum together, although there are many nuances among each of them of course. First, Citus is not a traditional data warehouse. We position Citus as the real-time, scalable database that serves your application under a mix of high- concurrency short requests and ad-hoc SQL analytics (i.e. think both random and sequential scans for a customer-facing analytics app). The default storage engine for Citus is the PostgreSQL storage engine, which is row-based. This is in contrast to many data warehouses, which often use a column store and/or batch data loads, and are focused purely on analytics. The trade-offs you get are: - Citus vs. DWH performance: DWH and Citus both have a similar parallelization for analytics queries (multi-core, multi-machine), but most data warehouses typically use a columnar storage engine instead of a row-based one. Columnar storage is designed for faster analytics queries, so that makes columnar DWH generally faster on longer running analytics queries. However, this comes at the expense of (1) concurrency and (2) short-request performance (think simple lookups, updates, real-time data ingest) vs. Citus' row-based storage. If you've tried having 10s of concurrent connections to Redshift for short lookups, or performing 100s/1000s of inserts/updates to power your application, these limitations will be familiar. This is to be expected, as Redshift is not designed as a real-time operational database, but an offline data warehouse. In essence, the two classes of products are more complimentary than substitutes, even while they have some overlaps in their analytic capabilities. Something like Redshift will give you fast offline analytics, after you move your data in batch (via S3); Citus will directly power your analytic apps in real-time; without ETL'ing your event/user data back and forth between separate OLTP and OLAP databases. Both can be extremely fast: Redshift can run complex data warehousing queries that take an hour in a few minutes, Citus can scan and aggregate 100 million records in a few seconds, while simultaneously ingesting your events in real-time. I hope that provides some clarification on the workloads. There is a lot more, including columnar storage and product approach (re: implications of extending Postgres 9.5 vs. forking Postgres 8.x), and I’ll dive into those in separate comments as well.
jjawssd 11y ago
My guess is that Citus is making enough money from consulting that they don't need to keep this code closed source when they can profit from free community-driven growth while they are expanding their sales pipeline through consulting.
- onRoadAgain23 11y ago
  They offer an enterprise paid version with more functionality. "for customers with large production deployments, we also offer an enterprise edition that comes with additional functionality"
- craigkerstiens 11y ago
  Hi, Craig from Citus here. In addition to the open source Citus, we have some premium features in our Enterprise edition. Many of these are ones that larger enterprises will want to pay for such as security features around roles, a tool for automated cluster resizing, and enhanced load balancing tools, and of course support. Beyond that we have a few other things in the work that will speak to various revenue models for the future.
exhilaration 11y ago
AGPL license if anyone's curious: https://github.com/citusdata/citus/blob/master/LICENSE https://github.com/citusdata/citus/blob/master/LICENSE
- rkrzr 11y ago
  GNU AGPL to be precise (GNU Affero General Public License).
- polskibus 11y ago
  Does that mean that whatever connects to this database needs to be AGPL too?
  - crudbug 11y ago
    No .. it means any changes you make to CitusDB should be made public. [0] https://en.wikipedia.org/wiki/Affero_General_Public_License https://en.wikipedia.org/wiki/Affero_General_Public_License
  - kodablah 11y ago
    No, it means if you make changes to the code and use it over a network you have to be AGPL too (think "network" as "distribution" in the GPL sense).
    - hlandau 11y ago
      Nope. https://www.devever.net/~hl/agplunenforceable https://www.devever.net/~hl/agplunenforceable
      - kod 11y ago
        This is a really bad argument, probably wouldn't stand up in court, and in any case doesn't have precedent. Hint: How did you (the developer) originally acquire a _copy_ of the AGPL'd sourcecode?
        11y ago
no1youknowz 11y ago
This is awesome. I have experience with running a CitusDB cluster and it pretty much solved a lot of the scaling problems I was having at the time. For it to go open source now, is of huge benefit to the future projects I have. > With the release of newly open sourced Citus v5.0, pg_shard's codebase has been merged into Citus... This is fantastic, sounds like the setup process is much simpler. I wonder if they have introduced the Active/Active Master solution they were working on? I know before, there is 1 Master and multiple Worker nodes. The solution before was to have a passive backup of the Master. If say, they released the Active/Active Master later on this year. That's huge. I can pretty much think of my DB solution as done at this point.
- ozgune 11y ago
  (Ozgun from Citus Data) We're working on making Citus masterless. In all openness, we evaluated two different approaches to this in the past six months, and wrapped up the design for one. This design works well on the cloud, and we already demonstrated a working version: https://youtu.be/_nun2S6EdWo?t=411 https://youtu.be/_nun2S6EdWo?t=411 For on-premise deployments, the primary challenge is set-up complexity. We're now prototyping one of those designs to know more: https://github.com/citusdata/citus/issues/389 https://github.com/citusdata/citus/issues/389 We expect to share all the details and a concrete timeline in April.
  - no1youknowz 11y ago
    Excellent news! Really looking forward to this one! :)
  - batbomb 11y ago
    Can Citus handle geospatial sharding?
    - tracker1 11y ago
      You could always compute a geohash and use that as a shard key... I'm not familiar enough with Citus' specific approach here, but using a limited geohash would give you close to what you're looking for.
onRoadAgain23 11y ago
Being burned before,I will never use an OS infrastructure project that has enterprise features you need to pay for. They always try to move you to paid and make the OSS version unpleasant to use over time as soon as the bean counters take over to milk you "For customers with large production deployments, we also offer an enterprise edition that comes with additional functionality"
- sergiosgc 11y ago
  This is the model used by many companies backing OSS. The fact that you have been burned before means the actor in that case (or cases) acted badly, not that the model is wrong. Software isn't free to produce, and the need to make money off software isn't something companies should be ashamed of. In fact, nowadays I'm leaning towards trusting OSS with clear financial sustainability over software whose long term existence seems shaky.
- gdulli 11y ago
  Do you often make big decisions based on extrapolation from so few data points? I use a major open source system with enterprise features and support but don't pay for any of those options. I've used it for 3 years and it's been invaluable. No pressure to start paying for anything. Some of its premium features have actually become free over that period. But I wouldn't decide that all open source systems with premium features are safe based on that experience.
  - onRoadAgain23 11y ago
    If this costed roughly a million dollars, then yes. Especially if you're locked in like with a DB. I use nginx because even though it has this mode it would be easy to replace with something else.
- jamespo 11y ago
  Feel free to code your own
- gtaylor 11y ago
  What are some alternatives for paying their employees to develop these products that are 100% free? There are some excellent commercial organizations that drive some of the tech that many or even most of us have came to rely on. This goes all the way down to Linux and BSD itself.
X86BSD 11y ago
AGPL? This is dead in the water :( It will never be integrated into PG. What a shame. It should have been a 2 clause BSDL. Sigh.
- andreasklinger 11y ago
  might be intentional
- tspiteri 11y ago
  The BSDL does not make much economic sense to the company open sourcing their code; a new competitor would fork the code, make closed improvements, and merge any changes from the open source code. That means that the competitor is always gaining by a one-way flow of improvements. To use open source code, the more permissive the license the better. But to actually open your own code, BSDL is a very tough sell. That's also why they use the AGPL. With database systems, even if they were under the GPL, some competitors could just modify the system and run it on their own server with improvements, and offer just the service to their clients. Again, the improvements go one way only: since the competitor would not distribute the modified system, as it's running on their servers, they would not need to distribute source changes. With the AGPL, that loophole is closed.
  - X86BSD 11y ago
    So you take a BSDL codebase, fork it, close it, make proprietary changes, profiting from the BSDL codebase, then slap the PG community in the face by open sourcing it under a more restrictive license hoping to benefit from the community you just slapped in the face but restricting competition. They are of course free to release their code under any license they wish. I just think releasing code under the *GPL when you profited from a liberal BSDL is a douche nozzle thing to do. But knock yourself out! This tells me all I need to know about the company.
    - tspiteri 11y ago
      I don't agree with you. The PG community is, I think, fine with that: that's exactly what the BSDL allows that the GPL doesn't, and they chose the BSDL. If the PG community don't like that, I really don't understand why they chose the BSDL.
erikb 11y ago
Unforking is a very smart decision. Postgres also has gained a lot of favour since MySQL was bought by Oracle. Altogether Citus has earned a lot of kudos for that move, at least with me, for all that may count!
azinman2 11y ago
I want it to be called citrus, which is what I always read it as....
- jrochkind1 11y ago
  Then I'd get confused and think it had something to do with Citrix.
  - azinman2 11y ago
    A cute lemon logo might help :)
- BinaryIdiot 11y ago
  Yikes I keep calling it citrus. The word citrus is so ingrain I'm actually having a hard time pronouncing citus when I see the word. I didn't even notice I was doing it wrong until I saw your comment.
- mrgreenfur 11y ago
  Only after reading this did I realize it's NOT citrus. Thanks!
faizshah 11y ago
So this sounds similar to Pivotal's Greenplum which is also open source, can anyone compare the two?
- frn 11y ago
  Greenplum is based on postgres 8.2, with the featureset you'd expect from pg 8.2 - basically none of the additions after 2006 have merged to GP.
  - faizshah 11y ago
    Ok, and what's the process like for disaster recovery with citus?
    - craigkerstiens 11y ago
      That depends on your setup, for the master instance you'd run it just as you would for other setups. Streaming replication is common there. For the sharded instances, Citus has the ability for you to set what your replication factor is. Here Citus is then aware of when a node fails and will automatically redistribute the data to a new node, essentially taking care of that for you.
  - ioltas 11y ago
    Since the move to open source, more recent upstream changes have been slowly merged in the code base, though they seem to be still on a 8.3 base, still a couple of years worth of code to go through.
- amitlan 11y ago
  Greenplum is a fork of Postgres codebase, Citus is not; it's an extension that leverages community Postgres's extensibilty APIs. This point seems to be highlighted in their post.
lcfcjs 11y ago
Postgres just doesn't scale, learnt that one the hard way
Dowwie 11y ago
is it correct to compare citusdb with pipelinedb?
BinaryIdiot 11y ago
I don't have a ton of experience scaling out and using different flavors of PostgreSQL but I had run across Postgres-XL not long ago; does anyone know how this compares to that?
Dowwie 11y ago
would a natural evolutionary path for start ups be to emerge with postgresql and grow to requiring citusdb?
devit 11y ago
I've been unable to find any clear description of the capabilities of Citus and competing solutions (postgres-x2 seems the other leader). Which of these are supported: 1. Full PostgreSQL SQL language 2. All isolation levels including Serializable (in the sense that they actually provide the same guarantees as normal PostgreSQL) 3. Never losing any committed data on sub-majority failures (i.e. synchronous replication) 4. Ability to automatically distribute the data (i.e. sharding) 5. Ability to replicate the data instead or in addition to sharding 6. Transactionally-correct read scalability 7. Transactionally-correct write scalability where possible (i.e. multi-master replication) 8. Automatic configuration only requiring to specify some sort of "cluster identifier" the node belongs to
- cowardlydragon 11y ago
  But if they answer those questions, you won't buy support/use it... Have a donut and look at our marketing spreadsheets. I'm so tired of "seamless" "effortless" "simple" distributed database lies. There's mathematical theorems as to why there is no free lunch.
- ozgune 11y ago
  (Ozgun from Citus Data) On PostgreSQL language support, we're updating our FAQ to have more information: https://www.citusdata.com/frequently-asked-questions https://www.citusdata.com/frequently-asked-questions Since the PostgreSQL manual (and its feature set) spans over 4K+ pages, we found that the best way to think about Citus' capabilities is from a use-case standpoint. If your workload needs distributed transactions that span across machines, or large ETL jobs, Citus currently isn't the best fit. Citus supports sharding and replication out of the box (#4, #5). On #6, reads go through a master node (metadata server) and you see what you write. We don't have #7. The way in which we implement this also has implications on your other questions. Multi-master (no single metadata server) is by far the biggest feature request that we receive: https://news.ycombinator.com/item?id=11353866 https://news.ycombinator.com/item?id=11353866 If we go with the approach in https://github.com/citusdata/citus/issues/389 https://github.com/citusdata/citus/issues/389, you will be able to configure #3, #6, #7 through PostgreSQL's streaming replication settings. We still won't support distributed transactions that span across multiple machines. On #8, could you elaborate a bit more? Do you mean a logical identifier for the node? Also, it's hard to write a concise reply on a topic that requires so much context. I'd love to grab coffee with anyone who's interested in diving deep into distributed databases. Feel free to shoot me an email at ozgun@citusdata.com
signalnine 11y ago
Congrats from Agari! We've been looking forward to this and continue to get a lot of value from both the product and the top-notch support.
ccleve 11y ago
I'd very much like to see what algorithm these systems are using to enable transactions in a distributed environment. Are they just using straight two-phase commit, and letting the whole transaction fail if a single server goes down? Or are are they getting fancy and doing some kind of replication with consensus?
- wmfiv 11y ago
  I believe transactions must process against a single node.
voctor 11y ago
Citus can parallelize SQL queries across a cluster and across multiple CPU cores. How does it compare with the upcoming 9.6 version of PostgreSQL which will support parallel-able sequential scans, parallel joins and parallel aggregate ?
- lfittl 11y ago
  AFAIK all the parallel work done in 9.6 refers to parallel operations on a single node (but multiple cores). This would be complimentary to what Citus does, which is distributing the load across multiple shard instances (each with their own cores, benefiting from the parallel work in 9.6).
  - voctor 11y ago
    Yes, but Citus can also parallelize on multiple cores when used on a single machine ("If you’re running Citus on a single machine, this will scale queries across multiple CPU cores. and create the impression of sharding across databases."). Will this functionality becomes obsolete with the 9.6 ?
    - lfittl 11y ago
      Fair point - I assume this will be merged together in some way (i.e. the Citus stuff building on top of the parallel scan infrastructure), but probably a better question to ask on #citus IRC / open a Github issue. Some Postgres committers work on Citus as well (e.g. Andres Freud), so I'm sure this has been thought through before.
ismail 11y ago
Any thoughts on using something like postgres+citrus vs hadoop+hbase+ecosystem vs druid for olap/analytics with very large volumes of data
Someone 11y ago
One must thank them for open sourcing this, and cannot blame them for using a different license, but using a different license makes me think calling this "unfork" is bending the truth a little bit.
- anarazel 11y ago
  The "unfork" part is primarily about not forking the postgres codebase anymore, as done before citus 5.0 (i.e. we modified parts of postgres, to make it citus). Citus now entirely works as an extension to postgres, using the extension facilities postgres provides.
- takeda 11y ago
  Perhaps I'm missing something, but this is just an extension that works with standard postgres, there are no code changes in postgres itself, so it doesn't look like it ever was a fork.
  - jasonmp85 11y ago
    (Jason from Citus here) Yes, that's what you're seeing right now, but in the past Citus (used to be "CitusDB") was a superset of the entire PostgreSQL codebase. During the lead-up to the open source release, we removed the use of any static methods or internal machinery and rewrote the installation process to use the PostgreSQL CREATE EXTENSION command. Additionally, we moved all of pg_shard's DML functionality into Citus to unify the product line. So ultimately CitusDB was a fork but is now entirely an extension.
    - Someone 11y ago
      Aha. Should have read the announcement better. Thanks for the explanation.
satygeek 11y ago
Does CitusDb fit in olap analytical workloads to do aggregations on hundreds millions of records using varying order and size of dimensions (eg druid) in max of 3 seconds response time using as few boxes as possible - Or there are other techniques have to be used along with Citusdb? Can you shed a light on your experience with CloudFlare in terms of cluster size and queries perf?
- lfittl 11y ago
  I'll leave the initial questions to the Citus team, but re: CloudFlare this link might be helpful: https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
  - satygeek 11y ago
    Thanks. I went through it but couldnt find info about their cluster size, data size and queries response time
- greggyb 11y ago
  Hundreds of millions of records in <=3 seconds is not really a big challenge with a good data model and proper indexing on even a single server. I work for a BI consultancy and we don't even bat an eye until we hit billions of records in a primary fact table. Certainly the DB server does need to scale vertically to some extent as you pass through the orders of magnitude > 10M. A good columnstore engine is also worthwhile to consider.
- mslot 11y ago
  Yes, Citus may be a good fit, for a complete example see: https://www.citusdata.com/blog/15-marco-slot/402-interactive-analytics-github-data-using-postgresql-citus https://www.citusdata.com/blog/15-marco-slot/402-interactive...
ahachete 11y ago
Congratulations, Citus. Since I heard last year at PgConfSV that you will be releasing CitusDB 5.0 as open source, I've been waiting for this moment to come. It makes 9.5's awesome capabilities to be augmented with sharding and distributed queries. While this targets real-time analytics and OLAP scenarios, being an open source extension to 9.5 means that a whole lot of users will benefit from this, even under more OLTP-like scenarios. Now that Citus is open source, ToroDB will add a new CitusDB backend soon, to scale-out the Citus way, rather than in a Mongo way :) Keep up with the good work!
ioltas 11y ago
Congrats to all for the release. That's a lot of work accomplished.
ksec 11y ago
Does anyone know How does Citus compared to Postgre XL ?
albasha 11y ago
I recently switched back to MariaDB because I didn't see a clear/easy path for Postgres scalability in case the project i am working on takes off. I am under the assumption there are at least two fairly simple approaches to scale MySQL; master-master replication using Galera and Aurora from AWS. What do you guys think? Am I right in thinking MySQL is easier to scale given I want to spend the least amount of time maintaining it.
lobster_johnson 11y ago
This is great! One thing I'm having trouble with is finding information about transactional semantics. If I make several updates (to differently sharded keys) in a single transaction, will the transaction boundaries be preserved (committed "locally" first, then replicated atomically to shards)? Or will they fan out to different shards with separate begin/commit statements? Or without transactional boundaries at all? In fact, I can't really find any information on how CitusDB achieves its transparent sharding for queries and writes. Does it add triggers to distributed tables to rewrite inserts, updates and deletes? Or are tables renamed and replaced with foreign tables? I wish the documentation was a bit more extensive.
lambdafunc 11y ago
Any benchmarks comparing CitusDB against Presto?
uberneo 11y ago
Great product - If would be nice to have a Admin interface like RethinkDB where you can clearly define your replication and Sharding settings. Any documentation around how to do this from command line ?

763 points by jamesheroku 11y ago

If anyone from Citus is reading this: how does this affect your business model? I remember when I asked at Strata conf a couple of years ago why isn't your stuff Open Source, the answer then was "because revenue". So what changed since then?

atonse 11y ago
My hunch is that the two are not really related. Companies of any appreciable size will be happy to pay for support if they choose to make Citus a part of their critical infrastructure. And the industry reached an inflection point where there are enough companies want as much of their infrastructure to be open source as possible, that you can run a company where most of your stuff is open source, while still making a ton of money (like RedHat, CoreOS, Docker, etc)
- SwellJoe 11y ago
  I know Red Hat is making a ton of money. But, CoreOS and Docker, are they at the "making a ton of money" stage, or merely well-funded by investors?
  - atonse 11y ago
    Good point. I wondered if I should edit that specific part, but kept it anyway. But a sign of getting investors is also that they see that there's still potential of making a lot of money, in spite of being open source. That's true here too. Citus mentioned they spoke to their board (and presumably their investors too) about this change.
ascetone 11y ago
It doesn't. AGPL means the only people using it have to licensed the same.
- jgreen10 11y ago
  Well, people use MongoDB... AGPL only requires open sourcing any modifications you make to the software when you give users direct access to a server running the software, which seems like something you would never want to do in case of a database. You can use database servers running AGPL software in a closed source SaaS: http://www.gnu.org/licenses/why-affero-gpl.en.html http://www.gnu.org/licenses/why-affero-gpl.en.html
  - reitanqild 11y ago
    MongoDB used to distribute drivers under a more permissive license for this exact reason IIRC.
    - jgreen10 11y ago
      Well, applications that want to use Citus can just use whichever PostgreSQL client is available.
      - reitanqild 11y ago
        Thanks, twists my head but as far as I see it could work.- IANAL
craigkerstiens 11y ago
Hi, Craig from Citus here. We have some premium features in our Enterprise edition. Many of these are features that larger enterprises will want to pay for such as security features around roles, a tool for automated cluster resizing, and enhanced load balancing tools, and of course support. Beyond that, we have a few other things in the works for the future that will cover other revenue models.
umur 11y ago
Umur from Citus here. Adding to Craig's comments: Several things have changed over the last two years that allowed us to make this happen: Most importantly, we've continued building out the product for a more broad user base, grown with more customers and users, received further funding as validation, and expanded both our team and product to offer additional revenue generating services. All put together, open sourcing Citus is something we've always wanted to do, and we are excited to continue building on it for many years to come, with the help of the community and our enterprise customers.

rkrzr 11y ago

This is fantastic news! Postgres does not have a terribly strong High Availability story so far and of course it also does not scale out vertically. I have looked at CitusDB in the past, but was always put off by its closed-source nature. Opening it up seems like a great move for them and for all Postgres users. I can imagine that a very active open-source community will develop around it.

hblanks 11y ago
We've been running CitusDB for a couple years now at CloudFlare for serving aggregated analytics to customers (cf. https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...). It's a good product, and it was even fairly easy to do a major version upgrade / cluster relocation. At least as easy as such a thing can be. :-)
- rkrzr 11y ago
  Are there any limitations you have run into? E.g. can you still use all index types that Postgres offers or are there any special distributed index types that CitusDB adds perhaps?
  - makr17 11y ago
    My recollection from the last time I played with it, some bits from core postgres are unsupported; things like sequences and recursive CTEs. Maybe all CTEs?
  - hblanks 11y ago
    Correct. CTEs are pretty much a no-go, and you also can't do things like join between distributed and non-distributed tables. Your queries tend to break down into ones where you're hitting a small number of shards (such as when we serve data for the analytics page), or else ones where you tend to aggregate up into temporary, non-distributed tables for further analysis (such as you'd do for infrequent business reporting). All told, it's been one of the easier tools for us to operationalize, largely thanks to the fact that it's "just PostgreSQL." The one thing I wish we had was better documentation of what kind of consistency guarantees to expect, although for an append-only store like our current use case, that's less of a concern. And to be fair, Ozgun and the other guys at Citus have always been really happy to answer any questions we have. With this code now going open source, it should be pretty easy to look into these sorts of internals.

TY 11y ago

This is awesome! Tebrikler (congrats) on the release of 5.0 and going OS, definitely great news. Can you publish competitive positioning of Citus vs Actian Matrix (nee ParAccel) and Vertica? I'd love to compare them side by side - even if it's just from your point of view :-)

peatmoss 11y ago
Second the request for comparison to Vertica. I've recentishly become a user at work, and I wonder how this compares. A quick googling didn't yield anything too informative.
flavor8 11y ago
...and Redshift. I love what Amazon provide, but it gets expensive.
- umur 11y ago
  Umur from Citus here. For purposes of this question, I’ll bucket traditional data warehousing (DWH) solutions like Redshift, Vertica, Greenplum together, although there are many nuances among each of them of course. First, Citus is not a traditional data warehouse. We position Citus as the real-time, scalable database that serves your application under a mix of high- concurrency short requests and ad-hoc SQL analytics (i.e. think both random and sequential scans for a customer-facing analytics app). The default storage engine for Citus is the PostgreSQL storage engine, which is row-based. This is in contrast to many data warehouses, which often use a column store and/or batch data loads, and are focused purely on analytics. The trade-offs you get are: - Citus vs. DWH performance: DWH and Citus both have a similar parallelization for analytics queries (multi-core, multi-machine), but most data warehouses typically use a columnar storage engine instead of a row-based one. Columnar storage is designed for faster analytics queries, so that makes columnar DWH generally faster on longer running analytics queries. However, this comes at the expense of (1) concurrency and (2) short-request performance (think simple lookups, updates, real-time data ingest) vs. Citus' row-based storage. If you've tried having 10s of concurrent connections to Redshift for short lookups, or performing 100s/1000s of inserts/updates to power your application, these limitations will be familiar. This is to be expected, as Redshift is not designed as a real-time operational database, but an offline data warehouse. In essence, the two classes of products are more complimentary than substitutes, even while they have some overlaps in their analytic capabilities. Something like Redshift will give you fast offline analytics, after you move your data in batch (via S3); Citus will directly power your analytic apps in real-time; without ETL'ing your event/user data back and forth between separate OLTP and OLAP databases. Both can be extremely fast: Redshift can run complex data warehousing queries that take an hour in a few minutes, Citus can scan and aggregate 100 million records in a few seconds, while simultaneously ingesting your events in real-time. I hope that provides some clarification on the workloads. There is a lot more, including columnar storage and product approach (re: implications of extending Postgres 9.5 vs. forking Postgres 8.x), and I’ll dive into those in separate comments as well.

jjawssd 11y ago

My guess is that Citus is making enough money from consulting that they don't need to keep this code closed source when they can profit from free community-driven growth while they are expanding their sales pipeline through consulting.

onRoadAgain23 11y ago
They offer an enterprise paid version with more functionality. "for customers with large production deployments, we also offer an enterprise edition that comes with additional functionality"
craigkerstiens 11y ago
Hi, Craig from Citus here. In addition to the open source Citus, we have some premium features in our Enterprise edition. Many of these are ones that larger enterprises will want to pay for such as security features around roles, a tool for automated cluster resizing, and enhanced load balancing tools, and of course support. Beyond that we have a few other things in the work that will speak to various revenue models for the future.

exhilaration 11y ago

AGPL license if anyone's curious: https://github.com/citusdata/citus/blob/master/LICENSE https://github.com/citusdata/citus/blob/master/LICENSE

rkrzr 11y ago
GNU AGPL to be precise (GNU Affero General Public License).
polskibus 11y ago
Does that mean that whatever connects to this database needs to be AGPL too?
- crudbug 11y ago
  No .. it means any changes you make to CitusDB should be made public. [0] https://en.wikipedia.org/wiki/Affero_General_Public_License https://en.wikipedia.org/wiki/Affero_General_Public_License
- kodablah 11y ago
  No, it means if you make changes to the code and use it over a network you have to be AGPL too (think "network" as "distribution" in the GPL sense).
  - hlandau 11y ago
    Nope. https://www.devever.net/~hl/agplunenforceable https://www.devever.net/~hl/agplunenforceable
    - kod 11y ago
      This is a really bad argument, probably wouldn't stand up in court, and in any case doesn't have precedent. Hint: How did you (the developer) originally acquire a _copy_ of the AGPL'd sourcecode?
      - 11y ago

no1youknowz 11y ago

This is awesome. I have experience with running a CitusDB cluster and it pretty much solved a lot of the scaling problems I was having at the time. For it to go open source now, is of huge benefit to the future projects I have. > With the release of newly open sourced Citus v5.0, pg_shard's codebase has been merged into Citus... This is fantastic, sounds like the setup process is much simpler. I wonder if they have introduced the Active/Active Master solution they were working on? I know before, there is 1 Master and multiple Worker nodes. The solution before was to have a passive backup of the Master. If say, they released the Active/Active Master later on this year. That's huge. I can pretty much think of my DB solution as done at this point.

ozgune 11y ago
(Ozgun from Citus Data) We're working on making Citus masterless. In all openness, we evaluated two different approaches to this in the past six months, and wrapped up the design for one. This design works well on the cloud, and we already demonstrated a working version: https://youtu.be/_nun2S6EdWo?t=411 https://youtu.be/_nun2S6EdWo?t=411 For on-premise deployments, the primary challenge is set-up complexity. We're now prototyping one of those designs to know more: https://github.com/citusdata/citus/issues/389 https://github.com/citusdata/citus/issues/389 We expect to share all the details and a concrete timeline in April.
- no1youknowz 11y ago
  Excellent news! Really looking forward to this one! :)
- batbomb 11y ago
  Can Citus handle geospatial sharding?
  - tracker1 11y ago
    You could always compute a geohash and use that as a shard key... I'm not familiar enough with Citus' specific approach here, but using a limited geohash would give you close to what you're looking for.

onRoadAgain23 11y ago

Being burned before,I will never use an OS infrastructure project that has enterprise features you need to pay for. They always try to move you to paid and make the OSS version unpleasant to use over time as soon as the bean counters take over to milk you "For customers with large production deployments, we also offer an enterprise edition that comes with additional functionality"

sergiosgc 11y ago
This is the model used by many companies backing OSS. The fact that you have been burned before means the actor in that case (or cases) acted badly, not that the model is wrong. Software isn't free to produce, and the need to make money off software isn't something companies should be ashamed of. In fact, nowadays I'm leaning towards trusting OSS with clear financial sustainability over software whose long term existence seems shaky.
gdulli 11y ago
Do you often make big decisions based on extrapolation from so few data points? I use a major open source system with enterprise features and support but don't pay for any of those options. I've used it for 3 years and it's been invaluable. No pressure to start paying for anything. Some of its premium features have actually become free over that period. But I wouldn't decide that all open source systems with premium features are safe based on that experience.
- onRoadAgain23 11y ago
  If this costed roughly a million dollars, then yes. Especially if you're locked in like with a DB. I use nginx because even though it has this mode it would be easy to replace with something else.
jamespo 11y ago
Feel free to code your own
gtaylor 11y ago
What are some alternatives for paying their employees to develop these products that are 100% free? There are some excellent commercial organizations that drive some of the tech that many or even most of us have came to rely on. This goes all the way down to Linux and BSD itself.

X86BSD 11y ago

AGPL? This is dead in the water :( It will never be integrated into PG. What a shame. It should have been a 2 clause BSDL. Sigh.

andreasklinger 11y ago
might be intentional
tspiteri 11y ago
The BSDL does not make much economic sense to the company open sourcing their code; a new competitor would fork the code, make closed improvements, and merge any changes from the open source code. That means that the competitor is always gaining by a one-way flow of improvements. To use open source code, the more permissive the license the better. But to actually open your own code, BSDL is a very tough sell. That's also why they use the AGPL. With database systems, even if they were under the GPL, some competitors could just modify the system and run it on their own server with improvements, and offer just the service to their clients. Again, the improvements go one way only: since the competitor would not distribute the modified system, as it's running on their servers, they would not need to distribute source changes. With the AGPL, that loophole is closed.
- X86BSD 11y ago
  So you take a BSDL codebase, fork it, close it, make proprietary changes, profiting from the BSDL codebase, then slap the PG community in the face by open sourcing it under a more restrictive license hoping to benefit from the community you just slapped in the face but restricting competition. They are of course free to release their code under any license they wish. I just think releasing code under the *GPL when you profited from a liberal BSDL is a douche nozzle thing to do. But knock yourself out! This tells me all I need to know about the company.
  - tspiteri 11y ago
    I don't agree with you. The PG community is, I think, fine with that: that's exactly what the BSDL allows that the GPL doesn't, and they chose the BSDL. If the PG community don't like that, I really don't understand why they chose the BSDL.

erikb 11y ago

Unforking is a very smart decision. Postgres also has gained a lot of favour since MySQL was bought by Oracle. Altogether Citus has earned a lot of kudos for that move, at least with me, for all that may count!

azinman2 11y ago

I want it to be called citrus, which is what I always read it as....

jrochkind1 11y ago
Then I'd get confused and think it had something to do with Citrix.
- azinman2 11y ago
  A cute lemon logo might help :)
BinaryIdiot 11y ago
Yikes I keep calling it citrus. The word citrus is so ingrain I'm actually having a hard time pronouncing citus when I see the word. I didn't even notice I was doing it wrong until I saw your comment.
mrgreenfur 11y ago
Only after reading this did I realize it's NOT citrus. Thanks!

faizshah 11y ago

So this sounds similar to Pivotal's Greenplum which is also open source, can anyone compare the two?

frn 11y ago
Greenplum is based on postgres 8.2, with the featureset you'd expect from pg 8.2 - basically none of the additions after 2006 have merged to GP.
- faizshah 11y ago
  Ok, and what's the process like for disaster recovery with citus?
  - craigkerstiens 11y ago
    That depends on your setup, for the master instance you'd run it just as you would for other setups. Streaming replication is common there. For the sharded instances, Citus has the ability for you to set what your replication factor is. Here Citus is then aware of when a node fails and will automatically redistribute the data to a new node, essentially taking care of that for you.
- ioltas 11y ago
  Since the move to open source, more recent upstream changes have been slowly merged in the code base, though they seem to be still on a 8.3 base, still a couple of years worth of code to go through.
amitlan 11y ago
Greenplum is a fork of Postgres codebase, Citus is not; it's an extension that leverages community Postgres's extensibilty APIs. This point seems to be highlighted in their post.

lcfcjs 11y ago

Postgres just doesn't scale, learnt that one the hard way

Dowwie 11y ago

is it correct to compare citusdb with pipelinedb?

BinaryIdiot 11y ago

I don't have a ton of experience scaling out and using different flavors of PostgreSQL but I had run across Postgres-XL not long ago; does anyone know how this compares to that?

Dowwie 11y ago

would a natural evolutionary path for start ups be to emerge with postgresql and grow to requiring citusdb?

devit 11y ago

I've been unable to find any clear description of the capabilities of Citus and competing solutions (postgres-x2 seems the other leader). Which of these are supported: 1. Full PostgreSQL SQL language 2. All isolation levels including Serializable (in the sense that they actually provide the same guarantees as normal PostgreSQL) 3. Never losing any committed data on sub-majority failures (i.e. synchronous replication) 4. Ability to automatically distribute the data (i.e. sharding) 5. Ability to replicate the data instead or in addition to sharding 6. Transactionally-correct read scalability 7. Transactionally-correct write scalability where possible (i.e. multi-master replication) 8. Automatic configuration only requiring to specify some sort of "cluster identifier" the node belongs to

cowardlydragon 11y ago
But if they answer those questions, you won't buy support/use it... Have a donut and look at our marketing spreadsheets. I'm so tired of "seamless" "effortless" "simple" distributed database lies. There's mathematical theorems as to why there is no free lunch.
ozgune 11y ago
(Ozgun from Citus Data) On PostgreSQL language support, we're updating our FAQ to have more information: https://www.citusdata.com/frequently-asked-questions https://www.citusdata.com/frequently-asked-questions Since the PostgreSQL manual (and its feature set) spans over 4K+ pages, we found that the best way to think about Citus' capabilities is from a use-case standpoint. If your workload needs distributed transactions that span across machines, or large ETL jobs, Citus currently isn't the best fit. Citus supports sharding and replication out of the box (#4, #5). On #6, reads go through a master node (metadata server) and you see what you write. We don't have #7. The way in which we implement this also has implications on your other questions. Multi-master (no single metadata server) is by far the biggest feature request that we receive: https://news.ycombinator.com/item?id=11353866 https://news.ycombinator.com/item?id=11353866 If we go with the approach in https://github.com/citusdata/citus/issues/389 https://github.com/citusdata/citus/issues/389, you will be able to configure #3, #6, #7 through PostgreSQL's streaming replication settings. We still won't support distributed transactions that span across multiple machines. On #8, could you elaborate a bit more? Do you mean a logical identifier for the node? Also, it's hard to write a concise reply on a topic that requires so much context. I'd love to grab coffee with anyone who's interested in diving deep into distributed databases. Feel free to shoot me an email at ozgun@citusdata.com

signalnine 11y ago

Congrats from Agari! We've been looking forward to this and continue to get a lot of value from both the product and the top-notch support.

ccleve 11y ago

I'd very much like to see what algorithm these systems are using to enable transactions in a distributed environment. Are they just using straight two-phase commit, and letting the whole transaction fail if a single server goes down? Or are are they getting fancy and doing some kind of replication with consensus?

wmfiv 11y ago
I believe transactions must process against a single node.

voctor 11y ago

Citus can parallelize SQL queries across a cluster and across multiple CPU cores. How does it compare with the upcoming 9.6 version of PostgreSQL which will support parallel-able sequential scans, parallel joins and parallel aggregate ?

lfittl 11y ago
AFAIK all the parallel work done in 9.6 refers to parallel operations on a single node (but multiple cores). This would be complimentary to what Citus does, which is distributing the load across multiple shard instances (each with their own cores, benefiting from the parallel work in 9.6).
- voctor 11y ago
  Yes, but Citus can also parallelize on multiple cores when used on a single machine ("If you’re running Citus on a single machine, this will scale queries across multiple CPU cores. and create the impression of sharding across databases."). Will this functionality becomes obsolete with the 9.6 ?
  - lfittl 11y ago
    Fair point - I assume this will be merged together in some way (i.e. the Citus stuff building on top of the parallel scan infrastructure), but probably a better question to ask on #citus IRC / open a Github issue. Some Postgres committers work on Citus as well (e.g. Andres Freud), so I'm sure this has been thought through before.

ismail 11y ago

Any thoughts on using something like postgres+citrus vs hadoop+hbase+ecosystem vs druid for olap/analytics with very large volumes of data

Someone 11y ago

One must thank them for open sourcing this, and cannot blame them for using a different license, but using a different license makes me think calling this "unfork" is bending the truth a little bit.

anarazel 11y ago
The "unfork" part is primarily about not forking the postgres codebase anymore, as done before citus 5.0 (i.e. we modified parts of postgres, to make it citus). Citus now entirely works as an extension to postgres, using the extension facilities postgres provides.
takeda 11y ago
Perhaps I'm missing something, but this is just an extension that works with standard postgres, there are no code changes in postgres itself, so it doesn't look like it ever was a fork.
- jasonmp85 11y ago
  (Jason from Citus here) Yes, that's what you're seeing right now, but in the past Citus (used to be "CitusDB") was a superset of the entire PostgreSQL codebase. During the lead-up to the open source release, we removed the use of any static methods or internal machinery and rewrote the installation process to use the PostgreSQL CREATE EXTENSION command. Additionally, we moved all of pg_shard's DML functionality into Citus to unify the product line. So ultimately CitusDB was a fork but is now entirely an extension.
  - Someone 11y ago
    Aha. Should have read the announcement better. Thanks for the explanation.

satygeek 11y ago

Does CitusDb fit in olap analytical workloads to do aggregations on hundreds millions of records using varying order and size of dimensions (eg druid) in max of 3 seconds response time using as few boxes as possible - Or there are other techniques have to be used along with Citusdb? Can you shed a light on your experience with CloudFlare in terms of cluster size and queries perf?

lfittl 11y ago
I'll leave the initial questions to the Citus team, but re: CloudFlare this link might be helpful: https://blog.cloudflare.com/scaling-out-postgresql-for-cloudflare-analytics-using-citusdb/ https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
- satygeek 11y ago
  Thanks. I went through it but couldnt find info about their cluster size, data size and queries response time
greggyb 11y ago
Hundreds of millions of records in <=3 seconds is not really a big challenge with a good data model and proper indexing on even a single server. I work for a BI consultancy and we don't even bat an eye until we hit billions of records in a primary fact table. Certainly the DB server does need to scale vertically to some extent as you pass through the orders of magnitude > 10M. A good columnstore engine is also worthwhile to consider.
mslot 11y ago
Yes, Citus may be a good fit, for a complete example see: https://www.citusdata.com/blog/15-marco-slot/402-interactive-analytics-github-data-using-postgresql-citus https://www.citusdata.com/blog/15-marco-slot/402-interactive...

ahachete 11y ago

Congratulations, Citus. Since I heard last year at PgConfSV that you will be releasing CitusDB 5.0 as open source, I've been waiting for this moment to come. It makes 9.5's awesome capabilities to be augmented with sharding and distributed queries. While this targets real-time analytics and OLAP scenarios, being an open source extension to 9.5 means that a whole lot of users will benefit from this, even under more OLTP-like scenarios. Now that Citus is open source, ToroDB will add a new CitusDB backend soon, to scale-out the Citus way, rather than in a Mongo way :) Keep up with the good work!

ioltas 11y ago

Congrats to all for the release. That's a lot of work accomplished.

ksec 11y ago

Does anyone know How does Citus compared to Postgre XL ?

albasha 11y ago

I recently switched back to MariaDB because I didn't see a clear/easy path for Postgres scalability in case the project i am working on takes off. I am under the assumption there are at least two fairly simple approaches to scale MySQL; master-master replication using Galera and Aurora from AWS. What do you guys think? Am I right in thinking MySQL is easier to scale given I want to spend the least amount of time maintaining it.

lobster_johnson 11y ago

This is great! One thing I'm having trouble with is finding information about transactional semantics. If I make several updates (to differently sharded keys) in a single transaction, will the transaction boundaries be preserved (committed "locally" first, then replicated atomically to shards)? Or will they fan out to different shards with separate begin/commit statements? Or without transactional boundaries at all? In fact, I can't really find any information on how CitusDB achieves its transparent sharding for queries and writes. Does it add triggers to distributed tables to rewrite inserts, updates and deletes? Or are tables renamed and replaced with foreign tables? I wish the documentation was a bit more extensive.

lambdafunc 11y ago

Any benchmarks comparing CitusDB against Presto?

uberneo 11y ago

Great product - If would be nice to have a Admin interface like RethinkDB where you can clearly define your replication and Sharding settings. Any documentation around how to do this from command line ?