4 ms·

Did you consider trying NewSQL DBs that would let you scale horizontally with much less complexity? Curious to see if the performance is comparable yet.

by d4l3k 9y ago

tschellenbach 9y ago
Postgres is generally a much better choice for relational data. For their hosted platform they should eventually consider Cassandra for some of the most commonly used tables. (the table storing revisions for instance)
YorickPeterse 9y ago
No, because it probably would have required a complete rewrite of GitLab to make it work.
- kodablah 9y ago
  User mentioned "trying". With an app that gets this much use, it might be reasonable to take a small piece and fork all writes to both the current + an experimental DB and then A/B test the reads. If this type of abstraction requires "a complete rewrite of GitLab" for "trying NewSQL DBs" then something is amiss.
  - mnutt 9y ago
    Given that Gitlab is open source, adding a NewSQL DB into the mix would be pushing that dependency to downstream users, who would then have to take on the maintenance burden. It may end up being the right call from a performance standpoint but just too complex to ask users to manage.
    - kodablah 9y ago
      This burden already exists with dependencies (and we should probably say downstream "admins" instead of "users" to clarify). A self-hosted app should not be hamstrung by its initial dependency choices. If tested and implemented properly it can go from optional experimental flag, to multiple-storage-backends-supported abstraction, to [maybe] even we're-migrating-to-new-storage-backend, etc. The primary downside in these cases is the flexibility often provided to plugin and extension writers which marries them to early tech choices. The more admins that demand low change count on forward movement are indirectly the reason many companies choose to offer SaaS only where the visibility and flexibility are removed. Note, I'm not saying this is the case with GitLab at all and I assume Postgres will remain the primary choice for most of their DB uses in perpetuity, but for some use cases there is better tech to use and the only reason to say no shouldn't just be a dependency addition (though it is often a good reason among many to say no).
    - deleted 9y ago
      [deleted]
  - YorickPeterse 9y ago
    It would require a complete rewrite for at least the data layer, as everything right now assumes an RDBMS. Further it would complicate installation as now everybody would have to use a fancy "NewSQL" database, some of which you may not even be able to run yourself.
tytytytytytytyt 9y ago
Are you talking about NoSQL databases, like Mongo and Redis?
- Floegipoky 9y ago
  I think NewSQL refers to database systems that still enforce ACID transactions, but use various strategies to avoid the scalability difficulties that have traditionally been associated with RDBMS as both consistent and available systems. https://en.wikipedia.org/wiki/NewSQL https://en.wikipedia.org/wiki/NewSQL

Did you consider trying NewSQL DBs that would let you scale horizontally with much less complexity? Curious to see if the performance is comparable yet.

by d4l3k 9y ago

Did you consider trying NewSQL DBs that would let you scale horizontally with much less complexity? Curious to see if the performance is comparable yet.

tschellenbach 9y ago

Postgres is generally a much better choice for relational data. For their hosted platform they should eventually consider Cassandra for some of the most commonly used tables. (the table storing revisions for instance)

YorickPeterse 9y ago

No, because it probably would have required a complete rewrite of GitLab to make it work.

kodablah 9y ago
User mentioned "trying". With an app that gets this much use, it might be reasonable to take a small piece and fork all writes to both the current + an experimental DB and then A/B test the reads. If this type of abstraction requires "a complete rewrite of GitLab" for "trying NewSQL DBs" then something is amiss.
- mnutt 9y ago
  Given that Gitlab is open source, adding a NewSQL DB into the mix would be pushing that dependency to downstream users, who would then have to take on the maintenance burden. It may end up being the right call from a performance standpoint but just too complex to ask users to manage.
  - kodablah 9y ago
    This burden already exists with dependencies (and we should probably say downstream "admins" instead of "users" to clarify). A self-hosted app should not be hamstrung by its initial dependency choices. If tested and implemented properly it can go from optional experimental flag, to multiple-storage-backends-supported abstraction, to [maybe] even we're-migrating-to-new-storage-backend, etc. The primary downside in these cases is the flexibility often provided to plugin and extension writers which marries them to early tech choices. The more admins that demand low change count on forward movement are indirectly the reason many companies choose to offer SaaS only where the visibility and flexibility are removed. Note, I'm not saying this is the case with GitLab at all and I assume Postgres will remain the primary choice for most of their DB uses in perpetuity, but for some use cases there is better tech to use and the only reason to say no shouldn't just be a dependency addition (though it is often a good reason among many to say no).
  - deleted 9y ago
    [deleted]
- YorickPeterse 9y ago
  It would require a complete rewrite for at least the data layer, as everything right now assumes an RDBMS. Further it would complicate installation as now everybody would have to use a fancy "NewSQL" database, some of which you may not even be able to run yourself.

tytytytytytytyt 9y ago

Are you talking about NoSQL databases, like Mongo and Redis?

Floegipoky 9y ago
I think NewSQL refers to database systems that still enforce ACID transactions, but use various strategies to avoid the scalability difficulties that have traditionally been associated with RDBMS as both consistent and available systems. https://en.wikipedia.org/wiki/NewSQL https://en.wikipedia.org/wiki/NewSQL