3 ms·

Show HN: My Private GitHub on Postgres

42 points by calebhwin 5mo ago

vishal_ch 5mo ago
Interesting approach using Postgres as the storage layer. Curious how you're handling the object model since Git's content-addressable storage maps pretty differently to relational tables. Are you storing blobs as bytea or going with something like a JSONB tree structure for the commit graph?
- munk-a 5mo ago
  While git internally uses a pretty loose system for connecting different model concepts that has always seemed more like a concession to the storage medium than a desired step. If git existed on an already ACID compliant system instead of trying to build one out of the filesystem itself I don't see a reason to keep all the references as loose as they are. If you can cascade changes with confidence you can likely just switch to using standard surrogate keys for linkages and allow the data to normalize more fully. The core model objects in git are all pretty straightforward and their interactions well defined.
Mic92 5mo ago
Nice idea.
JasonHEIN 5mo ago
Great idea
hk1337 5mo ago
Interesting idea but what's the use case for this? Why wouldn't I just create a private git server (gitlab, forgejo, etc) just for myself?
- hungryhobbit 5mo ago
  This seems like the elephant in the room. I'm not saying this project isn't cool, but whenever you have ANY software that's designed to be hosted A-style, and you host it B-style, the obvious question is "Why not host it the A way?"
- somat 5mo ago
  I suspect postgress just brings efficient queries. My initial thought was how fossil uses sqlite as a backing store. but... not only is sqlite intentionally designed as an interchange format(stable specification). the postgress disk structure is intentionally designed to not be a interchange format(they reserve the right to change it at any time) so not that. So the only real reason is you already have a postgres server and want the efficient query indexes. As an interesting side note. I found this document on the internal data structure of fossil. https://fossil-scm.org/home/doc/trunk/www/fossil-is-not-relational.md https://fossil-scm.org/home/doc/trunk/www/fossil-is-not-rela...
- tensegrist 5mo ago
  as the other replies mention, efficient querying can be fun https://oseifert.ch/blog/building-pgit https://oseifert.ch/blog/building-pgit
throwatdem12311 5mo ago
Just use Fossil at this point.
- sikozu 5mo ago
  I'm waiting for somebody to create fossilhub
  - somat 5mo ago
    Already exists. There are fossil hosting sites, which is probably what you are talking about. I don't use one but here is an example. https://chiselapp.com/ https://chiselapp.com/ But fossil itself can already serve many projects acting like a self contained fossil hub.
- lagniappe 5mo ago
  Fossil really has it all.
xp84 5mo ago
"doesn't support: ... Web UI." So, it's a git server with an interesting storage layer? Don't get me wrong, that part sounds like it might have been a ton of work to implement, but I think the web UI (pull requests, etc) is a lot of what Github has won on historically. Basically I don't feel qualified to judge the product itself, but I think positioning it against Github, while popular given the recent hard times, isn't quite correct.
- nomel 5mo ago
  > "doesn't support: ... Web UI." "Doesn't" doesn't mean "can't". Someone just needs to do the work (with no thanks or pay expected). edit: the perspective of open source projects has really changed in the last 10 years, from collaboration to nice personal projects now being referred to as "the product".
  - xp84 5mo ago
    I apologize if that’s how my tone has come across. I think I just got distracted by the comparison. I think it’s very cool as a project.
  - SahAssar 5mo ago
    > "Doesn't" doesn't mean "can't". Someone just needs to do the work Sure, but isn't that the case for any feature of any software?
    - nomel 5mo ago
      Sure, but the context changes who "someone" can be. In this case, it's a personal open source project with pull requests open, where that "someone" means "anyone". We can implement the feature if we want, and for our efforts, we must expect as much in return as the original author received: nothing. Open source used to be abut collaboration, now it's mostly abuse. If you want to add a feature to GitHub, that "someone" would neccesarily be a paid employee.
iririririr 5mo ago
just use ssh and git bare.
lisperforlife 5mo ago
This is really cool. PG has zlib compression on TOAST objects so this should still be okay even if you are not storing pack files. I am curious with your choice of hand-rolling pktline, upload-pack and receive pack implementations including rev-walking. Any particular reason you did not want to use libgit2 or something like the gitoxide implementation of pkt-line. Was it performance or is it because you wanted it to be in pure rust? Did you try running this on slightly heavier repository with a lot of commits, refs and objects?
- justinclift 5mo ago
  > PG has zlib compression on TOAST objects so this should still be okay even if you are not storing pack files. Along those lines, even zlib could probably be skipped if the underlying filesystem is a compression capable one. ZFS can, though there are probably others as well.
bitbasher 5mo ago
No license?
deleted 5mo ago
[deleted]
supriyo-biswas 5mo ago
I've always wanted to write something like this. The problem with Gitlab/Gitea etc. is their reliance on disk storage; which means self hosting them requires that I get the backup story just right. Whereas with this, I could just handle it as part of the database backup process. Having no web UI, at least even a rudimentary one is kinda a bummer though.
- subhobroto 5mo ago
  I've struggled with this decision myself but I came to the opposite conclusion as you: - Gitea's (I use Forgejo) reliance on disk storage for `.git` is perfect for me because files are well understood as a concept by most people. (To be clear, Gitea/Forgejo stores non `.git` artifacts in PostgreSQL.) Every battle hardened linux tool knows how to backup files. Plain old `rsync` can backup and restore files. I have heard people put their `.git` on something like Dropbox and have tit work both for sync and backup (I've never tried it myself). You can run checksums on files and ensure they are exactly how you expect them to be. There are multiple, well tested, well understood options to reliably backup, snapshot and restore files. Also, remote/cloud storage for files is really cheap. In most cases, if it's less than 10GB, you likely don't have to pay anything at all, as in $0 every month for having a backup on servers that won't go up in flames even if your laptop or house did. - OTOH, PostgreSQL backup and restore feels like they are less popular or accessible to the general population vs files' backup and restore. Infact, for non DBA folks who don't necessarily understand PostgreSQL WAL, backup snapshotting, what asynchronous and synchronous WAL replication means and how they affect RTO and RPO, there are definitely multiple and non-obvious ways to get more things wrong than right, and lose your data - something you wouldn't have to worry about when using files backup and restore. > Whereas with this, I could just handle it as part of the database backup process What's the database backup and restore process you follow right now and what are the tools you use?
solid_fuel 5mo ago
I can't even view the commits because GitHub is claiming I'm over a request rate limit. This is the first time I've even opened GitHub today. Time to knock another 9 off their status page.
Backtrawen 5mo ago
[dead]

42 points by calebhwin 5mo ago

Interesting approach using Postgres as the storage layer. Curious how you're handling the object model since Git's content-addressable storage maps pretty differently to relational tables. Are you storing blobs as bytea or going with something like a JSONB tree structure for the commit graph?

munk-a 5mo ago
While git internally uses a pretty loose system for connecting different model concepts that has always seemed more like a concession to the storage medium than a desired step. If git existed on an already ACID compliant system instead of trying to build one out of the filesystem itself I don't see a reason to keep all the references as loose as they are. If you can cascade changes with confidence you can likely just switch to using standard surrogate keys for linkages and allow the data to normalize more fully. The core model objects in git are all pretty straightforward and their interactions well defined.

Mic92 5mo ago

Nice idea.

JasonHEIN 5mo ago

Great idea

hk1337 5mo ago

Interesting idea but what's the use case for this? Why wouldn't I just create a private git server (gitlab, forgejo, etc) just for myself?

hungryhobbit 5mo ago
This seems like the elephant in the room. I'm not saying this project isn't cool, but whenever you have ANY software that's designed to be hosted A-style, and you host it B-style, the obvious question is "Why not host it the A way?"
somat 5mo ago
I suspect postgress just brings efficient queries. My initial thought was how fossil uses sqlite as a backing store. but... not only is sqlite intentionally designed as an interchange format(stable specification). the postgress disk structure is intentionally designed to not be a interchange format(they reserve the right to change it at any time) so not that. So the only real reason is you already have a postgres server and want the efficient query indexes. As an interesting side note. I found this document on the internal data structure of fossil. https://fossil-scm.org/home/doc/trunk/www/fossil-is-not-relational.md https://fossil-scm.org/home/doc/trunk/www/fossil-is-not-rela...
tensegrist 5mo ago
as the other replies mention, efficient querying can be fun https://oseifert.ch/blog/building-pgit https://oseifert.ch/blog/building-pgit

throwatdem12311 5mo ago

Just use Fossil at this point.

sikozu 5mo ago
I'm waiting for somebody to create fossilhub
- somat 5mo ago
  Already exists. There are fossil hosting sites, which is probably what you are talking about. I don't use one but here is an example. https://chiselapp.com/ https://chiselapp.com/ But fossil itself can already serve many projects acting like a self contained fossil hub.
lagniappe 5mo ago
Fossil really has it all.

xp84 5mo ago

"doesn't support: ... Web UI." So, it's a git server with an interesting storage layer? Don't get me wrong, that part sounds like it might have been a ton of work to implement, but I think the web UI (pull requests, etc) is a lot of what Github has won on historically. Basically I don't feel qualified to judge the product itself, but I think positioning it against Github, while popular given the recent hard times, isn't quite correct.

nomel 5mo ago
> "doesn't support: ... Web UI." "Doesn't" doesn't mean "can't". Someone just needs to do the work (with no thanks or pay expected). edit: the perspective of open source projects has really changed in the last 10 years, from collaboration to nice personal projects now being referred to as "the product".
- xp84 5mo ago
  I apologize if that’s how my tone has come across. I think I just got distracted by the comparison. I think it’s very cool as a project.
- SahAssar 5mo ago
  > "Doesn't" doesn't mean "can't". Someone just needs to do the work Sure, but isn't that the case for any feature of any software?
  - nomel 5mo ago
    Sure, but the context changes who "someone" can be. In this case, it's a personal open source project with pull requests open, where that "someone" means "anyone". We can implement the feature if we want, and for our efforts, we must expect as much in return as the original author received: nothing. Open source used to be abut collaboration, now it's mostly abuse. If you want to add a feature to GitHub, that "someone" would neccesarily be a paid employee.

iririririr 5mo ago

just use ssh and git bare.

lisperforlife 5mo ago

This is really cool. PG has zlib compression on TOAST objects so this should still be okay even if you are not storing pack files. I am curious with your choice of hand-rolling pktline, upload-pack and receive pack implementations including rev-walking. Any particular reason you did not want to use libgit2 or something like the gitoxide implementation of pkt-line. Was it performance or is it because you wanted it to be in pure rust? Did you try running this on slightly heavier repository with a lot of commits, refs and objects?

justinclift 5mo ago
> PG has zlib compression on TOAST objects so this should still be okay even if you are not storing pack files. Along those lines, even zlib could probably be skipped if the underlying filesystem is a compression capable one. ZFS can, though there are probably others as well.

bitbasher 5mo ago

No license?

deleted 5mo ago

[deleted]

supriyo-biswas 5mo ago

I've always wanted to write something like this. The problem with Gitlab/Gitea etc. is their reliance on disk storage; which means self hosting them requires that I get the backup story just right. Whereas with this, I could just handle it as part of the database backup process. Having no web UI, at least even a rudimentary one is kinda a bummer though.

subhobroto 5mo ago
I've struggled with this decision myself but I came to the opposite conclusion as you: - Gitea's (I use Forgejo) reliance on disk storage for `.git` is perfect for me because files are well understood as a concept by most people. (To be clear, Gitea/Forgejo stores non `.git` artifacts in PostgreSQL.) Every battle hardened linux tool knows how to backup files. Plain old `rsync` can backup and restore files. I have heard people put their `.git` on something like Dropbox and have tit work both for sync and backup (I've never tried it myself). You can run checksums on files and ensure they are exactly how you expect them to be. There are multiple, well tested, well understood options to reliably backup, snapshot and restore files. Also, remote/cloud storage for files is really cheap. In most cases, if it's less than 10GB, you likely don't have to pay anything at all, as in $0 every month for having a backup on servers that won't go up in flames even if your laptop or house did. - OTOH, PostgreSQL backup and restore feels like they are less popular or accessible to the general population vs files' backup and restore. Infact, for non DBA folks who don't necessarily understand PostgreSQL WAL, backup snapshotting, what asynchronous and synchronous WAL replication means and how they affect RTO and RPO, there are definitely multiple and non-obvious ways to get more things wrong than right, and lose your data - something you wouldn't have to worry about when using files backup and restore. > Whereas with this, I could just handle it as part of the database backup process What's the database backup and restore process you follow right now and what are the tools you use?

solid_fuel 5mo ago

I can't even view the commits because GitHub is claiming I'm over a request rate limit. This is the first time I've even opened GitHub today. Time to knock another 9 off their status page.

Backtrawen 5mo ago

[dead]