6 ms·
Ask HN: Any C programming Postgres wizards willing to port 460 lines of code?
I'd really like to use Postgres as a backing store for libgit2 https://libgit2.github.com/
I wonder if there are any kind C programming wizards who know Postgres and might consider doing the open source port? It's beyond my Python programming skills and I dare not write crappy C code for fear of creating something nasty and insecure.
I can repay either by reciprocating with Python/web development/Linux/AWS knowledge, or if I have nothing of value to offer then I can offer thanks and praise.
The existing MySQL implementation is 460 lines of code.
There's a MySQL implementation here:
https://github.com/libgit2/libgit2-backends/blob/master/mysql/mysql.c
There's a sqlite implementation too:
https://github.com/libgit2/libgit2-backends/blob/master/sqlite/sqlite.c
Some relevant links:
http://blog.deveo.com/your-git-repository-in-a-database-pluggable-backends-in-libgit2/
- lfowles 11y agoIf it's so short, why not give it a shot and put it up for code review so others can catch the nasty and insecure bits? Great learning experience :)
- collyw 11y agoWhere would you put it up for review?
- _asummers 11y agohttps://codereview.stackexchange.com https://codereview.stackexchange.com or the project itself.
- deleted 11y ago[deleted]
- phuff 11y agoPut it up as a gist on github. Or put it up as a PR on github. Branches are cheap! :)
- atsaloli 11y agoThe PostgreSQL developers team lives on the pgsql-hackers mailing list (http://www.postgresql.org/list/pgsql-hackers/ http://www.postgresql.org/list/pgsql-hackers/) which is appropriate for discussion of current development issues, problems and bugs, and PROPOSED NEW FEATURES. =) See also http://www.postgresql.org/developer/ http://www.postgresql.org/developer/
- andrewstuart 11y agoIt would be good if it was awesomely and expertly done rather than amateurish, which my fumbling would result it. Good idea though, time I started getting some C skills but on a less important bit of code.
- phuff 11y agoCourage! You can do it! It would also be a great learning opportunity since you have the other 2 libs to use as an example. You should go for it and then put it up for code reviews before using it. Seriously! Give it a shot and then ask HN to review it in another Ask HN.
- exDM69 11y agoThis doesn't look too difficult to do (definitely not a "wizard" level task), and I think there will be more work in setting up tests, etc than actually writing the code. A good learning opportunity so if you've got the time go ahead and do it yourself. If you end up doing it yourself, I'll volunteer to do a quick review.
- angersock 11y agoUnless you're doing this for a business use case (which your insistence on expert quality and security earlier kinda suggest), it shouldn't matter. Don't fish for free work when you could turn this into a learning opportunity. Besides, why would you want to use PG as a backing store unless you were doing weird multitenant stuff?
- andrewstuart 11y agoI'm a sole developer in pyjamas coding in the lounge room trying to come up with an idea that will make an income when I'm not doing my day job to pay the bills, in case you're asking if I am a rich company trying to get free development services. I'm happy to reciprocate technical services if what I know is of value. I write open source code too so I'm contributing although not to the libgit2 project. Same as many other developers here on HN I suspect. Coding in the hope of building something people want to use.
- Zenst 11y agoI had a quick look and agreed for a learning exercise this would be fantastic and given you have two existing examples to compare with it would be possible to work it out and learn some C at the same time and even if you did not know C; You would be able to hack it together without much brain taxing.
- kungfooman 11y agoDidn't know that libgit2 has a MySQL backend, thanks. :D
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- mrkmcknz 11y agoI wish there were more Ask HN posts like this. It's also a great chance for someone who perhaps is looking for a gig to put some code up here that quite a few people will review.
- synthmeat 11y agoYou know what? I propose we use "Task HN:" and do these requests more frequently. We'll surely learn a lot, come to ingenious solutions, and maybe, just maybe, make our time on HN more productive. I, for one, want to at least see, if not help, what small but interesting byte-size hurdles others encounter and how others can solve it in different ways, and all the discussion around it. Working on something together binds communities even tighter.
- iwillinc 11y agoThis is such a good god damn idea.
- logicrime 11y agoYeah, what harm could come of asking people interested in news aggregation to complete tasks they have little interest in and little reason to do so? Surely the overall quality of HN won't suffer even more from people asking other hackers to write their C for them, what a great idea! fucking /s
- theVirginian 11y agoIf you don't like it then don't participate. Some of these projects might be interesting. Unlike you apparently I actually like programming and do have an interest in taking part.
- synthmeat 11y agoUninteresting tasks won't get upvoted. Interesting tasks get upvoted, and become a story in itself. Surely it won't have as detrimental effect as anonymous throwaway accounts rich with shallow sarcasm and curse words.
- logicrime 11y agoThe truth hurts, but lies kill. The large majority of HN readers are lurkers, who don't vote at all, the next largest group are those who aren't able to downvote, followed by the smallest group, the downvote-capable. I've got a few accounts that have downvote and dozens without, but it's not enough to stop the constant barrage of stupidity that floods what is supposed to be a news aggregation list. I don't mind seeing SHOW HN:'s, I like seeing what hackers are working on. On the other hand, a bunch of posts TASK HN: Write a C wrapper because I can't do it, that's gonna get old real fuckin' quick. I will shill as hard as I can, but I fear that it won't be enough.
- CyberDildonics 11y agoIs there a reason it couldn't be done with C++11/14 with an extern interface to C? Here is a lib to match python's string functions in C++. https://code.google.com/p/pystring/ https://code.google.com/p/pystring/ When working with string and vectors C++11 can actually be pretty straight forward, productive, and clear with no manual memory management. It definitely doesn't have as many string functions out of the box as modern scripting languages though.
- richardwooding 11y agoWow! My C knowledge is dated, but my Postgresql knowledge is good. Would be fun to attempt if I had time.
- adwmayer 11y agoHave you tried https://github.com/davidbalbert/libgit2-postgresql https://github.com/davidbalbert/libgit2-postgresql? I know nothing about libgit2, but came across this repo while looking it up. If it's got some issues it's a great place to start from, since it also gives you somebody who could code review and you don't need to start completely from scratch.
- lunixbochs 11y agoI did a quick review. This project uses PQescapeLiteral() and asprintf() to build queries. I'd recommend prepared statements over this approach (which is how the MySQL backend builds queries). It doesn't implement anything but write() and free() yet, so it's not actually functional. Unfortunately I believe this would clobber the library license (GPLv3 on this Postgres plugin vs GPLv2 with a linking exception for libgit). With this in mind I'd probably just recommend a rewrite. It's not that much code. Update: it looks like the libgit2 backends aren't well maintained: https://github.com/libgit2/libgit2-backends/issues/13#issuecomment-112319590 https://github.com/libgit2/libgit2-backends/issues/13#issuec...
- andrewstuart 11y agoI did notice it's a different license. Do you think there is value in porting the MySQL or SQLite version? They would share the libgit2 license.
- andrewstuart 11y agoWould you mind casting your expert eye please over the solution that has magically arrived in this thread? I'd be interested in your thoughts.
- userbinator 11y agoIt probably isn't the most efficient way, but were an ODBC backend written for it, libgit2 would be able to use any database that has an ODBC driver - which AFAIK is almost all SQL-based ones.
- ugexe 11y agoIt's sort of telling that you refuse to put any effort into showing anyone you can at least put together a crappy implementation to demonstrate understanding and contribute back to the same knowledge space (beginner knowledge is knowledge) you wish to extract from. The demand for the highest quality code for what is essentially begging is also not becoming. Even contributing a terrible implementation would tickle people's motivation button (so they can teach someone without building your entire project, show they are measurable better than someone, etc) "Beggars can't be choosers"
- andrewstuart 11y agoI don't see why I shouldn't ask if anyone is willing to help. 1: it's open source snd presumably would be committed to the libgit2 project. 2: it's likely to be useful to others 3: it's central, critical code that isn't well suited to beginners. Of course it should be quality code, it's responsible for storing data. 4: it would strongly benefit from the eye of a Postgres expert, which I am not 5: I do not have money to hire developers 6: I write open source code too so I contribute my time to the public
- scrollaway 11y agoDon't let the haters fool you - they're the same guys who told dropbox that they didn't see the point of their services over sharing usb flash drives. This is a good idea. Other people could abuse it but this is certainly not abuse, any more than it would be to just file a feature request as an issue on a project. Hope it comes through.
- csomar 11y agoI have no problem with such requests, and I think there should be more. But there is one issue with that. What's your motive? Are you looking for this to complete a $xxK client project? Or are you looking for this to help students understand a C implementation? It makes a world of difference. If you are getting paid for the result, then you should share with the developer. If you are not, and doing this for the greater good, then it's fine. We should really define the line here. There is enough exploitation in the dev world and the last thing we want is developers exploiting other developers.
- mhodgson 11y agoLucky for you I actually did exactly this over a year ago. We're not using it anymore so I'll just open source it for you: https://gist.github.com/mhodgson/d29bbd35e1a8db5e0800 https://gist.github.com/mhodgson/d29bbd35e1a8db5e0800 Please note that I also don't know much C, but this implementation does work. Also included is a Postgres version of the Ref DB backend (so nothing hits the filesystem). There are a few bits that are not implemented since we didn't have use for the reflog and those parts are technically optional. Would probably be good to get another set of eyes on this from someone much more familiar with C. Hope this helps!
- mhodgson 11y agoAlso, note that some of the code will need updating since it was written specifically with Ruby bindings in mind. Should be easy to spot and update/remove the rb_* method calls.
- andrewstuart 11y agoHow awesome is that! What is the license? Any reason you didn't use it in the end? What was your use case for it? Question for you... and I'll read the source in the morning...but just quickly does it prevent storage of duplicate objects? That's one of the main things Im interested in is saving space when multiple git repos contain exactly the same object. And THANKS again. Awesome. Can I do anything for you? Send you a bottle of wine? Help with some Python or Linux? If you put your contact in your profile I'll drop you an email.
- mhodgson 11y agoHappy to help. The license is MIT (just added). In terms of duplicating objects, I believe that if you do choose to store objects from many repos in the same table, they will NOT be duplicated and you will get your space savings. Don't take my word for it though. We actually did use this code in production for a period of time. In the end we realized that one of the main features of Git, immutability, didn't suit our needs well and we designed a versioning system based closely on Git, but built on Postgres directly. The main benefit of this is using primary keys as the object ids, instead of hashes of the content. This means we can change the content without changing the object's id (which in normal Git then means changing the tree, commit, and every parent commit). Good luck!
- andrewstuart 11y agoPerformance is an interesting question, I wonder how a Postgres backend would compare to serving git from file system.
- deleted 11y ago[deleted]
- _cbdev 11y agoI decided to do it just for the fun of it. https://github.com/cbdevnet/libgit2-backends/blob/master/postgres/postgres.c https://github.com/cbdevnet/libgit2-backends/blob/master/pos... Caveat Emptor: I did not test this since somehow the Debian package of libgit2-dev seems not to include the GIT_* constants. It should probably work, though.
- ed_blackburn 11y agoAs an aside is their any crossover or anticapted convergence between libgit2 and git?
- ed_blackburn 11y agoPresume from the down votes that's a contentious question? I never knew you could add different back ends to libgit2, maybe this'd be a useful feature to cmd line version? I'll answer my own question in anyone else is interested in the diversity of how git is or can be implemented: http://thread.gmane.org/gmane.comp.version-control.git/204210/focus=204270 http://thread.gmane.org/gmane.comp.version-control.git/20421...