13 ms·
Connecting a Git Repository to Amazon S3 and AWS Services
- hgontijo 9y agoAll of this just to avoid creating a Git connector?!
- korzun 9y agoYou should understand what problem they are solving first; then comment.
- EgoIncarnate 9y agoand a 3rd party git repository: https://aws.amazon.com/quickstart/architecture/git-to-s3-using-webhooks/ https://aws.amazon.com/quickstart/architecture/git-to-s3-usi...
- ge96 9y agoI like the free t2.small EC2 instance, pretty sweet, got a private git server on there but still figuring out how to actually use Git.
- jshmrsn 9y agoJust to make sure you're aware, your free 750 hours per month will only be available for the first 12 months of your AWS account. According to the AWS website, it's t2.micro instances that have the available free hours, not t2.small. Make sure you're not getting unexpected costs in the billing section. See https://aws.amazon.com/free/ https://aws.amazon.com/free/
- ge96 9y agoYeah you're right, sorry said the wrong one (t2.micro) it's funny when you look at it, it seems cheap like $0.012/hr but you multiply that by 750 and yeah... I briefly went up to t2.medium and I was like damn $30/mo for a server, that's a lot (to me). OVH has nicer prices for roughly same specs eg. 2cores 8GB ram at I think $16/mo but that detachable/scalable is really nice without data loss on server configs. Yeah good point about the year, I was surprised when I was charged the $99.99 for Amazon Prime haha
- isatty 9y agoTrust me, you don't need scalability. For a personal server, get a VPS (get one from a provider that charges you a fixed monthly, don't get one with a shitty cpu (somewhere like DO/Linode does fine for your use case)).
- ge96 9y agoI use OVH right now I like them but I recently came across the free AWS tier as well, just messing around, a client insisted on using AWS so I set one up with CentOS (not regarding the private Git). Migrated their WP site from DO to AWS EC2. I'm messing with Private git because I can't justify the $7/mo cost through GitHub. yeah I run a vps too, multiple domains but I do like that functions as a service approach. I unfortunately started with GoDaddy haha hilarious, overpriced that economy $1 thing garbage. Just my opinions but yeah... had some caching problems too wouldn't replace the content ahhhhh. Saying it was my fault "flush dns" I'm testing with Tor externally in other countries. Also I'm using Ubuntu no flushing required I don't know. Edit: moving up to t2.medium was because I implemented defuse/php-encryption and it took a toll encrypting decrypting, didn't help I had a dumb mistake where the cloning of data was stacking and not deleting so it was getting longer and longer... I know can use a better language but PHP is what I know at this time. I was able to drop back down to a t2.small but damn MySQL man... uses so much resources... with 2GB RAM it's idling like at 1.4GB or something... it just grows. I know you can tweak the settings. It's built on WordPress, I'm suggesting to strip the function of the site out of WordPress but future improvement. Edit: not crapping on WordPress the plugins/dynamic aspect is cool and if I thoroughly understood it my shit wouldn't be so janky. I interfaced with an Evernote Sync plugin and was surprised the comments were in chinese haha but it's PHP so I could read the code. Using memcache(d) was cool, yeah it was a cool job all in all. Edit: You know what's hilarious is HIPAA compliant servers DAMN! That's expensive, I clearly was not using HIPAA servers as those were $300 - $600/mo at least. But not just that but the architecture of the site, I clearly expressed that I am not a security expert. But AWS goes over a bit on the design of separating interface with the storage of data. At least we've got zero-knowledge design of everything encrypted... that's kind of a problem in a way too, when the key-obj to encrypt is not known and you've got strings that you can't decrypt and you start getting "failed to decrypt" error messages like "oh shit..." I think that was just an initial config though hopefully. Had a recent never-ending loop problem wow... you can watch the server spike to 100% CPU usage in HTOP and you're like "ah crap something's wrong"
- sigjuice 9y agoYou don't need a private server at all to figure out how to use Git.
- ge96 9y agoNo I was just being cheap, I didn't want to pay for the Private Git server through Git Hub, but I do need to know how to use it. Like the push to live(instead of SFTP), separate branches, allocating resources assuming you're using Git on the same server that's hosting your site, apparently it's not bad if you don't have a lot of users/commits.
- rblatz 9y agoGit is decentralized, you don't need any server anywhere for it. You could just use it locally and email patches around.
- freedomben 9y agoYes you COULD, but you can also haul water into your home via buckets from the well :-) But setting up a git server is so easy if you don't care about having a UI like github or gitlab. Git talks SSH, so pretty much just spin up a VPS, get it SSHable, then install git, create a bare repo, and push to that path.
- rblatz 9y agoOh yeah I don't use it that way, but if you just want to learn how to use git setting up a server isn't needed. Also I agree that emailing patches seems like an archaic way to do things, but that's how Linux handles PRs. http://elinux.org/Patch_Submission_HOWTO http://elinux.org/Patch_Submission_HOWTO
- sigjuice 9y agoWhile learning git, if you want a place to push, an SSHable place is still unnecessary. The bare repo can simply be another directory on the same system.
- joshka 9y agoIf you don't want to run your own server, https://aws.amazon.com/codecommit/pricing/ https://aws.amazon.com/codecommit/pricing/ is pretty much free for many use cases
- samueloph 9y agoFun fact about codecommit: If you want to set up a mirror from other repository (eg Github), you'll have to have an ec2 instance just to deal with the mirroring (codecommit doesn't support)[1]. Yeah, no thanks, if i'll have to spend money on an EC2 instance to run my mirror, i rather use this ec2 instance to run the mirror itself (no need to spend more money with codecommit on my usecase). Also, it's really strange that codecommit doesn't offer this option, i know gitlab does. [1]https://aws.amazon.com/blogs/devops/replicating-and-automating-sync-ups-for-a-repository-with-aws-codecommit/ https://aws.amazon.com/blogs/devops/replicating-and-automati...
- markshead 9y agoIf you want a Git repo hosted on Amazon, I think you can get 50 Gigs of storage using their CodeCommit service for free. It obviously isn't the same as learning how to set things up on the server for yourself, but it might be useful depending on what you are trying to do. https://aws.amazon.com/codecommit/ https://aws.amazon.com/codecommit/
- rubenbe 9y agoFrom the title of this post, I was expecting something like Dulwich [0] which has the ability to store individual objects in an Swift object store. But instead it is a webhook which reacts to pushes on a repo. [0] https://github.com/jelmer/dulwich/blob/master/README.swift.md https://github.com/jelmer/dulwich/blob/master/README.swift.m... Edit: title has been improved in the meantime
- falsedan 9y agoI thought it would be about jgit's S3 support[0], or git-remote-helpers[1]. 0: http://download.eclipse.org/jgit/docs/jgit-2.0.0.201206130900-r/apidocs/org/eclipse/jgit/transport/AmazonS3.html http://download.eclipse.org/jgit/docs/jgit-2.0.0.20120613090... 1: https://git-scm.com/docs/git-remote-helpers https://git-scm.com/docs/git-remote-helpers
- berlam 9y agoA few months back I created a tool [0], which deploys static content from GitHub to S3, with GitHub push events and AWS Lambda. This was more like a PoC but worked quite well for me. During development I was very happy with the usage of jgit, which allowed me to stay in the Java stack (even if it is not the perfect choice for Lambda imho). There was and still is much on the ToDo list. If you are interested in the sources, you can take a look at it. [0] https://github.com/berlam/github-bucket https://github.com/berlam/github-bucket
- falsedan 9y agoNice! I'm planning to make my travis build handle the publishing to S3.
- rsync 9y agoI don't know about lambda functions but I think you can do all of this with rsync.net since we have both s3cmd and git in our environment: ssh user@rsync.net s3cmd get s3://rsync/mscdex.exe ssh user@rsync.net "git clone git://github.com/freebsd/freebsd.git freebsd" So you could run a cron job that would clone a repo to your rsync.net account and then publish (or backup) that repo to an s3 account. Or whatever.
- korzun 9y agoThis allows you to hook your repository directly to Amazon AWS. You are suggesting the exact opposite.
- falsedan 9y agoSurely the exact opposite is, whenever a new object is written to some S3 bucket, it gets automatically added to a git repo in GitHub…
- rsync 9y agoI'm not suggesting anything. Those two tools (s3cmd and git) are in place and you can do whatever you'd like with them in whatever order. I just pasted those two example commands in that order because that's what order they are in on our howto page...
- Kpourdeilami 9y agoIf you're planning to use this as a way to deploy your static website to S3, I highly recommend using Netlify[0] instead. I have configured it to automatically publish anything that I put on the `deploy` branch and it supports letsencrypt certificates as well as rewrite rules 0: https://www.netlify.com https://www.netlify.com
- marban 9y ago+1 Also way better than surge.sh in this area.