9 ms·

AWS CodeCommit

285 points by qnk 12y ago

arenaninja 12y ago
You mean to tell me I just spent a day last week setting up Jenkins, GitHub and a VM when I could've waited for my second week on the job and move it all to Amazon? That aside.. This looks great. If I'm not fully satisfied with my setup (right now I'm not) I might make the move to this. Lower friction is always welcome
- arafalov 12y ago
  "CodeCommit will be available in early 2015" - not all is lost.
- jarito 12y ago
  Doesn't seem like this addresses Jenkins? Just source hosting.
  - arenaninja 12y ago
    > CodeCommit integrates with AWS CodePipeline and AWS CodeDeploy to streamline your development and release process. CodeCommit keeps your repositories close to your build, staging, and production environments in the AWS Clou I think you're right
- mtbcoder 12y ago
  Personally I'd hold off committing any source code (outside of personal projects) to something this new anyway, especially if it's only your first week on the job.
- Gigablah 12y ago
  I switched from Jenkins to CircleCI (they offer a free plan now). Took me only 20 minutes to get everything running, and with Docker integration to boot. Really pleased with it.
kcorbitt 12y ago
To me, it looks like the biggest advantage over existing solutions is point 4: "Faster Development Lifecycle." If you're running all your infrastructure out of an AWS datacenter anyway moving your source control servers into the same center will make checkouts in automated deploys marginally faster.
- rynop 12y ago
  Have you ever used github for automated/autoscaled deploys? Github is too unstable to rely on. We autodeploy from s3 backed git repo via jgit.
  - fooforge 12y ago
    Actually, things are running pretty smooth for months now - see for yourself at: https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month. If that's not your experience, you can reach out to support@github.com and we'll look into it.
    - rynop 12y ago
      We love github, don't get me wrong (and we pay decent $ for it). But I'm looking at track record / experience with it doing production auto deploys for years. S3 has 9 9s of durability and is "closer" to where our servers run so there are less possible points of failure. We have had 0 issues since migrating to s3 backed git repo 2 years ago. We still use github for day-to-day scm mgmt as you guys have tons of extra value add.
- softdev12 12y ago
  Not only faster, but better. I love AWS and love how they keep rolling out new features to make deployment easier. My current development lifecycle isn't yet close to perfect because of all the gaps with piecing together different providers and tool sets. To the degree that things can be streamlined and automated, all the better. In the end, I'm hoping AWS comes out with a 1 click continuous integration solution - from dev to deploy.
djtidau 12y ago
Though many won't see any sort of advantage to switching, it would be very interesting to know what kind of effect this has on both Github and Bitbucket. There doesn't seem to be any major drawcard that will have people chomping at the bit to change over that I can see. I'm happy to be enlightened though. On another note, is it just me or has Amazon really been ramping up with their releases lately? It feels like there is something new each week at the moment.
gfunk911 12y ago
Competition is good, so I say hooray for this. That said, doesn't seem to offer a compelling alternative to Github, based on what they're currently saying. There could be value in tight integration with other Amazon tools, but that seems like it'll come more from intentional lock-in than added value.
- ruairinewman 12y ago
  I don't know. I used to work for AWS, and this sounds very link an internal tool that is used to manage the entire Amazon retail infrastructure. That being the case, this is going to be a lot more powerful and useful for online infrastructure (in AWS, obviously) than anything else out there. Even if it's not all that initially, I suspect the extra functionality will come sooner rather than later. It's somewhat annoying I know, but I'm not going to go into more detail as I'm not sure about the legality of my position should I do so. Edit: On review of their website, I had missed the announcements for Amazon CodePipeline and Amazon CodeDeploy, which between them provide all the missing functionality I hinted at above. And so yes, it looks like this is the internal tool I was referring to.
  - idunno246 12y ago
    In the announcement they said they are basically releasing Apollo.
    - yazaddaruvala 12y ago
      Apollo(http://aws.amazon.com/codedeploy http://aws.amazon.com/codedeploy) is nice, but Pipelines are easily my favorite internal tool, glad they are releasing it. http://aws.amazon.com/codepipeline/ http://aws.amazon.com/codepipeline/
    - ignoramous 12y ago
      CodeDeploy is essentially Apollo (lite, so to speak). CodePipelines is Amazon Pipelines. CodeCommit is persumably Amazon GitFarm (according to our former AWS engineer here). We've got an amazing Builder Tools team here at Amazon, I must say.
btucker 12y ago
This + GitLab might prove to be a great alternative to GitHub Enterprise.
- flurdy 12y ago
  Surely this replaces GitLab as well?
  - ceejayoz 12y ago
    I don't see any indication it comes with a web UI like Github/Gitlab.
    - jdub 12y ago
      There were screenshots shown during the re:Invent keynote. It looks remarkably similar to GitHub.
      - ceejayoz 12y ago
        The only screenshot I can find is https://media.amazonwebservices.com/blog/2014/code_commit_main_1.png https://media.amazonwebservices.com/blog/2014/code_commit_ma... but that looks like it's just a code browser. No issues, PRs, etc.
        jdub 12y ago
        Yep, it's only revision control plus user interface.
latch 12y ago
I recommend Gitlab for anyone looking for something Github-like. On a $10/m DO instance you can be up and running quickly [1]. There's a rake task for backups and it's easy to get it to sync to S3. [1] https://www.digitalocean.com/community/tutorials/how-to-use-the-gitlab-one-click-install-image-to-manage-git-repositories https://www.digitalocean.com/community/tutorials/how-to-use-...
- serkanh 12y ago
  Atlassian Stash is also a good solution. I believe it is $10/annual for the first 10 users. https://www.atlassian.com/software/stash https://www.atlassian.com/software/stash
- tthayer 12y ago
  Their omnibus installer is ridiculously easy to use if you don't mind seeing Chef messages scroll past for 20 minutes. I had it tied into Active Directory and everything in less than an hour after downloading it. Compare that to using apache-svn and those goddamn LDAP sync scripts and apache conf folder definitions and it feels like you're using some shit from the future.
foolinaround 12y ago
If they can integrate gitolite into this and make a breeze to set up, then that would be awesome. If they use custom permissions ( directly tied to IAM ), then it would be another closed solution like github.
justinsb 12y ago
I've thought about building GitHub / BitBucket tools (issue management, pull requests etc) that work with data stored in git repositories, as opposed to one big tool that owns the git repository and manages the data in a locked-up SQL database. We already have this for code browsers (e.g. cgit), but that alone can't compete with the "full suite" that others offer. It would have several advantages for the user: 1) You could pick your issue management separately from your code browser 2) You don't have to worry that the best tooling (Github) has some of the least reliable storage. 3) If it stored the data in git, it would work offline! The big downside is that having to "assemble your own" is more complicated for the user, and that it isn't Github/BitBucket's business model...
- stephenr 12y ago
  I like the idea of some good, open source tools to do the different bits, preferably focused on flexibility and ease of data access (as you mentioned) to me this would mean an issue tracker, and some kind of merge request/review tool. these could be paired with the likes of Gollum, providing a wiki. One thing though, I would really love to see them support more than just git. Mercurial shouldn't be hard to support, as it has largely similar concepts, and even though SVN is not as popular as it was at its peak, its still widely used and a very good choice for some workflows/use cases. In the same vane, I would love to see a decent stand-alone code browser that supports different VCS.
  - justinsb 12y ago
    Thanks - great input (and the Gollum suggestion is a good starting point!) On multiple VCS: On the one hand, git is so dominant (for better or worse) that I don't know if the complexity is worth it. On the other, perhaps git is so dominant because github is so dominant. The answer in open source land is probably to set up a reasonable abstraction layer and let interested people provide their own implementations of the VCS-interface.
    - stephenr 12y ago
      I think I should also clarify a little.. My key focus for this would be on a more unix-philosophy approach (separate tools that do one thing well). I like the idea of using a VCS repo (again, ideally any of those supported not just git) to store related information such as issues/tickets, etc however I don't know what the practical constraints of that would be, my first thought is that issues/tickets stored primarily in a (version controlled) filesystem (and then presumably indexed on a server for web views, searching etc) would not be as intuitive as say a wiki like gollum. If the issues/tickets tool used a flexible SQL model (e.g. choice of mysql, postgresql and sqlite for instance) I would be happy with that, as the data is still quite open (to the system, owner, not as much to the individual developers admittedly)
lentil_soup 12y ago
Off topic, but the sign up process is very bad. Why do they need full name, email, company name, role, address, phone number, catcha, etc to just sign up for more information?
- lentil_soup 12y ago
  and while we're at it, why the need for a "State" field that's only relevant to one country?
  - ceejayoz 12y ago
    Did you click the state field? There are states for Australia, Canada, China, India, and others listed.
- andsmi2 12y ago
  It actually isn't out till 2015.... so this is just asking if you are interested. Perhaps it's an MVP -- with a signup form....and if there is enough interest they will pursue it...
vladimirralev 12y ago
Strange use of "git-based" wording. Do I understand correctly, they implemented their own git command line binaries that you install to replace the original git binaries? Possibly a fork of git client and server?
- justinsb 12y ago
  Probably based on JGit/Gerrit. It is much easier to build "cloud" git (i.e. reliable, redundant git) that way than it is to make your disk reliable. Github went the reliable-disk route with DRBD, AFAIK, which is why Github is always going down ;-) I wrote a blog post about this: http://blog.justinsb.com/blog/2013/12/14/cloudata-day-8/ http://blog.justinsb.com/blog/2013/12/14/cloudata-day-8/
  - btown 12y ago
    Intriguing post! For those who hadn't heard about DRBD, it's http://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device http://en.wikipedia.org/wiki/Distributed_Replicated_Block_De... - it makes failover incredibly simple to integrate by pushing a lot of distributed-systems responsibility down to the filesystem level, but in doing so it forces you to have a single-master setup, and you can't take advantage of parts of your domain model that are well-suited to eventual consistency. Thinking about it, now that Riak supports strong (quorum-based) consistency on a per-bucket basis, Riak-backed scalable Git hosting would probably be relatively easy to implement. Looks like you were building your own system - did you ever release the code?
    - justinsb 12y ago
      Ooops... looks like I didn't link to the code in that post. The whole series was open source; here's the git code itself: https://github.com/justinsb/cloudata/tree/master/cloudata-git https://github.com/justinsb/cloudata/tree/master/cloudata-gi... The git object data itself goes into a blob-store (like S3); it can be stored without strong consistency. It turns out you only need to keep track of a very small amount of metadata consistently (the refs). Riak, etcd, DynamoDB or the Google Cloud DataStore would all be good choices, I think. I was working on an open-source implementation of Raft as part of this (called barge), but it isn't as reliable as the alternatives above - yet!
riveralabs 12y ago
I'm more interested in their pricing structure. Knowing Amazon, will we see something in the line of $0.25 per repo plus bandwidth.
- TillE 12y ago
  A nominal fee on top of their usual storage and bandwidth rates sounds about right. But I'm not sure if this gives you much compared to just pointing Mercurial/largefiles (or git-annex) at S3.
- aragot 12y ago
  Knowing Amazon, will we see something like $0.0013 per thousand non-ASCII character, "because this is what it costs them after compression".
thewopr 12y ago
Here's why I'm excited about this: Data Versioning Github is awesome for science. But my code and workflows often are just as reliant on the data I have as the code I've written. With Github, I always have to treat my data separately. A different workflow, a different storage location, different (and often manual) versioning. Now, I can start integrating my data into the same workflow. No size limits. Just drop it in and version it like everything else. 1GB? No problem. 500GB? Just pay the money. This is especially awesome because, as a scientist/dev, I do not want to stand up my own infrastructure and servers/vms. I don't want to do linux updates. I don't want to worry about things going down. It just needs to be there when I need it. When I don't need it anymore, I'll take it down. Done and done.
- blaedj 12y ago
  I've run into problems with large-ish files in git repos, binaries accidentally committed etc. I'm genuinely curious, is there a good way to use git for the size repositories that you mention?
  - bkeroack 12y ago
    Git is great for smaller binaries. Ideal in fact, given that it stores differences between revisions as binary deltas. For large >1GB files, I believe the diffing algorithm is the limiting factor (I would be interested in getting confirmation of that, though). For those files something like git-annex is useful (http://git-annex.branchable.com/ http://git-annex.branchable.com/) I've used git to push around a lot of binary application packages and it's very nice. Previously I was copying around 250-300MB of binaries for every deployment--after switching to a git workflow (via Elita) the binary changesets were typically around 12MB or so.
    - blaedj 12y ago
      Hmm, I'll have to take a look at git-annex, I've run across it but never investigated. Thanks!
  - notduncansmith 12y ago
oblio 12y ago
I'm shocked that at this point AWS is not offering a build service. Or are they?
- deleted 12y ago
  [deleted]
- deleted 12y ago
  [deleted]
- kgrin 12y ago
  AWS CodePipeline and AWS CodeDeploy (both announced simultaneously I think)
- kuhhk 12y ago
  They just announced Amazon CodePipeline and Amazon CodeDeploy. http://aws.amazon.com/codepipeline/ http://aws.amazon.com/codepipeline/ http://aws.amazon.com/codedeploy/ http://aws.amazon.com/codedeploy/
valarauca1 12y ago
>Worry about scaling your own infrastructure Does git have scaling issues? I know git manages the 60mil+ SLOC of the Linux Kernel, that's what it was designed for. I don't see how even medium/large size enterprise will have difficulty. Or am I missing something?
- maccard 12y ago
  Doesn't handle large binary files
  - deleted 12y ago
    [deleted]
  - MichaelGG 12y ago
    As I understand, this is precisely why companies like Plastic SCM exist: to service game dev and other projects that need to manage large assets.
    - maccard 12y ago
      Yep. We use Perforce in work, and from my experience, most other game dev's do too.
- cdr 12y ago
  Very few organizations use git the way the linux kernel project does. Most organizations use github, gitlab, etc, which definitely have had scaling issues.
  - tedchs 12y ago
    You might be surprised to learn the Linux Kernel is on Github: https://github.com/torvalds/linux https://github.com/torvalds/linux
    - stephenr 12y ago
      I don't think thats their sole central hosted copy of the repo though. I believe they use git.kernel.org
gexla 12y ago
Bezos to kids being born 3 years from now. "You kids are just features. Keep n eye on our announcements page." It's interesting how they seem to be adding all the stuff in the workflow that we are okay with just being good enough and don't benefit from bells and whistles. Github is very cool for managing an open source project, but for my own stuff just about anything which makes managing my repos relatively simple and saves me a bit of time is fine. I don't need much for an interface or features. Just being able to push a button and having a hosted repo ready for other users and with instructions for less technical contributors is awesome. If I'm already on Github, I'm probably staying there. Competition is good though. Competition will push Github to continue doing more to differentiate themselves as being something much greater than simply hosting Git repo's.
- whichdan 12y ago
  Does GitHub need to do anything beyond hosting git repos, though?
forrestthewoods 12y ago
Interesting. I recently started mixing GitHub and S3 for blog content and it's a pain in the ass. Asking the internet about pushing GitHub content to S3 results in just a bunch of mediocre scripts. I like it. I also look forward to Perforce's response. For versioned binary files Perforce is still the best. Competition, hooray!
EpicDavi 12y ago
My uncle is a DevOps person and said all of the new tools that Amazon is rolling out would put him out of his job if they caught on.
- jakejake 12y ago
  If I were a DevOps person I would learn and master all of the AWS tools (as well as competitor services). It could be a great opportunity rather than a threat.
- hga 12y ago
  But that would also a 100% bet on Amazon. There will also be a number of companies where a "small" DevOps teams leverages a large number of cheaper systems (owned, leased or whatever) for lower cost.
pearjuice 12y ago
I still have trouble trusting a company which I primarily know as "that all-round webshop" to get involved withy my tech stack. It feels as-if McDonalds would over night offer server infrastructure and get away with it.
- mrjatx 12y ago
  McDonalds doesn't make money from their website, which is the problem in your comment. If we were comparing McDonalds we would be comparing their fast food infrastructure. If you were an up and coming fast food burger joint that was going from 1 shop to 2 shops, and possibly more shops, you sure as hell would want to ask McDonalds for it's strategic advice and further advice on how to run your burger shop as well as possible. They've been there, they've done it, not that you need to follow their guidelines, but you definitely want their knowledge in the back of your pocket. They've got about 70 years of experience. Same with Amazon, who has to keep Amazon.com online or they lose massive amounts of money.
  - fletchowns 12y ago
    Does Amazon actually use AWS to host Amazon.com?
    - mrjatx 12y ago
      Geez, oh yes. They built their platform and decided to start reselling it. That's why Netflix, Pintrest, Airbnb, Expedia, Foursquare, NASA, the MLB all use AWS. They converted 100% in 2011 according to Jon Jenkins https://www.youtube.com/watch?v=dxk8b9rSKOo#t=449 https://www.youtube.com/watch?v=dxk8b9rSKOo#t=449
    - discodave 12y ago
      Most, but not all of Amazon.com runs on AWS.
sk5t 12y ago
A good start for AWS, but I doubt they will ever quite fill it out to Beanstalk's level for online browsing, commit reports, etc.
elchief 12y ago
Not to be a negative Nelly or anything, but with the speed at which Amazon is releasing this stuff, they are going to have to be a Google eventually and pull the rug out from under our feet, discontinuing several products.
- hga 12y ago
  Maybe they'll distinguish themselves by not doing any or very much of that at all. Do you have any candidates for service termination?
- debaserab2 12y ago
  These aren't being given out free, they are charged per usage - there is obvious incentives to keep the products alive. Correct me if I'm wrong, but most failed Google products had no direct revenue stream other than advertisements.
- ceejayoz 12y ago
  AWS SimpleDB is an example of a product that Google would've likely killed years ago but Amazon maintains. It hasn't gotten any significant update since 2010 (http://aws.amazon.com/blogs/aws/amazon-simpledb-consistency-enhancements/ http://aws.amazon.com/blogs/aws/amazon-simpledb-consistency-...) and, judging from the fact that it only gets a couple posts a month in its AWS Forums, it's very sparsely used.
cheshire137 12y ago
Uggh no way. The AWS Web Console is a beastly enough UI to get around, I have no interest in migrating away from the very comfortable, friendly Github interface.
msoad 12y ago
The best thing about GitHub is it's UI. It will take a long time to polish a UI like GitHub's

285 points by qnk 12y ago

You mean to tell me I just spent a day last week setting up Jenkins, GitHub and a VM when I could've waited for my second week on the job and move it all to Amazon? That aside.. This looks great. If I'm not fully satisfied with my setup (right now I'm not) I might make the move to this. Lower friction is always welcome

arafalov 12y ago
"CodeCommit will be available in early 2015" - not all is lost.
jarito 12y ago
Doesn't seem like this addresses Jenkins? Just source hosting.
- arenaninja 12y ago
  > CodeCommit integrates with AWS CodePipeline and AWS CodeDeploy to streamline your development and release process. CodeCommit keeps your repositories close to your build, staging, and production environments in the AWS Clou I think you're right
mtbcoder 12y ago
Personally I'd hold off committing any source code (outside of personal projects) to something this new anyway, especially if it's only your first week on the job.
Gigablah 12y ago
I switched from Jenkins to CircleCI (they offer a free plan now). Took me only 20 minutes to get everything running, and with Docker integration to boot. Really pleased with it.

kcorbitt 12y ago

To me, it looks like the biggest advantage over existing solutions is point 4: "Faster Development Lifecycle." If you're running all your infrastructure out of an AWS datacenter anyway moving your source control servers into the same center will make checkouts in automated deploys marginally faster.

rynop 12y ago
Have you ever used github for automated/autoscaled deploys? Github is too unstable to rely on. We autodeploy from s3 backed git repo via jgit.
- fooforge 12y ago
  Actually, things are running pretty smooth for months now - see for yourself at: https://status.github.com/graphs/past_month https://status.github.com/graphs/past_month. If that's not your experience, you can reach out to support@github.com and we'll look into it.
  - rynop 12y ago
    We love github, don't get me wrong (and we pay decent $ for it). But I'm looking at track record / experience with it doing production auto deploys for years. S3 has 9 9s of durability and is "closer" to where our servers run so there are less possible points of failure. We have had 0 issues since migrating to s3 backed git repo 2 years ago. We still use github for day-to-day scm mgmt as you guys have tons of extra value add.
softdev12 12y ago
Not only faster, but better. I love AWS and love how they keep rolling out new features to make deployment easier. My current development lifecycle isn't yet close to perfect because of all the gaps with piecing together different providers and tool sets. To the degree that things can be streamlined and automated, all the better. In the end, I'm hoping AWS comes out with a 1 click continuous integration solution - from dev to deploy.

djtidau 12y ago

Though many won't see any sort of advantage to switching, it would be very interesting to know what kind of effect this has on both Github and Bitbucket. There doesn't seem to be any major drawcard that will have people chomping at the bit to change over that I can see. I'm happy to be enlightened though. On another note, is it just me or has Amazon really been ramping up with their releases lately? It feels like there is something new each week at the moment.

gfunk911 12y ago

Competition is good, so I say hooray for this. That said, doesn't seem to offer a compelling alternative to Github, based on what they're currently saying. There could be value in tight integration with other Amazon tools, but that seems like it'll come more from intentional lock-in than added value.

ruairinewman 12y ago
I don't know. I used to work for AWS, and this sounds very link an internal tool that is used to manage the entire Amazon retail infrastructure. That being the case, this is going to be a lot more powerful and useful for online infrastructure (in AWS, obviously) than anything else out there. Even if it's not all that initially, I suspect the extra functionality will come sooner rather than later. It's somewhat annoying I know, but I'm not going to go into more detail as I'm not sure about the legality of my position should I do so. Edit: On review of their website, I had missed the announcements for Amazon CodePipeline and Amazon CodeDeploy, which between them provide all the missing functionality I hinted at above. And so yes, it looks like this is the internal tool I was referring to.
- idunno246 12y ago
  In the announcement they said they are basically releasing Apollo.
  - yazaddaruvala 12y ago
    Apollo(http://aws.amazon.com/codedeploy http://aws.amazon.com/codedeploy) is nice, but Pipelines are easily my favorite internal tool, glad they are releasing it. http://aws.amazon.com/codepipeline/ http://aws.amazon.com/codepipeline/
  - ignoramous 12y ago
    CodeDeploy is essentially Apollo (lite, so to speak). CodePipelines is Amazon Pipelines. CodeCommit is persumably Amazon GitFarm (according to our former AWS engineer here). We've got an amazing Builder Tools team here at Amazon, I must say.

btucker 12y ago

This + GitLab might prove to be a great alternative to GitHub Enterprise.

flurdy 12y ago
Surely this replaces GitLab as well?
- ceejayoz 12y ago
  I don't see any indication it comes with a web UI like Github/Gitlab.
  - jdub 12y ago
    There were screenshots shown during the re:Invent keynote. It looks remarkably similar to GitHub.
    - ceejayoz 12y ago
      The only screenshot I can find is https://media.amazonwebservices.com/blog/2014/code_commit_main_1.png https://media.amazonwebservices.com/blog/2014/code_commit_ma... but that looks like it's just a code browser. No issues, PRs, etc.
      - jdub 12y ago
        Yep, it's only revision control plus user interface.

latch 12y ago

I recommend Gitlab for anyone looking for something Github-like. On a $10/m DO instance you can be up and running quickly [1]. There's a rake task for backups and it's easy to get it to sync to S3. [1] https://www.digitalocean.com/community/tutorials/how-to-use-the-gitlab-one-click-install-image-to-manage-git-repositories https://www.digitalocean.com/community/tutorials/how-to-use-...

serkanh 12y ago
Atlassian Stash is also a good solution. I believe it is $10/annual for the first 10 users. https://www.atlassian.com/software/stash https://www.atlassian.com/software/stash
tthayer 12y ago
Their omnibus installer is ridiculously easy to use if you don't mind seeing Chef messages scroll past for 20 minutes. I had it tied into Active Directory and everything in less than an hour after downloading it. Compare that to using apache-svn and those goddamn LDAP sync scripts and apache conf folder definitions and it feels like you're using some shit from the future.

foolinaround 12y ago

If they can integrate gitolite into this and make a breeze to set up, then that would be awesome. If they use custom permissions ( directly tied to IAM ), then it would be another closed solution like github.

justinsb 12y ago

I've thought about building GitHub / BitBucket tools (issue management, pull requests etc) that work with data stored in git repositories, as opposed to one big tool that owns the git repository and manages the data in a locked-up SQL database. We already have this for code browsers (e.g. cgit), but that alone can't compete with the "full suite" that others offer. It would have several advantages for the user: 1) You could pick your issue management separately from your code browser 2) You don't have to worry that the best tooling (Github) has some of the least reliable storage. 3) If it stored the data in git, it would work offline! The big downside is that having to "assemble your own" is more complicated for the user, and that it isn't Github/BitBucket's business model...

stephenr 12y ago
I like the idea of some good, open source tools to do the different bits, preferably focused on flexibility and ease of data access (as you mentioned) to me this would mean an issue tracker, and some kind of merge request/review tool. these could be paired with the likes of Gollum, providing a wiki. One thing though, I would really love to see them support more than just git. Mercurial shouldn't be hard to support, as it has largely similar concepts, and even though SVN is not as popular as it was at its peak, its still widely used and a very good choice for some workflows/use cases. In the same vane, I would love to see a decent stand-alone code browser that supports different VCS.
- justinsb 12y ago
  Thanks - great input (and the Gollum suggestion is a good starting point!) On multiple VCS: On the one hand, git is so dominant (for better or worse) that I don't know if the complexity is worth it. On the other, perhaps git is so dominant because github is so dominant. The answer in open source land is probably to set up a reasonable abstraction layer and let interested people provide their own implementations of the VCS-interface.
  - stephenr 12y ago
    I think I should also clarify a little.. My key focus for this would be on a more unix-philosophy approach (separate tools that do one thing well). I like the idea of using a VCS repo (again, ideally any of those supported not just git) to store related information such as issues/tickets, etc however I don't know what the practical constraints of that would be, my first thought is that issues/tickets stored primarily in a (version controlled) filesystem (and then presumably indexed on a server for web views, searching etc) would not be as intuitive as say a wiki like gollum. If the issues/tickets tool used a flexible SQL model (e.g. choice of mysql, postgresql and sqlite for instance) I would be happy with that, as the data is still quite open (to the system, owner, not as much to the individual developers admittedly)

lentil_soup 12y ago

Off topic, but the sign up process is very bad. Why do they need full name, email, company name, role, address, phone number, catcha, etc to just sign up for more information?

lentil_soup 12y ago
and while we're at it, why the need for a "State" field that's only relevant to one country?
- ceejayoz 12y ago
  Did you click the state field? There are states for Australia, Canada, China, India, and others listed.
andsmi2 12y ago
It actually isn't out till 2015.... so this is just asking if you are interested. Perhaps it's an MVP -- with a signup form....and if there is enough interest they will pursue it...

vladimirralev 12y ago

Strange use of "git-based" wording. Do I understand correctly, they implemented their own git command line binaries that you install to replace the original git binaries? Possibly a fork of git client and server?

justinsb 12y ago
Probably based on JGit/Gerrit. It is much easier to build "cloud" git (i.e. reliable, redundant git) that way than it is to make your disk reliable. Github went the reliable-disk route with DRBD, AFAIK, which is why Github is always going down ;-) I wrote a blog post about this: http://blog.justinsb.com/blog/2013/12/14/cloudata-day-8/ http://blog.justinsb.com/blog/2013/12/14/cloudata-day-8/
- btown 12y ago
  Intriguing post! For those who hadn't heard about DRBD, it's http://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device http://en.wikipedia.org/wiki/Distributed_Replicated_Block_De... - it makes failover incredibly simple to integrate by pushing a lot of distributed-systems responsibility down to the filesystem level, but in doing so it forces you to have a single-master setup, and you can't take advantage of parts of your domain model that are well-suited to eventual consistency. Thinking about it, now that Riak supports strong (quorum-based) consistency on a per-bucket basis, Riak-backed scalable Git hosting would probably be relatively easy to implement. Looks like you were building your own system - did you ever release the code?
  - justinsb 12y ago
    Ooops... looks like I didn't link to the code in that post. The whole series was open source; here's the git code itself: https://github.com/justinsb/cloudata/tree/master/cloudata-git https://github.com/justinsb/cloudata/tree/master/cloudata-gi... The git object data itself goes into a blob-store (like S3); it can be stored without strong consistency. It turns out you only need to keep track of a very small amount of metadata consistently (the refs). Riak, etcd, DynamoDB or the Google Cloud DataStore would all be good choices, I think. I was working on an open-source implementation of Raft as part of this (called barge), but it isn't as reliable as the alternatives above - yet!

riveralabs 12y ago

I'm more interested in their pricing structure. Knowing Amazon, will we see something in the line of $0.25 per repo plus bandwidth.

TillE 12y ago
A nominal fee on top of their usual storage and bandwidth rates sounds about right. But I'm not sure if this gives you much compared to just pointing Mercurial/largefiles (or git-annex) at S3.
aragot 12y ago
Knowing Amazon, will we see something like $0.0013 per thousand non-ASCII character, "because this is what it costs them after compression".

thewopr 12y ago

Here's why I'm excited about this: Data Versioning Github is awesome for science. But my code and workflows often are just as reliant on the data I have as the code I've written. With Github, I always have to treat my data separately. A different workflow, a different storage location, different (and often manual) versioning. Now, I can start integrating my data into the same workflow. No size limits. Just drop it in and version it like everything else. 1GB? No problem. 500GB? Just pay the money. This is especially awesome because, as a scientist/dev, I do not want to stand up my own infrastructure and servers/vms. I don't want to do linux updates. I don't want to worry about things going down. It just needs to be there when I need it. When I don't need it anymore, I'll take it down. Done and done.

blaedj 12y ago
I've run into problems with large-ish files in git repos, binaries accidentally committed etc. I'm genuinely curious, is there a good way to use git for the size repositories that you mention?
- bkeroack 12y ago
  Git is great for smaller binaries. Ideal in fact, given that it stores differences between revisions as binary deltas. For large >1GB files, I believe the diffing algorithm is the limiting factor (I would be interested in getting confirmation of that, though). For those files something like git-annex is useful (http://git-annex.branchable.com/ http://git-annex.branchable.com/) I've used git to push around a lot of binary application packages and it's very nice. Previously I was copying around 250-300MB of binaries for every deployment--after switching to a git workflow (via Elita) the binary changesets were typically around 12MB or so.
  - blaedj 12y ago
    Hmm, I'll have to take a look at git-annex, I've run across it but never investigated. Thanks!
- notduncansmith 12y ago

oblio 12y ago

I'm shocked that at this point AWS is not offering a build service. Or are they?

deleted 12y ago
[deleted]
deleted 12y ago
[deleted]
kgrin 12y ago
AWS CodePipeline and AWS CodeDeploy (both announced simultaneously I think)
kuhhk 12y ago
They just announced Amazon CodePipeline and Amazon CodeDeploy. http://aws.amazon.com/codepipeline/ http://aws.amazon.com/codepipeline/ http://aws.amazon.com/codedeploy/ http://aws.amazon.com/codedeploy/

valarauca1 12y ago

>Worry about scaling your own infrastructure Does git have scaling issues? I know git manages the 60mil+ SLOC of the Linux Kernel, that's what it was designed for. I don't see how even medium/large size enterprise will have difficulty. Or am I missing something?

maccard 12y ago
Doesn't handle large binary files
- deleted 12y ago
  [deleted]
- MichaelGG 12y ago
  As I understand, this is precisely why companies like Plastic SCM exist: to service game dev and other projects that need to manage large assets.
  - maccard 12y ago
    Yep. We use Perforce in work, and from my experience, most other game dev's do too.
cdr 12y ago
Very few organizations use git the way the linux kernel project does. Most organizations use github, gitlab, etc, which definitely have had scaling issues.
- tedchs 12y ago
  You might be surprised to learn the Linux Kernel is on Github: https://github.com/torvalds/linux https://github.com/torvalds/linux
  - stephenr 12y ago
    I don't think thats their sole central hosted copy of the repo though. I believe they use git.kernel.org

gexla 12y ago

Bezos to kids being born 3 years from now. "You kids are just features. Keep n eye on our announcements page." It's interesting how they seem to be adding all the stuff in the workflow that we are okay with just being good enough and don't benefit from bells and whistles. Github is very cool for managing an open source project, but for my own stuff just about anything which makes managing my repos relatively simple and saves me a bit of time is fine. I don't need much for an interface or features. Just being able to push a button and having a hosted repo ready for other users and with instructions for less technical contributors is awesome. If I'm already on Github, I'm probably staying there. Competition is good though. Competition will push Github to continue doing more to differentiate themselves as being something much greater than simply hosting Git repo's.

whichdan 12y ago
Does GitHub need to do anything beyond hosting git repos, though?

forrestthewoods 12y ago

Interesting. I recently started mixing GitHub and S3 for blog content and it's a pain in the ass. Asking the internet about pushing GitHub content to S3 results in just a bunch of mediocre scripts. I like it. I also look forward to Perforce's response. For versioned binary files Perforce is still the best. Competition, hooray!

EpicDavi 12y ago

My uncle is a DevOps person and said all of the new tools that Amazon is rolling out would put him out of his job if they caught on.

jakejake 12y ago
If I were a DevOps person I would learn and master all of the AWS tools (as well as competitor services). It could be a great opportunity rather than a threat.
hga 12y ago
But that would also a 100% bet on Amazon. There will also be a number of companies where a "small" DevOps teams leverages a large number of cheaper systems (owned, leased or whatever) for lower cost.

pearjuice 12y ago

I still have trouble trusting a company which I primarily know as "that all-round webshop" to get involved withy my tech stack. It feels as-if McDonalds would over night offer server infrastructure and get away with it.

mrjatx 12y ago
McDonalds doesn't make money from their website, which is the problem in your comment. If we were comparing McDonalds we would be comparing their fast food infrastructure. If you were an up and coming fast food burger joint that was going from 1 shop to 2 shops, and possibly more shops, you sure as hell would want to ask McDonalds for it's strategic advice and further advice on how to run your burger shop as well as possible. They've been there, they've done it, not that you need to follow their guidelines, but you definitely want their knowledge in the back of your pocket. They've got about 70 years of experience. Same with Amazon, who has to keep Amazon.com online or they lose massive amounts of money.
- fletchowns 12y ago
  Does Amazon actually use AWS to host Amazon.com?
  - mrjatx 12y ago
    Geez, oh yes. They built their platform and decided to start reselling it. That's why Netflix, Pintrest, Airbnb, Expedia, Foursquare, NASA, the MLB all use AWS. They converted 100% in 2011 according to Jon Jenkins https://www.youtube.com/watch?v=dxk8b9rSKOo#t=449 https://www.youtube.com/watch?v=dxk8b9rSKOo#t=449
  - discodave 12y ago
    Most, but not all of Amazon.com runs on AWS.

sk5t 12y ago

A good start for AWS, but I doubt they will ever quite fill it out to Beanstalk's level for online browsing, commit reports, etc.

elchief 12y ago

Not to be a negative Nelly or anything, but with the speed at which Amazon is releasing this stuff, they are going to have to be a Google eventually and pull the rug out from under our feet, discontinuing several products.

hga 12y ago
Maybe they'll distinguish themselves by not doing any or very much of that at all. Do you have any candidates for service termination?
debaserab2 12y ago
These aren't being given out free, they are charged per usage - there is obvious incentives to keep the products alive. Correct me if I'm wrong, but most failed Google products had no direct revenue stream other than advertisements.
ceejayoz 12y ago
AWS SimpleDB is an example of a product that Google would've likely killed years ago but Amazon maintains. It hasn't gotten any significant update since 2010 (http://aws.amazon.com/blogs/aws/amazon-simpledb-consistency-enhancements/ http://aws.amazon.com/blogs/aws/amazon-simpledb-consistency-...) and, judging from the fact that it only gets a couple posts a month in its AWS Forums, it's very sparsely used.

cheshire137 12y ago

Uggh no way. The AWS Web Console is a beastly enough UI to get around, I have no interest in migrating away from the very comfortable, friendly Github interface.

msoad 12y ago

The best thing about GitHub is it's UI. It will take a long time to polish a UI like GitHub's