10 ms·

GIT is the new FTP

212 points by bitsweet 14y ago

iambibhas 14y ago
I feel, Your exact method might be useful for Test servers. I won't want all my git pushes reflect on the live server. For deployment, I think it's better if one maintains the post-commit-hook to a certain branch, named, say, "Deploy". Any push to Deploy branch would make it deploy maybe?
- tikhonj 14y ago
  That sounds exactly like a system I set up with Jenkins a while back. It would deploy to a test server and run the tests from any branch. If you pushed to the deploy branch, it would deploy to the real server assuming all the tests passed. The actual deploying and testing was all done with Fabric, but the idea was the same. I think that's a pretty good case for using a CI system like Jenkins, especially because it's really easy to set up. It didn't support Git by default, but there was a plugin that was both easy to install and fairly comprehensive.
bnegreve 14y ago
I would rather say ssh (scp) is the new ftp.
- tspiteri 14y ago
  And it comes with a compatibility mode (sftp).
  - olalonde 14y ago
    Not really: > SFTP is not FTP run over SSH, but rather a new protocol designed from the ground up by the IETF SECSH working group. http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol
    - tspiteri 14y ago
      I know it's not ftp over ssh. I meant what I said in the context of the original comment; if ssh is the ftp replacement, then there is an application which transfers files over ssh with commands similar to those of ftp, which is sort of a compatibility mode to help ftp users migrate to ssh. (Although it is a new protocol from the ground up, it is usually used as a subsystem of ssh, thus using the same passwords and authentications and stuff.)
      - jedbrown 14y ago
        Bah, I can't think of a case where the user interface of ftp is so precious it can't be replaced with rsync or sshfs.
  - danbee 14y ago
    You're thinking of FTPS.
- polyfractal 14y ago
  I just found out recently that SCP is horribly slow when transferring lots of small files. I found a blog post showing how to send compressed tarballs over SSH and it worked really well: ssh username@from_server "tar czf - directory_to_get" | tar xzvf - -C path_to_save
  - regularfry 14y ago
    The trick is knowing when that's going to be faster than rsync.
  - zemo 14y ago
    ...or use rsync? rsync -avz somehost:/path/to/src /path/to/dest
    - jackalope 14y ago
      For a more accurate sync, you can also delete obsolete files on the destination: rsync -avz --delete src dest This assumes there are no files you want to keep on the destination, of course. You can exclude them or reorganize to keep them safe.
languagehacker 14y ago
This tutorial is missing a lot of steps. It therefore expects the user to know how to use Git in order to consistently commit and push their changes under such a scheme. If the reader is a self-sufficient Git user (i.e. doesn't need a full step-by-step walkthrough), they wouldn't need this tutorial. Anyhow, this scheme sort of old hat for Git users. Most of us require a build to deploy code, which generally results in SCM-polling continuous integration solutions like Jenkins to fully automate the preparations required to run new code. This blog post sort of suggests a WordPress crowd's version of continuous integration. Go ahead and use it for a blog theme, or a set of assets. If you're doing this for a web application, you're probably doing something wrong.
- webreac 14y ago
  Hello, I am learning git. Even if this tutorial does not comply to the rules of the art or omit basic git knowledge, it fits exactly my actual learning level. I have read the most basic tutorials and this kind of "almost second" level tutorial is perfect to have a better understanding of how pieces fit together. Thank you Kerrick Long
avstraliitski 14y ago
Yeah yeah. Greatest thing since sliced bread. What if we go to http://yoursite/.git http://yoursite/.git or http://yoursite/.svn http://yoursite/.svn ? Often we get free passwords and other information.
drewmclellan 14y ago
The very first sentence is why Git is not the new FTP: > First, create a directory on your server and initialize an empty git repository. The number of people who deploy sites via FTP who also have SSH access and the ability to install Git on their hosting is small. Most people deploying to the web do not control their own servers.
- Sunlis 14y ago
  I almost got kicked off of my old shared host for installing Java on my machine after I faxed them a copy of my driver's license to get SSH access. Got myself a shiny new VPS a few months ago and haven't looked back. I can finally do things I could only dream about with the shared host.
lixef 14y ago
saying that git is the new ftp is to my mind not correct. I read that git doesnt handle big files very well the coder from the backup tool "bup" has written. That was why he invented bup. Correct me if I am wrong, I would really love to replace ftp with git.
- reidrac 14y ago
  You're right. git may require lots of RAM for big changesets (involving big files), and definitely more RAM than SFTP (oh, please never use FTP). But the post is about deployments and not about transferring files.
rubendevries 14y ago
afaik serving websites from your home dir is a bit odd, since the user running nginx/apache shouldn't have access to it, something like /var/www is way more common. and asuming you don't just use git for deployment but also for it's core purpose version control you should setup the origin to the actual origin and name the remote where you push to accordingly (production, dev, test, staging) and you shouldn't set it as the default push, to avoid excedently deploying when you just wanted to checkin the code to the origin repo!
schiptsov 14y ago
Blue - it is new red!
Osiris 14y ago
I built a system to deploy to our test servers that uses a git repository on the test server, clones the repo into a named folder, and checks out a specific branch. QA is able to use a webpage to select a branch, which is automatically checked out in its own folder and a subdomain or cookie is used to do isolated testing of that branch. So for us, git is an important part of our deployment process, though there's more that goes on especially for production deployments.
reidrac 14y ago
I think posts like this tells you only part of the story. Ops it is not just pushing code live. I'd be great they explain how to know which code is live (ie. tag "releases"), how to revert to a previous version (ie. something went wrong), potentially something needs to be restarted, etc.
cduser 14y ago
For non-technical users, dropbox (self hosted) is a good replacement for FTP, IMO
- danbee 14y ago
  How do you self host dropbox?
  - cduser 14y ago
    http://www.readwriteweb.com/cloud/2011/05/4-ways-to-build-your-own-dropbox.php http://www.readwriteweb.com/cloud/2011/05/4-ways-to-build-yo... http://lifehacker.com/5821145/how-to-set-up-a-file+syncing-dropbox-clone-you-control http://lifehacker.com/5821145/how-to-set-up-a-file+syncing-d...
nviennot 14y ago
Once, I had to maintain a website on godaddy, where I only had ftp access. I hated it. We would have interns working on it, and giving them the ftp password sucked. Having git as a proxy, hooked to sync master on ~/www and staging on ~/www/staging was awesome. If you want it: https://github.com/ezyang/git-ftp https://github.com/ezyang/git-ftp
telent 14y ago
I'd question the assumption that ftp was ever the old ftp, if by "ftp" we mean "deployment channel to production services". As soon as your web site is complicated enough to need a database, you have database migrations or their moral equivalent to worry about as part of the upgrade process, an simply dumping files in a directory somewhere doesn't cut it
sry_not4sale 14y ago
The first thing that came to mind after reading this post's title was insecurity (e.g. .git directories accessible in the wwwroot) Isn't that what FTP is most known for - being insecure?
- samuellb 14y ago
  I agree the blog post should warn about it - but it can be perfectly secure if you do it correctly. I usually put all public files inside a directory inside the repo, e.g. public_html, and/or block access to dot-files altogether in the server configuration. E.g. in Apache 2: <Files ~ "^\."> Order allow,deny Deny from all </Files>
EnderMB 14y ago
Using post-commit hooks to handle your deployment for you can have undesired effects if your scripts aren't perfect. I would recommend looking into continuous integration over this method. I've been using TeamCity for just over a year and it is fantastic for a staging environment.
pfanner 14y ago
can anyone recommend a tutorial for diving into git?
- dt7 14y ago
  The book on the Git website (Pro Git) is an easy read (at least to begin with): http://www.git-scm.com/documentation http://www.git-scm.com/documentation
- arocks 14y ago
  Can't think of a better one than Git Magic[1]. It is well written and covers really useful scenarios. [1]: http://www-cs-students.stanford.edu/~blynn/gitmagic/ http://www-cs-students.stanford.edu/~blynn/gitmagic/
- graup 14y ago
  A very basic tutorial can be found here: http://try.github.com/ http://try.github.com/
- hisyam 14y ago
  Git Immersion — http://gitimmersion.com/index.html http://gitimmersion.com/index.html
earnubs 14y ago
FWIW if you use Bash shell you can also use history expansion to shorten the mkdir && cd line further (the line already includes tilde expansion): mkdir ~/www/example.com && cd !$ !$ is the last argument of the preceding command
- obtu 14y ago
  You mean "$_". !$ refers to the previous command in the history.
  - deleted 14y ago
    [deleted]
  - earnubs 14y ago
    Fairly sure I meant !$ :) http://www.gnu.org/software/bash/manual/bashref.html#Word-Designators http://www.gnu.org/software/bash/manual/bashref.html#Word-De...
    - obtu 14y ago
      Seriously, try it. History expansion works on input lines separated by the user asking the shell to run them, it can't cross reference command arguments on the same command line.
      - earnubs 14y ago
        You are correct, so have an upvote. && in OS X seems to put the left side into history and works as I described, but it didn't work when I tried it in Ubuntu 10. $_ works in both.
      - earnubs 14y ago
        I have tried both, the difference is only that !$ echos the expanded command, while $_ simply runs the expanded command; both create a directory and cd into it. (I'm running OS X, FWIW)
  - seclorum 14y ago
loup-vaillant 14y ago
Err, what about rsync? My blog is versioned with git, but compiled with USSM¹. I don't want to version the generated html code, so I deploy it with a one line shell script that just calls rsync through ssh. Simple, and fast. So in my case, rsync looks better than git at deployment. I can't imagine a scenario where the reverse is true. Does someone have a counterexample? [1] My home-made Jekyll clone: http://loup-vaillant.fr/projects/ussm/ http://loup-vaillant.fr/projects/ussm/
- ZoFreX 14y ago
  An alternative, if you do want to use VCS for deployment, is rather than versioning the compiled HTML, create a hook that compiles it. My website is stored in mercurial currently, and when I push to the server it runs bundle and compiles the site.
Xion 14y ago
Unfortunately, this doesn't take into account the fact that stuff you keep in your development repository is almost never identical to stuff you want on your deployment server. There are files you keep in the former but not necessarily want/need in the latter (unit tests, documentation, etc.). Similarly, there are stuff to be put on deployment server which don't want to clutter your dev repository, like compiled CSS/JS assets. As other have said, deployment should really be done by continuous integration server, and preferably using a protocol that's designed for transmitting files, not version control.
j_col 14y ago
I thought SCP was the new FTP. Or rsync.
- veyron 14y ago
  git is closer to rsync than to ftp. scp and ftp are file transfer protocols, while git and rsync are higher-level systems that can use other underlying file transfer protocols
dinkumthinkum 14y ago
No it's not. Let's revisit this in 5 years. We'll still be using FTP. Thanks though. :)
rickmb 14y ago
Every time someone comes up it with a "cool" way to deploy via Git, it turns out their definition of "deployment" is nothing more than dumping something on a server. It completely ignores every aspect of deployment other than putting the files somewhere else, which is 5% of the work and 1% of the complexity of deployment. That's not even a sufficient professional scenario for anything but the simplest of non-essential website. Hell, this won't even do for the simplest of WordPress sites. BTW, who on earth still uses FTP?
- wilhil 14y ago
  I hope you just mean for source control... FTP is used all the time!
  - astrodust 14y ago
    Like telnet, this protocol should be dead. It's an enormous liability.
- masklinn 14y ago
  > BTW, who on earth still uses FTP? Does sftp[0] count? [0] http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13 http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
  - archivator 14y ago
    The commonalities between SFTP and FTP end with the three letters in the acronyms. SFTP, unlike FTP with its passive mode and whatnot, is actually a decent protocol.
    - masklinn 14y ago
      rickmb did not come even remotely close to specifying on which grounds he was asking his question. He might be a proponent of "if it's not automated [via capistrano/fabric/etc...] it's not deployed" for all I know.
- 14y ago
nikic 14y ago
Just a bit feedback on the blog: The background image was very distracting for me. My eyes were always wandering off from the text. Not sure if that's just me or if others had that problem too.
- alan_cx 14y ago
  Me too. Had similar problem with a site called TVCatchup. They stream free TV in the UK. Their player sits in the middle of the page and that is surrounded by an advertising border. Nothing else on the page of consequence. What I found was that I found it very hard to focus on the player in the middle. Often I would get a headache watching. Told the admins, who didn't want to know. Basically I got an angry "Foxtrot Oscar". Ended up using an ad blocker until they found a way to defeat that. Now I don't use the site any more.
  - justincormack 14y ago
    You can full screen the tv on tvcatchup...
  - JonWood 14y ago
    Completely off topic, but TVCatchup provide an XBMC plugin which completely avoids the advertising they add. No idea why, since it presumably destroys their business model, but it's great.
- Kerrick 14y ago
  Coderwall forces you to upload an image, then it "hipsterizes" it (their words, not mine). From what I can tell, this involves significantly reducing the size of the image, increasing the saturation and local contrast, and then stretching and blurring it. As a photographer, this is quite annoying, because it's hard to predict what something will look like. I'll try to find a photograph of mine that'll stand up to being "hipsterized" better.
  - tamasnet 14y ago
    How about a single pixel of your favourite colour? It might be a bit bland but I'd like to see them screw that one up.
lsiebert 14y ago
When git software is easy for ordinary people to use and install on windows, easy enough that people are using it to share porn and pirated material, then maybe it will be like ftp. As it is to run git on windows you, best case, have to install a lightweight nix OS. Cygwin mingw,whatever. I think we techies and power users don't realize how messed up that is if you want git to be more than a program for programmers. Ftp was a protocol, you could make a native client for everything. Git is a great tool, but not yet comparable to ftp.
Tichy 14y ago
Wouldn't this approach also put the .git directory below ~/www, making it accessible to the world? That might not be such a good idea.
- Xymak1y 14y ago
  Hidden directories are usully not accessible from outside I think.
  - gurkendoktor 14y ago
    I had to change my Apache config on OS X 10.6 exactly because I was leaking .svn directories. Unbelievable.
- rlpb 14y ago
  You can have git separate the .git directory and the working directory by setting GIT_WORK_TREE and GIT_DIR in the post-receive hook. I usually put the git directories in /var/local/git, and I have some scripts to automate everything, but essentially I use this mechanism for certain projects (and for testing).
- mappu 14y ago
  Surely the entire project doesn't consist of world-readable files? There's likely to be a htdocs or wwwroot subdirectory in there alongside .git that apache/nginx point to. Then, you have an extra level of breathing room for other files for the site (config files, session data, user uploads, ...)
ioquatix 14y ago
One problem with this technique is that over time you will end up with more and more data on the production server, even stuff that has since been deleted. Even for a small website over the course of a few years can end up being GBs of wasted space which on production VPSs can be quite expensive.
- Someone 14y ago
  Also, if your production server is breached, attackers can mine that data, for example for that accidentally checked-in password, or to search for security fixes that other sites might not have applied.
pitzips 14y ago
Here is the same principle, but better documented: http://www.saintsjd.com/2011/03/automated-deployment-of-wordpress-using-git/ http://www.saintsjd.com/2011/03/automated-deployment-of-word...
ivanhoe 14y ago
text looks very much like a copycat of this tutorial, minus all the explanations: http://toroid.org/ams/git-website-howto http://toroid.org/ams/git-website-howto
- obtu 14y ago
  No, the toroid one is much better because it changes the worktree to export the data. That way the git directory won't be exposed to the web server.
olliesaunders 14y ago
If you do this make sure you set the permissions such that it is not possible to make changes to the files on the server independently of the remote otherwise next time you push you’ll end up with conflicts and, if you’re using an interpreted language, the version control markup that indicates where the conflicts are in files will break your code and bring down your website.
- VBprogrammer 14y ago
  Best way to do this is to init the remote repository using: git init --bare
  - uxp 14y ago
    Bare repositories cannot have a working tree. fatal: core.bare and core.worktree do not make sense
ww520 14y ago
While people are not too hot on the idea of pushing the git codeline to the production server, the same idea is actually pretty good for kicking off a build or running test suits after a git push.
xxiao 14y ago
ftp can overwrite a file, git won't, the old one is still in the history, and it will take space esp when the pushed files are in large size(binary files,etc).
Tyr42 14y ago
This is kinda what Heroku does, right?
veyron 14y ago
As used, I think the more appropriate comparison is GIT is the new RSYNC
Kerrick 14y ago
Wow, my coderwall post got posted to HN? That's pretty cool! Interesting title choice, though... FTP will still be useful for a long time. I didn't really write this as a tutorial as much as an easy way for me to look back and find the commands when I have a forgetful moment. That's why it doesn't go into explaining how or why things work, explaining how to set up git, explaining how to set up your web server to serve files from a directory, or other preliminary information.
- loeschg 14y ago
  That's interesting, haha. Would you or somebody be willing to give explanations for the instructions?
jameswyse 14y ago
In my node.js application I use the excellent Up[1] library along with the Up-Hook[2] plugin. This allows me to update my code locally, push to github and have my server automatically download the changes (using girror[3]) and update the code, all with zero downtime. What I like most about it is that it's all nicely contained inside the same code as my application! [1] https://github.com/LearnBoost/up https://github.com/LearnBoost/up [2] https://github.com/LearnBoost/up-hook https://github.com/LearnBoost/up-hook [3] https://github.com/node-migrator-bot/node-girror https://github.com/node-migrator-bot/node-girror
- obtu 14y ago
  I like this girror thing, it looks like it will fit my non-node projects much better than Capistrano's awkward git integration. (For some reason you linked to a fork, but it's no longer current)
  - jameswyse 14y ago
    woops! the correct link is: https://github.com/eladb/node-girror https://github.com/eladb/node-girror
debacle 14y ago
There was a post yesterday about deploying small sites and I mentioned that svn export was the way to go, not thinking that there really is no simple git equivalent. You can do a one-liner with git archive but I don't think it's as easy.
danmofo 14y ago
As someone who uses FTP (it's all I've ever known, at work I just check my work into CVS and I have 0 idea how the build process works), what would be a better way to deploy a website?
- olefoo 14y ago
  It depends on the site, and the stack you are using to manage it, but ideally you would have the following features. 1. All assets are under version control. 2. Committing code of any sort (HTML, CSS, Javascript, Ruby, Python, PHP, ...etc) triggers the unit tests and a failure of the unit tests prevents the commit from completing. #unit tests should take n < 3 seconds 3. Pushing the code to the deploy repo (or branch) triggers the acceptance tests (behavior driven tests that look at the functionality of the site overall, account creation, login, profile_view, checkout) failure of the acceptance tests sends an email to the whole team. Code changes affecting security sensitive modules should be considered failing until they have been manually qualified (e.g. anything touching passwords or money). 4. If the acceptance tests pass and no critical modules were affected by the commit, that commit gets deployed into the production environment. That fact is recorded on the dashboard and emailed to the team. This is difficult to achieve but once reached, it's like being over the step on a boat that hydroplanes; friction is reduced and you get greater velocity for the power expended.
firemanx 14y ago
We use TeamCity and deploy right from our build server. We have a build configuration called "Deploy" that requires a couple parameters - the artifact to deploy, the environment to deploy to, and the build number (so we can deploy past builds, ie roll back). Our application software is built of a series of distributed .war and .jar files, but each represent a complete "service" within our system, and can be upgraded independently. This means we can deploy single artifacts and not cause conflicts within the overall system. When we click the build button, a small Python script is called that rsyncs over ssh the already built artifact (pulled in from the existing TC build) over to the environment, which runs JBoss, and gets auto-deployed. For our developers, our build process amounts to committing their code, getting an email from TeamCity when the build completes, then clicking the Deploy button. The tools take care of the rest, and we end up with a nice trail of logs available in case anything goes wrong, and a clear rollback strategy. We also don't get an ever-growing git repository, or have to maintain a bunch of FTP configuration. Everything is done over ssh via passwordless pki. Best of all, it didn't take long at all to set up (the longest part was probably writing the python deployment script, which has some basic logic to know which environments live at which addresses, and also how to run some post-deploy commands for the 1 non-JBoss deployed .jar based service we have). It's not perfect - we don't have an automated database migration mechanism right now, though in a distributed system like ours I'm not entirely sure one would work at this point (and we've got other plans to deal with that weakness anyway). I'd also like to move to Hudson or Jenkins at some point, as there is a real attraction to the rich ecosystem there (TeamCity is a commercial product).
snowwrestler 14y ago
Reverting a push should be at least as fast and easy as the push itself. This minimizes downtime if you introduce a new bug. (Which I know you, the reader, would never do, but other people might, so this is for them.) Using this workflow, if you break your site, reverting would require either a) fixing the bug and then doing another push or b) looking up and typing in the hash of the last known good commit. Neither of these are as fast and easy as "git push". Our solution is to deploy with tags. Every push to production gets its own tag, and that is what we checkout into production. If the site breaks, you just checkout the previous tag name and you're reverted. This way reverting is exactly as easy and fast as pushing new code.
- mnutt 14y ago
  You could even streamline it a bit, by having a 'previous' branch that you automatically sync with the previous tag on deploy. So if something goes wrong you can always just `git checkout previous`.
jackalope 14y ago
This is clever, but the idea of serving a website directly from a repository makes me very uncomfortable. If you already have git and ssh access on a server, then you probably have rsync access, as well. I simply include a Makefile in my repo with a publish target that runs a command like this: rsync -avz --exclude '*.swp' --delete site/ $(WEBUSER)@$(WEBHOST):site/ This way I'm only putting the necessary files on the server and transferring changes is incredibly fast and secure (ssh is used by default). In rare cases where I need to deploy from a repo on the server, I'll clone it to the remote host and use rsync locally to copy the files. But I still avoid serving directly out of the repo. There's too much sensitive information in version control to even risk exposing it during a misconfiguration.
- cgdangelo 14y ago
  I'm curious what kind of sensitive information you're concerned about exposing. I don't mind deploying with rsync--in fact, I'll do it quite a lot when the remote host lacks git--but I'd usually prefer to use git where possible. I typically deploy with a similar setup that you described, but instead of rsync'ing the files locally I just make a clone of the repo and serve that. Though it is another step to fetch/merge, it stops me from losing any changes that someone did to production without telling me. I only recently used Capistrano to deploy a project and it was very satisfying, so I'm gravitating towards that as my default deployment method.
  - donavanm 14y ago
    > I'm curious what kind of sensitive information you're concerned about exposing +1. Secrets should not be in your VCS repo. I'd guess the parent is talking about user/pass or pub/priv key creds. I really wish "we" had better tools for passing around config, and secrets in particular. Chefs data bags are close, but I still don't want the master knowing my secrets.
trebor 14y ago
I could do automated deployments via FTP (I wrote scripts to upload certain parts of a project during the build process) with a cron job on the server. Just check for changes and do the real "deployment" stuff on the server itself.
rocky1138 14y ago
I almost never FTP stuff except when I have to get a single, unimportant file somewhere or if we're sharing stuff between friends' houses. Springloops has a useful deploy-by-ftp feature. When you commit a change using their Subversion host it automatically pushes it out to the development or production server. It's easy for designers to use since they can update a jpg or two for a client, commit a change, then hit refresh on the live site and it shows up. I know it's not perfect, but for small web shops and non-technical people, it works great.
deleted 14y ago
[deleted]
fcurella 14y ago
I've been following this http://toroid.org/ams/git-website-howto http://toroid.org/ams/git-website-howto for several years now.
emmelaich 14y ago
I've been looking into git (or dvcs generally) for deployment and have realised that deployment is like the inverse (dare I say dual :) of normal version control. vc: metadata (permissions) are irrelevant. dep: metadata is critical vc: version everything by default (explicitly exclude) dep: version by explicit inclusion I've have a doc with numerous links and notes that I should share sometime.
mkopinsky 14y ago
Best move I did for a past client was getting them to deploy using Assembla's FTP tool. There is essentially a post-commit hook in their SVN repo that automatically pushes to the test environment on every commit. Deployments to prod use the same tool, but it is manually triggered rather than on every commit. (Competing for best move, was getting them to have separate test and prod environments.) If you (or an unfortunate client) are using a cheap shared host that only provides you with FTP, this approach at least guarantees that the only code on the site is what exists in your VCS.

212 points by bitsweet 14y ago

I feel, Your exact method might be useful for Test servers. I won't want all my git pushes reflect on the live server. For deployment, I think it's better if one maintains the post-commit-hook to a certain branch, named, say, "Deploy". Any push to Deploy branch would make it deploy maybe?

tikhonj 14y ago
That sounds exactly like a system I set up with Jenkins a while back. It would deploy to a test server and run the tests from any branch. If you pushed to the deploy branch, it would deploy to the real server assuming all the tests passed. The actual deploying and testing was all done with Fabric, but the idea was the same. I think that's a pretty good case for using a CI system like Jenkins, especially because it's really easy to set up. It didn't support Git by default, but there was a plugin that was both easy to install and fairly comprehensive.

bnegreve 14y ago

I would rather say ssh (scp) is the new ftp.

tspiteri 14y ago
And it comes with a compatibility mode (sftp).
- olalonde 14y ago
  Not really: > SFTP is not FTP run over SSH, but rather a new protocol designed from the ground up by the IETF SECSH working group. http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol
  - tspiteri 14y ago
    I know it's not ftp over ssh. I meant what I said in the context of the original comment; if ssh is the ftp replacement, then there is an application which transfers files over ssh with commands similar to those of ftp, which is sort of a compatibility mode to help ftp users migrate to ssh. (Although it is a new protocol from the ground up, it is usually used as a subsystem of ssh, thus using the same passwords and authentications and stuff.)
    - jedbrown 14y ago
      Bah, I can't think of a case where the user interface of ftp is so precious it can't be replaced with rsync or sshfs.
- danbee 14y ago
  You're thinking of FTPS.
polyfractal 14y ago
I just found out recently that SCP is horribly slow when transferring lots of small files. I found a blog post showing how to send compressed tarballs over SSH and it worked really well: ssh username@from_server "tar czf - directory_to_get" | tar xzvf - -C path_to_save
- regularfry 14y ago
  The trick is knowing when that's going to be faster than rsync.
- zemo 14y ago
  ...or use rsync? rsync -avz somehost:/path/to/src /path/to/dest
  - jackalope 14y ago
    For a more accurate sync, you can also delete obsolete files on the destination: rsync -avz --delete src dest This assumes there are no files you want to keep on the destination, of course. You can exclude them or reorganize to keep them safe.

languagehacker 14y ago

This tutorial is missing a lot of steps. It therefore expects the user to know how to use Git in order to consistently commit and push their changes under such a scheme. If the reader is a self-sufficient Git user (i.e. doesn't need a full step-by-step walkthrough), they wouldn't need this tutorial. Anyhow, this scheme sort of old hat for Git users. Most of us require a build to deploy code, which generally results in SCM-polling continuous integration solutions like Jenkins to fully automate the preparations required to run new code. This blog post sort of suggests a WordPress crowd's version of continuous integration. Go ahead and use it for a blog theme, or a set of assets. If you're doing this for a web application, you're probably doing something wrong.

webreac 14y ago
Hello, I am learning git. Even if this tutorial does not comply to the rules of the art or omit basic git knowledge, it fits exactly my actual learning level. I have read the most basic tutorials and this kind of "almost second" level tutorial is perfect to have a better understanding of how pieces fit together. Thank you Kerrick Long

avstraliitski 14y ago

Yeah yeah. Greatest thing since sliced bread. What if we go to http://yoursite/.git http://yoursite/.git or http://yoursite/.svn http://yoursite/.svn ? Often we get free passwords and other information.

drewmclellan 14y ago

The very first sentence is why Git is not the new FTP: > First, create a directory on your server and initialize an empty git repository. The number of people who deploy sites via FTP who also have SSH access and the ability to install Git on their hosting is small. Most people deploying to the web do not control their own servers.

Sunlis 14y ago
I almost got kicked off of my old shared host for installing Java on my machine after I faxed them a copy of my driver's license to get SSH access. Got myself a shiny new VPS a few months ago and haven't looked back. I can finally do things I could only dream about with the shared host.

lixef 14y ago

saying that git is the new ftp is to my mind not correct. I read that git doesnt handle big files very well the coder from the backup tool "bup" has written. That was why he invented bup. Correct me if I am wrong, I would really love to replace ftp with git.

reidrac 14y ago
You're right. git may require lots of RAM for big changesets (involving big files), and definitely more RAM than SFTP (oh, please never use FTP). But the post is about deployments and not about transferring files.

rubendevries 14y ago

afaik serving websites from your home dir is a bit odd, since the user running nginx/apache shouldn't have access to it, something like /var/www is way more common. and asuming you don't just use git for deployment but also for it's core purpose version control you should setup the origin to the actual origin and name the remote where you push to accordingly (production, dev, test, staging) and you shouldn't set it as the default push, to avoid excedently deploying when you just wanted to checkin the code to the origin repo!

schiptsov 14y ago

Blue - it is new red!

Osiris 14y ago

I built a system to deploy to our test servers that uses a git repository on the test server, clones the repo into a named folder, and checks out a specific branch. QA is able to use a webpage to select a branch, which is automatically checked out in its own folder and a subdomain or cookie is used to do isolated testing of that branch. So for us, git is an important part of our deployment process, though there's more that goes on especially for production deployments.

reidrac 14y ago

I think posts like this tells you only part of the story. Ops it is not just pushing code live. I'd be great they explain how to know which code is live (ie. tag "releases"), how to revert to a previous version (ie. something went wrong), potentially something needs to be restarted, etc.

cduser 14y ago

For non-technical users, dropbox (self hosted) is a good replacement for FTP, IMO

danbee 14y ago
How do you self host dropbox?
- cduser 14y ago
  http://www.readwriteweb.com/cloud/2011/05/4-ways-to-build-your-own-dropbox.php http://www.readwriteweb.com/cloud/2011/05/4-ways-to-build-yo... http://lifehacker.com/5821145/how-to-set-up-a-file+syncing-dropbox-clone-you-control http://lifehacker.com/5821145/how-to-set-up-a-file+syncing-d...

nviennot 14y ago

Once, I had to maintain a website on godaddy, where I only had ftp access. I hated it. We would have interns working on it, and giving them the ftp password sucked. Having git as a proxy, hooked to sync master on ~/www and staging on ~/www/staging was awesome. If you want it: https://github.com/ezyang/git-ftp https://github.com/ezyang/git-ftp

telent 14y ago

I'd question the assumption that ftp was ever the old ftp, if by "ftp" we mean "deployment channel to production services". As soon as your web site is complicated enough to need a database, you have database migrations or their moral equivalent to worry about as part of the upgrade process, an simply dumping files in a directory somewhere doesn't cut it

sry_not4sale 14y ago

The first thing that came to mind after reading this post's title was insecurity (e.g. .git directories accessible in the wwwroot) Isn't that what FTP is most known for - being insecure?

samuellb 14y ago
I agree the blog post should warn about it - but it can be perfectly secure if you do it correctly. I usually put all public files inside a directory inside the repo, e.g. public_html, and/or block access to dot-files altogether in the server configuration. E.g. in Apache 2: <Files ~ "^\."> Order allow,deny Deny from all </Files>

EnderMB 14y ago

Using post-commit hooks to handle your deployment for you can have undesired effects if your scripts aren't perfect. I would recommend looking into continuous integration over this method. I've been using TeamCity for just over a year and it is fantastic for a staging environment.

pfanner 14y ago

can anyone recommend a tutorial for diving into git?

dt7 14y ago
The book on the Git website (Pro Git) is an easy read (at least to begin with): http://www.git-scm.com/documentation http://www.git-scm.com/documentation
arocks 14y ago
Can't think of a better one than Git Magic[1]. It is well written and covers really useful scenarios. [1]: http://www-cs-students.stanford.edu/~blynn/gitmagic/ http://www-cs-students.stanford.edu/~blynn/gitmagic/
graup 14y ago
A very basic tutorial can be found here: http://try.github.com/ http://try.github.com/
hisyam 14y ago
Git Immersion — http://gitimmersion.com/index.html http://gitimmersion.com/index.html

earnubs 14y ago

FWIW if you use Bash shell you can also use history expansion to shorten the mkdir && cd line further (the line already includes tilde expansion): mkdir ~/www/example.com && cd !$ !$ is the last argument of the preceding command

obtu 14y ago
You mean "$_". !$ refers to the previous command in the history.
- deleted 14y ago
  [deleted]
- earnubs 14y ago
  Fairly sure I meant !$ :) http://www.gnu.org/software/bash/manual/bashref.html#Word-Designators http://www.gnu.org/software/bash/manual/bashref.html#Word-De...
  - obtu 14y ago
    Seriously, try it. History expansion works on input lines separated by the user asking the shell to run them, it can't cross reference command arguments on the same command line.
    - earnubs 14y ago
      You are correct, so have an upvote. && in OS X seems to put the left side into history and works as I described, but it didn't work when I tried it in Ubuntu 10. $_ works in both.
    - earnubs 14y ago
      I have tried both, the difference is only that !$ echos the expanded command, while $_ simply runs the expanded command; both create a directory and cd into it. (I'm running OS X, FWIW)
- seclorum 14y ago

loup-vaillant 14y ago

Err, what about rsync? My blog is versioned with git, but compiled with USSM¹. I don't want to version the generated html code, so I deploy it with a one line shell script that just calls rsync through ssh. Simple, and fast. So in my case, rsync looks better than git at deployment. I can't imagine a scenario where the reverse is true. Does someone have a counterexample? [1] My home-made Jekyll clone: http://loup-vaillant.fr/projects/ussm/ http://loup-vaillant.fr/projects/ussm/

ZoFreX 14y ago
An alternative, if you do want to use VCS for deployment, is rather than versioning the compiled HTML, create a hook that compiles it. My website is stored in mercurial currently, and when I push to the server it runs bundle and compiles the site.

Xion 14y ago

Unfortunately, this doesn't take into account the fact that stuff you keep in your development repository is almost never identical to stuff you want on your deployment server. There are files you keep in the former but not necessarily want/need in the latter (unit tests, documentation, etc.). Similarly, there are stuff to be put on deployment server which don't want to clutter your dev repository, like compiled CSS/JS assets. As other have said, deployment should really be done by continuous integration server, and preferably using a protocol that's designed for transmitting files, not version control.

j_col 14y ago

I thought SCP was the new FTP. Or rsync.

veyron 14y ago
git is closer to rsync than to ftp. scp and ftp are file transfer protocols, while git and rsync are higher-level systems that can use other underlying file transfer protocols

dinkumthinkum 14y ago

No it's not. Let's revisit this in 5 years. We'll still be using FTP. Thanks though. :)

rickmb 14y ago

Every time someone comes up it with a "cool" way to deploy via Git, it turns out their definition of "deployment" is nothing more than dumping something on a server. It completely ignores every aspect of deployment other than putting the files somewhere else, which is 5% of the work and 1% of the complexity of deployment. That's not even a sufficient professional scenario for anything but the simplest of non-essential website. Hell, this won't even do for the simplest of WordPress sites. BTW, who on earth still uses FTP?

wilhil 14y ago
I hope you just mean for source control... FTP is used all the time!
- astrodust 14y ago
  Like telnet, this protocol should be dead. It's an enormous liability.
masklinn 14y ago
> BTW, who on earth still uses FTP? Does sftp[0] count? [0] http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13 http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
- archivator 14y ago
  The commonalities between SFTP and FTP end with the three letters in the acronyms. SFTP, unlike FTP with its passive mode and whatnot, is actually a decent protocol.
  - masklinn 14y ago
    rickmb did not come even remotely close to specifying on which grounds he was asking his question. He might be a proponent of "if it's not automated [via capistrano/fabric/etc...] it's not deployed" for all I know.
14y ago

nikic 14y ago

Just a bit feedback on the blog: The background image was very distracting for me. My eyes were always wandering off from the text. Not sure if that's just me or if others had that problem too.

alan_cx 14y ago
Me too. Had similar problem with a site called TVCatchup. They stream free TV in the UK. Their player sits in the middle of the page and that is surrounded by an advertising border. Nothing else on the page of consequence. What I found was that I found it very hard to focus on the player in the middle. Often I would get a headache watching. Told the admins, who didn't want to know. Basically I got an angry "Foxtrot Oscar". Ended up using an ad blocker until they found a way to defeat that. Now I don't use the site any more.
- justincormack 14y ago
  You can full screen the tv on tvcatchup...
- JonWood 14y ago
  Completely off topic, but TVCatchup provide an XBMC plugin which completely avoids the advertising they add. No idea why, since it presumably destroys their business model, but it's great.
Kerrick 14y ago
Coderwall forces you to upload an image, then it "hipsterizes" it (their words, not mine). From what I can tell, this involves significantly reducing the size of the image, increasing the saturation and local contrast, and then stretching and blurring it. As a photographer, this is quite annoying, because it's hard to predict what something will look like. I'll try to find a photograph of mine that'll stand up to being "hipsterized" better.
- tamasnet 14y ago
  How about a single pixel of your favourite colour? It might be a bit bland but I'd like to see them screw that one up.

lsiebert 14y ago

When git software is easy for ordinary people to use and install on windows, easy enough that people are using it to share porn and pirated material, then maybe it will be like ftp. As it is to run git on windows you, best case, have to install a lightweight nix OS. Cygwin mingw,whatever. I think we techies and power users don't realize how messed up that is if you want git to be more than a program for programmers. Ftp was a protocol, you could make a native client for everything. Git is a great tool, but not yet comparable to ftp.

Tichy 14y ago

Wouldn't this approach also put the .git directory below ~/www, making it accessible to the world? That might not be such a good idea.

Xymak1y 14y ago
Hidden directories are usully not accessible from outside I think.
- gurkendoktor 14y ago
  I had to change my Apache config on OS X 10.6 exactly because I was leaking .svn directories. Unbelievable.
rlpb 14y ago
You can have git separate the .git directory and the working directory by setting GIT_WORK_TREE and GIT_DIR in the post-receive hook. I usually put the git directories in /var/local/git, and I have some scripts to automate everything, but essentially I use this mechanism for certain projects (and for testing).
mappu 14y ago
Surely the entire project doesn't consist of world-readable files? There's likely to be a htdocs or wwwroot subdirectory in there alongside .git that apache/nginx point to. Then, you have an extra level of breathing room for other files for the site (config files, session data, user uploads, ...)

ioquatix 14y ago

One problem with this technique is that over time you will end up with more and more data on the production server, even stuff that has since been deleted. Even for a small website over the course of a few years can end up being GBs of wasted space which on production VPSs can be quite expensive.

Someone 14y ago
Also, if your production server is breached, attackers can mine that data, for example for that accidentally checked-in password, or to search for security fixes that other sites might not have applied.

pitzips 14y ago

Here is the same principle, but better documented: http://www.saintsjd.com/2011/03/automated-deployment-of-wordpress-using-git/ http://www.saintsjd.com/2011/03/automated-deployment-of-word...

ivanhoe 14y ago

text looks very much like a copycat of this tutorial, minus all the explanations: http://toroid.org/ams/git-website-howto http://toroid.org/ams/git-website-howto

obtu 14y ago
No, the toroid one is much better because it changes the worktree to export the data. That way the git directory won't be exposed to the web server.

olliesaunders 14y ago

If you do this make sure you set the permissions such that it is not possible to make changes to the files on the server independently of the remote otherwise next time you push you’ll end up with conflicts and, if you’re using an interpreted language, the version control markup that indicates where the conflicts are in files will break your code and bring down your website.

VBprogrammer 14y ago
Best way to do this is to init the remote repository using: git init --bare
- uxp 14y ago
  Bare repositories cannot have a working tree. fatal: core.bare and core.worktree do not make sense

ww520 14y ago

While people are not too hot on the idea of pushing the git codeline to the production server, the same idea is actually pretty good for kicking off a build or running test suits after a git push.

xxiao 14y ago

ftp can overwrite a file, git won't, the old one is still in the history, and it will take space esp when the pushed files are in large size(binary files,etc).

Tyr42 14y ago

This is kinda what Heroku does, right?

veyron 14y ago

As used, I think the more appropriate comparison is GIT is the new RSYNC

Kerrick 14y ago

Wow, my coderwall post got posted to HN? That's pretty cool! Interesting title choice, though... FTP will still be useful for a long time. I didn't really write this as a tutorial as much as an easy way for me to look back and find the commands when I have a forgetful moment. That's why it doesn't go into explaining how or why things work, explaining how to set up git, explaining how to set up your web server to serve files from a directory, or other preliminary information.

loeschg 14y ago
That's interesting, haha. Would you or somebody be willing to give explanations for the instructions?

jameswyse 14y ago

In my node.js application I use the excellent Up[1] library along with the Up-Hook[2] plugin. This allows me to update my code locally, push to github and have my server automatically download the changes (using girror[3]) and update the code, all with zero downtime. What I like most about it is that it's all nicely contained inside the same code as my application! [1] https://github.com/LearnBoost/up https://github.com/LearnBoost/up [2] https://github.com/LearnBoost/up-hook https://github.com/LearnBoost/up-hook [3] https://github.com/node-migrator-bot/node-girror https://github.com/node-migrator-bot/node-girror

obtu 14y ago
I like this girror thing, it looks like it will fit my non-node projects much better than Capistrano's awkward git integration. (For some reason you linked to a fork, but it's no longer current)
- jameswyse 14y ago
  woops! the correct link is: https://github.com/eladb/node-girror https://github.com/eladb/node-girror

debacle 14y ago

There was a post yesterday about deploying small sites and I mentioned that svn export was the way to go, not thinking that there really is no simple git equivalent. You can do a one-liner with git archive but I don't think it's as easy.

danmofo 14y ago

As someone who uses FTP (it's all I've ever known, at work I just check my work into CVS and I have 0 idea how the build process works), what would be a better way to deploy a website?

olefoo 14y ago
It depends on the site, and the stack you are using to manage it, but ideally you would have the following features. 1. All assets are under version control. 2. Committing code of any sort (HTML, CSS, Javascript, Ruby, Python, PHP, ...etc) triggers the unit tests and a failure of the unit tests prevents the commit from completing. #unit tests should take n < 3 seconds 3. Pushing the code to the deploy repo (or branch) triggers the acceptance tests (behavior driven tests that look at the functionality of the site overall, account creation, login, profile_view, checkout) failure of the acceptance tests sends an email to the whole team. Code changes affecting security sensitive modules should be considered failing until they have been manually qualified (e.g. anything touching passwords or money). 4. If the acceptance tests pass and no critical modules were affected by the commit, that commit gets deployed into the production environment. That fact is recorded on the dashboard and emailed to the team. This is difficult to achieve but once reached, it's like being over the step on a boat that hydroplanes; friction is reduced and you get greater velocity for the power expended.

firemanx 14y ago

We use TeamCity and deploy right from our build server. We have a build configuration called "Deploy" that requires a couple parameters - the artifact to deploy, the environment to deploy to, and the build number (so we can deploy past builds, ie roll back). Our application software is built of a series of distributed .war and .jar files, but each represent a complete "service" within our system, and can be upgraded independently. This means we can deploy single artifacts and not cause conflicts within the overall system. When we click the build button, a small Python script is called that rsyncs over ssh the already built artifact (pulled in from the existing TC build) over to the environment, which runs JBoss, and gets auto-deployed. For our developers, our build process amounts to committing their code, getting an email from TeamCity when the build completes, then clicking the Deploy button. The tools take care of the rest, and we end up with a nice trail of logs available in case anything goes wrong, and a clear rollback strategy. We also don't get an ever-growing git repository, or have to maintain a bunch of FTP configuration. Everything is done over ssh via passwordless pki. Best of all, it didn't take long at all to set up (the longest part was probably writing the python deployment script, which has some basic logic to know which environments live at which addresses, and also how to run some post-deploy commands for the 1 non-JBoss deployed .jar based service we have). It's not perfect - we don't have an automated database migration mechanism right now, though in a distributed system like ours I'm not entirely sure one would work at this point (and we've got other plans to deal with that weakness anyway). I'd also like to move to Hudson or Jenkins at some point, as there is a real attraction to the rich ecosystem there (TeamCity is a commercial product).

snowwrestler 14y ago

Reverting a push should be at least as fast and easy as the push itself. This minimizes downtime if you introduce a new bug. (Which I know you, the reader, would never do, but other people might, so this is for them.) Using this workflow, if you break your site, reverting would require either a) fixing the bug and then doing another push or b) looking up and typing in the hash of the last known good commit. Neither of these are as fast and easy as "git push". Our solution is to deploy with tags. Every push to production gets its own tag, and that is what we checkout into production. If the site breaks, you just checkout the previous tag name and you're reverted. This way reverting is exactly as easy and fast as pushing new code.

mnutt 14y ago
You could even streamline it a bit, by having a 'previous' branch that you automatically sync with the previous tag on deploy. So if something goes wrong you can always just `git checkout previous`.

jackalope 14y ago

This is clever, but the idea of serving a website directly from a repository makes me very uncomfortable. If you already have git and ssh access on a server, then you probably have rsync access, as well. I simply include a Makefile in my repo with a publish target that runs a command like this: rsync -avz --exclude '*.swp' --delete site/ $(WEBUSER)@$(WEBHOST):site/ This way I'm only putting the necessary files on the server and transferring changes is incredibly fast and secure (ssh is used by default). In rare cases where I need to deploy from a repo on the server, I'll clone it to the remote host and use rsync locally to copy the files. But I still avoid serving directly out of the repo. There's too much sensitive information in version control to even risk exposing it during a misconfiguration.

cgdangelo 14y ago
I'm curious what kind of sensitive information you're concerned about exposing. I don't mind deploying with rsync--in fact, I'll do it quite a lot when the remote host lacks git--but I'd usually prefer to use git where possible. I typically deploy with a similar setup that you described, but instead of rsync'ing the files locally I just make a clone of the repo and serve that. Though it is another step to fetch/merge, it stops me from losing any changes that someone did to production without telling me. I only recently used Capistrano to deploy a project and it was very satisfying, so I'm gravitating towards that as my default deployment method.
- donavanm 14y ago
  > I'm curious what kind of sensitive information you're concerned about exposing +1. Secrets should not be in your VCS repo. I'd guess the parent is talking about user/pass or pub/priv key creds. I really wish "we" had better tools for passing around config, and secrets in particular. Chefs data bags are close, but I still don't want the master knowing my secrets.

trebor 14y ago

I could do automated deployments via FTP (I wrote scripts to upload certain parts of a project during the build process) with a cron job on the server. Just check for changes and do the real "deployment" stuff on the server itself.

rocky1138 14y ago

I almost never FTP stuff except when I have to get a single, unimportant file somewhere or if we're sharing stuff between friends' houses. Springloops has a useful deploy-by-ftp feature. When you commit a change using their Subversion host it automatically pushes it out to the development or production server. It's easy for designers to use since they can update a jpg or two for a client, commit a change, then hit refresh on the live site and it shows up. I know it's not perfect, but for small web shops and non-technical people, it works great.

deleted 14y ago

[deleted]

fcurella 14y ago

I've been following this http://toroid.org/ams/git-website-howto http://toroid.org/ams/git-website-howto for several years now.

emmelaich 14y ago

I've been looking into git (or dvcs generally) for deployment and have realised that deployment is like the inverse (dare I say dual :) of normal version control. vc: metadata (permissions) are irrelevant. dep: metadata is critical vc: version everything by default (explicitly exclude) dep: version by explicit inclusion I've have a doc with numerous links and notes that I should share sometime.

mkopinsky 14y ago

Best move I did for a past client was getting them to deploy using Assembla's FTP tool. There is essentially a post-commit hook in their SVN repo that automatically pushes to the test environment on every commit. Deployments to prod use the same tool, but it is manually triggered rather than on every commit. (Competing for best move, was getting them to have separate test and prod environments.) If you (or an unfortunate client) are using a cheap shared host that only provides you with FTP, this approach at least guarantees that the only code on the site is what exists in your VCS.