6 ms·

ASP.NET Deployment Needs To Be Fixed

48 points by robconery 16y ago

kprobst 16y ago
I've used NAnt to build/deploy happily for years. I suppose it depends on each project or application's particular context. I will say this: If there's some kind of requirement where Dev1 will overwrite Dev2's changes on the target server, especially if we're talking about production, then you're probably doing it wrong. We never deploy from a developer's box or ad-hoc copying of files. Anything that needs to be deployed needs to be in source control, and needs to be deployed from the build box. I don't care if someone forgot an ASCX template and it takes a half hour to redo the whole thing.
aaronrc 16y ago
Didn't really understand what's wrong with msdeploy? Deploying can't get much easier than clicking a single button in VS.Net... Rolling back IS an issue but it can be mitigated by versioning one's source code and testing the site locally before deploying.
- latch 16y ago
  I'm envious that your experience lets you believe that local testing can mitigate the need for rolling back a production deployment.
- cubicle67 16y ago
  I spent a few years working for BigCo, and the one thing I learned there was an appreciation for rollback scripts. every single change had to be accompanied by a full and tested rollback script, and given often the deployment was messy (lots of integration/hardware mixes/etc) writing these was difficult and time consuming. ... but, they saved my arse on more than on occasion. moral of the story: always have a backout plan. Stuff goes wrong. not often, and if you're good, then rarely, but when it does you need a way out.
- davidradcliffe 16y ago
  If you don't understand what's wrong, you've never seen/used/experienced the beauty that is capistrano deployment.
  - aaronrc 16y ago
    Nope I haven't. Guess I'll just keep pressing my one step VS.Net publish button in blissful ignorance :).
    - robconery 16y ago
      If you're one person working alone - then yes a "button push" might work for you. For teams of many people that like to push "panic patches" - that doesn't work out so well.
- notJim 16y ago
  Back in the day, a company I worked for had a very “manual” deploy process—basically, whoever deployed it (me) had to be sure to follow a 7 step process to the letter, or else Bad Things Would Happen. One time, I accidentally skipped a step or two, and Bad Things were happening, and to make it worse, I couldn't figure out what they were. There was, however, a working version of the code sitting in the directory alongside this horrifically broken monster. I knew the configs were correct because they had been working not 5 minutes ago, and I hadn't changed them. So I updated a symlink (this was on Linux/Apache, but it would apply on IIS/Windows too), and everything was happy again, though using a slightly older codebase.
- trezor 16y ago
  I'm sure MSDeploy is nice when it works. Serious emphasis on "when it works". I spent an entire evening trying to get MSDeploy to push something from my local machine to my VPS. Conditions: The VPS is remote and not a member of my home-domain, but the path between them is 100% whitelisted in all intermediate firewalls. How hard can that be? Evidently very hard. Despite spending the entire evening trying to get them to communicate and push files, it simply didn't happen. In the end I ended up writing my own deployment-system based on source-control, Samba and rsync in Bash. It was easier, it worked and I know why it works. If that is easier to get working than a "one click" solution, the authors of said solution better get a bigger button. I can't seem to click this one.
  - jimfl 16y ago
    Oh, thank goodness. Just yesterday I tried to deploy 6 different apps from the IIS app gallery, with 0 successful deploys. I thought maybe I was starting to go retarded. Apparently there are some bugs which prevent parameterized deploys from working properly. Also, the documentation for Microsoft.Web.Deployment is very poor.
  - mountaineer 16y ago
    I've had similar frustrations with msdeploy. Are you initializing the rsync/Bash from your local machine? What are you running on the server to receivie it?
    - trezor 16y ago
      It's all done by a bash-sc ript on an intermediate linux-shell which has access to source-control (perforce) and the server (via smb). I can post the script later if you are interested.
MartinCron 16y ago
I am doing some very straightforward asp.net deployment using git, TeamCity, nunit, and powershell. If the gauntlet of (unit-integration-acceptance) tests (in the master branch, naturally) all pass, the code is deployed to a staging server. Then there are some tests which run against staging. If those tests all pass, then the artifacts for the new version of the site are copied into a new directory (side by side) on the server. finally, a powershell script tells IIS to serve from the new directory. End result? Deployments are zero-downtime non-events tha happen multiple times daily. Rollbacks are trivial (and rare). Any not-yet-ready for prime time code can be checked into any other git branches. Database changes? I have a tiny bit of code, called from Application_OnStart that checks to see if it needs to do any CREATE TABLE or ALTER TABLE statements. Sure, I had to create all of this stuff myself, but it's all crazy simple, reliable, and does just what the project needs. Maybe someone could make some product to handle all of this, but the flexibility of linking together the best tools for the job wins for now. Also, it's just easy.
- sajidnizami 16y ago
  I am with you. I've just written powershell scripts that work in the following manner. 1. Make a dated folder and copy the new build to the folder on all servers. 2. Point IIS to the folder on all servers. Just in case a backup powershell script is kept. It works by just pointing to the previous dated folder of the build. Not sure why this is such an issue. Solutions can get complicated and I don't think any tool can work properly in all scenarios especially when downtime is a concern. Thats why scripts exist. Currently my solution is on Amazon, and if there was a way I'd integrate the load balancer with the scripts so that when a server is being updated load balancer doesn't send requests to it. However IIS is fast enough so this is just part of a wishlist.
- notJim 16y ago
  I think your solution is the mishmash of products (to paraphrase) that the author of the article describes. It's absolutely possible to do this, and do it well (the company I worked for previously did it for years before finally moving to capistrano), but the point is, that means every dev team in every company is spending precious time figuring out how to do this, writing the code, testing it, debugging it etc. When the developer who wrote it gets hit by a bus, someone has to go through his/her code and learn it so that it can be maintained. The alternative would be to have a standardized tool (like capistrano) that is commonly used.
drivebyacct2 16y ago
If anyone can tell me how to have a proper Debug/Release App.Config for an Azure Worker Role? I've tried all sorts of voodoo magic and hand editting ccproj files. It's a joke.
- smhinsey 16y ago
  Do you mean using the *.config transform system, or something else?
  - drivebyacct2 16y ago
    Yes. The existing ones don't work inside of the Azure packaging and upload process. At least not when invoked through Visual Studio.
fleitz 16y ago
You need MSIs and PSExec. It's the best deployment system I've used. The MSI configures your server, alerts you to missing dependencies, ngen's your DLLs and loads them into the GAC. If you tag your MSIs with the build number you've also got easy rollback. There are a couple gotchas that generally require writing an MSI helper DLL but it's no biggie. The only PITA is that if you precompile during the build stage you have to know the path of the application in advance. Then you just use a little VBScript to let PSExec work it's magic. Maybe I should put together an MSI that installs all the stuff you need to make it work. Email me if you want some help setting up an MSI / PSExec based deploy system.
- aaronbrethorst 16y ago
  No offense intended, but that sounds like an absolutely terrifying solution for deploying a website. Edit: to expand on my sentiment, this doesn't really seem like the sort of thing that an msi was designed for; and, having to potentially crank out some one-off vbscript seems scary at best.
- henrikschroder 16y ago
  The GAC? You actually use the GAC? Successfully? You're the first person I've seen that does that.
  - fleitz 16y ago
    It's pretty easy, most commercial DLLs are signed, and most open source projects that aren't simply get compiled from source. MSI makes it super simple to add as DLL to the GAC. You do need versioning support in your DLLs but once you have the infrastructure in place it's not a big deal.
plasma 16y ago
We deploy our releases by switching to a subversion branch on the web server/s - at the same time if necessary. Rollbacks are easy because we just switch back to a previous 'release' snapshot branch. Works well enough for us so far.
minhajuddin 16y ago
ASP.NET deployment is broken, but any good dev should be able to come up with his own recipe very easily. I had psexec/rake/albacore/fluentmigrator deployment setup for my asp.net mvc servers and it worked out fine for me. I even did a blog post on it: http://minhajuddin.com/2011/02/04/deploy-asp-net-mvc-applications-using-psexec-and-git/ http://minhajuddin.com/2011/02/04/deploy-asp-net-mvc-applica...
- trezor 16y ago
  Agreed. I've setup my own based on SSH, bash, Perforce and rsync. That said, Rob has a good point about MS owning the entire stack and it's pretty odd how they can't seem to put together a working deployment-process. Whatever you do: Don't say MSDeploy. It's a joke and getting it working requires more voodoo than cooking together your own ad-hoc stuff, like I did. One of the comments on the blog made me lighten up though: Rails Developer: if only you had a good MVC framework! MS: Done Rails Developer: if only you had a good View Engine, none of that aspx crap! MS: Done Rails Developer: If only you had a good pacakage manager like RubyGems! MS: Done Rails Developer: If only you had a good depolyment tool like Capistrano! MS:... Who knows? It might right around the corner ;)
  - chrismo 16y ago
    What's the gems-like pkg manager?
    - statenjason 16y ago
      NuGet[1] is the package manager trezor is talking about. [1]: http://nuget.codeplex.com/ http://nuget.codeplex.com/
mwdev 16y ago
git push appharbor master? Maybe appharbor should open source their deployment routine because I sure don't have any issues with deploying my apps there and rolling back when I need to.
- robconery 16y ago
  It's an HTTP git receiver - nothing magical there
mariusmg 16y ago
aspnet_compiler and push it over ftp. Some guys are just overreacting drama queens.
- robconery 16y ago
  A very convincing argument.
macca321 16y ago
I pretty much agree with MartinCron that it is easy to chain together the tools you need for your particular situation, although I run LINQPad scripts instead of powershell or ruby or xml so I get c# intellisense goodness everywhere (great for Ms.Web.Adminstration and SMO). I do wonder if some kind of multi-tenant app that holds application versions in assemblies and can roll up or down between them at will is the future though.
JonoW 16y ago
I reckon the article title isn't really accurate, there isn't anything wrong or broken with asp.net deployment, it's just that there isn't a single, complete sanctioned product from Microsoft. I don't see this as a problem, I think using a bunch of techniques (NAnt, MSBuild, xcopy, PowerShell etc) is ok, gives you lots of flexibility. Not that MS creating such an integrated tool would be a bad thing.
chrisjsmith 16y ago
I agree - it doth stink pretty badly. You either have to roll your own deployment platform (which we did but was expensive and time consuming) or you have to put up with immature half finished tools with a learning curve from hell such as msdeploy. Just make something that works, doesn't require acres of XML, does configuration properly (web.config transforms don't work on app.config btw which sucks and you cant msdeploy anything related to clickonce). Batch files + robocopy are more useful at the moment than anything OOB.
mdpm 16y ago
There is too much variance in app configuration, implementation and infrastructure to have a single tool for the job. I'm glad there's a functional gap - it leads to solving your 'issue' with code, making sure you have your own continuous integration process. Another 'button pushing' solution from Microsoft is the _last_ thing we need. People follow 'best practices' that are not actually pragmatic for the need at hand far too much already.

48 points by robconery 16y ago

I've used NAnt to build/deploy happily for years. I suppose it depends on each project or application's particular context. I will say this: If there's some kind of requirement where Dev1 will overwrite Dev2's changes on the target server, especially if we're talking about production, then you're probably doing it wrong. We never deploy from a developer's box or ad-hoc copying of files. Anything that needs to be deployed needs to be in source control, and needs to be deployed from the build box. I don't care if someone forgot an ASCX template and it takes a half hour to redo the whole thing.

aaronrc 16y ago

Didn't really understand what's wrong with msdeploy? Deploying can't get much easier than clicking a single button in VS.Net... Rolling back IS an issue but it can be mitigated by versioning one's source code and testing the site locally before deploying.

latch 16y ago
I'm envious that your experience lets you believe that local testing can mitigate the need for rolling back a production deployment.
cubicle67 16y ago
I spent a few years working for BigCo, and the one thing I learned there was an appreciation for rollback scripts. every single change had to be accompanied by a full and tested rollback script, and given often the deployment was messy (lots of integration/hardware mixes/etc) writing these was difficult and time consuming. ... but, they saved my arse on more than on occasion. moral of the story: always have a backout plan. Stuff goes wrong. not often, and if you're good, then rarely, but when it does you need a way out.
davidradcliffe 16y ago
If you don't understand what's wrong, you've never seen/used/experienced the beauty that is capistrano deployment.
- aaronrc 16y ago
  Nope I haven't. Guess I'll just keep pressing my one step VS.Net publish button in blissful ignorance :).
  - robconery 16y ago
    If you're one person working alone - then yes a "button push" might work for you. For teams of many people that like to push "panic patches" - that doesn't work out so well.
notJim 16y ago
Back in the day, a company I worked for had a very “manual” deploy process—basically, whoever deployed it (me) had to be sure to follow a 7 step process to the letter, or else Bad Things Would Happen. One time, I accidentally skipped a step or two, and Bad Things were happening, and to make it worse, I couldn't figure out what they were. There was, however, a working version of the code sitting in the directory alongside this horrifically broken monster. I knew the configs were correct because they had been working not 5 minutes ago, and I hadn't changed them. So I updated a symlink (this was on Linux/Apache, but it would apply on IIS/Windows too), and everything was happy again, though using a slightly older codebase.
trezor 16y ago
I'm sure MSDeploy is nice when it works. Serious emphasis on "when it works". I spent an entire evening trying to get MSDeploy to push something from my local machine to my VPS. Conditions: The VPS is remote and not a member of my home-domain, but the path between them is 100% whitelisted in all intermediate firewalls. How hard can that be? Evidently very hard. Despite spending the entire evening trying to get them to communicate and push files, it simply didn't happen. In the end I ended up writing my own deployment-system based on source-control, Samba and rsync in Bash. It was easier, it worked and I know why it works. If that is easier to get working than a "one click" solution, the authors of said solution better get a bigger button. I can't seem to click this one.
- jimfl 16y ago
  Oh, thank goodness. Just yesterday I tried to deploy 6 different apps from the IIS app gallery, with 0 successful deploys. I thought maybe I was starting to go retarded. Apparently there are some bugs which prevent parameterized deploys from working properly. Also, the documentation for Microsoft.Web.Deployment is very poor.
- mountaineer 16y ago
  I've had similar frustrations with msdeploy. Are you initializing the rsync/Bash from your local machine? What are you running on the server to receivie it?
  - trezor 16y ago
    It's all done by a bash-sc ript on an intermediate linux-shell which has access to source-control (perforce) and the server (via smb). I can post the script later if you are interested.

MartinCron 16y ago

I am doing some very straightforward asp.net deployment using git, TeamCity, nunit, and powershell. If the gauntlet of (unit-integration-acceptance) tests (in the master branch, naturally) all pass, the code is deployed to a staging server. Then there are some tests which run against staging. If those tests all pass, then the artifacts for the new version of the site are copied into a new directory (side by side) on the server. finally, a powershell script tells IIS to serve from the new directory. End result? Deployments are zero-downtime non-events tha happen multiple times daily. Rollbacks are trivial (and rare). Any not-yet-ready for prime time code can be checked into any other git branches. Database changes? I have a tiny bit of code, called from Application_OnStart that checks to see if it needs to do any CREATE TABLE or ALTER TABLE statements. Sure, I had to create all of this stuff myself, but it's all crazy simple, reliable, and does just what the project needs. Maybe someone could make some product to handle all of this, but the flexibility of linking together the best tools for the job wins for now. Also, it's just easy.

sajidnizami 16y ago
I am with you. I've just written powershell scripts that work in the following manner. 1. Make a dated folder and copy the new build to the folder on all servers. 2. Point IIS to the folder on all servers. Just in case a backup powershell script is kept. It works by just pointing to the previous dated folder of the build. Not sure why this is such an issue. Solutions can get complicated and I don't think any tool can work properly in all scenarios especially when downtime is a concern. Thats why scripts exist. Currently my solution is on Amazon, and if there was a way I'd integrate the load balancer with the scripts so that when a server is being updated load balancer doesn't send requests to it. However IIS is fast enough so this is just part of a wishlist.
notJim 16y ago
I think your solution is the mishmash of products (to paraphrase) that the author of the article describes. It's absolutely possible to do this, and do it well (the company I worked for previously did it for years before finally moving to capistrano), but the point is, that means every dev team in every company is spending precious time figuring out how to do this, writing the code, testing it, debugging it etc. When the developer who wrote it gets hit by a bus, someone has to go through his/her code and learn it so that it can be maintained. The alternative would be to have a standardized tool (like capistrano) that is commonly used.

drivebyacct2 16y ago

If anyone can tell me how to have a proper Debug/Release App.Config for an Azure Worker Role? I've tried all sorts of voodoo magic and hand editting ccproj files. It's a joke.

smhinsey 16y ago
Do you mean using the *.config transform system, or something else?
- drivebyacct2 16y ago
  Yes. The existing ones don't work inside of the Azure packaging and upload process. At least not when invoked through Visual Studio.

fleitz 16y ago

You need MSIs and PSExec. It's the best deployment system I've used. The MSI configures your server, alerts you to missing dependencies, ngen's your DLLs and loads them into the GAC. If you tag your MSIs with the build number you've also got easy rollback. There are a couple gotchas that generally require writing an MSI helper DLL but it's no biggie. The only PITA is that if you precompile during the build stage you have to know the path of the application in advance. Then you just use a little VBScript to let PSExec work it's magic. Maybe I should put together an MSI that installs all the stuff you need to make it work. Email me if you want some help setting up an MSI / PSExec based deploy system.

aaronbrethorst 16y ago
No offense intended, but that sounds like an absolutely terrifying solution for deploying a website. Edit: to expand on my sentiment, this doesn't really seem like the sort of thing that an msi was designed for; and, having to potentially crank out some one-off vbscript seems scary at best.
henrikschroder 16y ago
The GAC? You actually use the GAC? Successfully? You're the first person I've seen that does that.
- fleitz 16y ago
  It's pretty easy, most commercial DLLs are signed, and most open source projects that aren't simply get compiled from source. MSI makes it super simple to add as DLL to the GAC. You do need versioning support in your DLLs but once you have the infrastructure in place it's not a big deal.

plasma 16y ago

We deploy our releases by switching to a subversion branch on the web server/s - at the same time if necessary. Rollbacks are easy because we just switch back to a previous 'release' snapshot branch. Works well enough for us so far.

minhajuddin 16y ago

ASP.NET deployment is broken, but any good dev should be able to come up with his own recipe very easily. I had psexec/rake/albacore/fluentmigrator deployment setup for my asp.net mvc servers and it worked out fine for me. I even did a blog post on it: http://minhajuddin.com/2011/02/04/deploy-asp-net-mvc-applications-using-psexec-and-git/ http://minhajuddin.com/2011/02/04/deploy-asp-net-mvc-applica...

trezor 16y ago
Agreed. I've setup my own based on SSH, bash, Perforce and rsync. That said, Rob has a good point about MS owning the entire stack and it's pretty odd how they can't seem to put together a working deployment-process. Whatever you do: Don't say MSDeploy. It's a joke and getting it working requires more voodoo than cooking together your own ad-hoc stuff, like I did. One of the comments on the blog made me lighten up though: Rails Developer: if only you had a good MVC framework! MS: Done Rails Developer: if only you had a good View Engine, none of that aspx crap! MS: Done Rails Developer: If only you had a good pacakage manager like RubyGems! MS: Done Rails Developer: If only you had a good depolyment tool like Capistrano! MS:... Who knows? It might right around the corner ;)
- chrismo 16y ago
  What's the gems-like pkg manager?
  - statenjason 16y ago
    NuGet[1] is the package manager trezor is talking about. [1]: http://nuget.codeplex.com/ http://nuget.codeplex.com/

mwdev 16y ago

git push appharbor master? Maybe appharbor should open source their deployment routine because I sure don't have any issues with deploying my apps there and rolling back when I need to.

robconery 16y ago
It's an HTTP git receiver - nothing magical there

mariusmg 16y ago

aspnet_compiler and push it over ftp. Some guys are just overreacting drama queens.

robconery 16y ago
A very convincing argument.

macca321 16y ago

I pretty much agree with MartinCron that it is easy to chain together the tools you need for your particular situation, although I run LINQPad scripts instead of powershell or ruby or xml so I get c# intellisense goodness everywhere (great for Ms.Web.Adminstration and SMO). I do wonder if some kind of multi-tenant app that holds application versions in assemblies and can roll up or down between them at will is the future though.

JonoW 16y ago

I reckon the article title isn't really accurate, there isn't anything wrong or broken with asp.net deployment, it's just that there isn't a single, complete sanctioned product from Microsoft. I don't see this as a problem, I think using a bunch of techniques (NAnt, MSBuild, xcopy, PowerShell etc) is ok, gives you lots of flexibility. Not that MS creating such an integrated tool would be a bad thing.

chrisjsmith 16y ago

I agree - it doth stink pretty badly. You either have to roll your own deployment platform (which we did but was expensive and time consuming) or you have to put up with immature half finished tools with a learning curve from hell such as msdeploy. Just make something that works, doesn't require acres of XML, does configuration properly (web.config transforms don't work on app.config btw which sucks and you cant msdeploy anything related to clickonce). Batch files + robocopy are more useful at the moment than anything OOB.

mdpm 16y ago

There is too much variance in app configuration, implementation and infrastructure to have a single tool for the job. I'm glad there's a functional gap - it leads to solving your 'issue' with code, making sure you have your own continuous integration process. Another 'button pushing' solution from Microsoft is the _last_ thing we need. People follow 'best practices' that are not actually pragmatic for the need at hand far too much already.