4 ms·

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't repl

by aasasd 7y ago

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and the data was migrated, all while the site was online. The code expected to find either version of the db, usually signaled by a flag for the shard. If the changes were too difficult to do online, relatively small shards of the db were put on maintenance. Errors were normally caught after a migration of one shard, so switching it back wasn't too painful. Database operations for the migrations were a mix of automatic updates to the list of table fields, and data-moving queries written by hand in the migration scripts—the db structure didn't really look like your regular ORM anyway. This approach served us quite well, for about five years that I was there—with a lot of visitors and data, dozens of servers and multiple deployments a day.

wy35 7y ago
Since you mentioned symlink, does that mean you have each production version of the app on the same machine?
- twunde 7y ago
  Typically yes. This pattern was more common prior to containers and I think built into Ruby's Capistrano. The main benefit is that rollbacks were _quick_ especially compared to kubernetes or ecs
- aasasd 7y ago
  Not ‘each,’ but a few recent versions. Not sure what you mean by ‘on the same machine,’ though. It's not like we flip server roles between releases. Multiple backend servers all had those several versions of the backend code, and the machines serving the frontend stuff had corresponding versions of that.
ransom1538 7y ago
"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, and more dangerous with prod. "For the database, during a migration" There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point. Altering a column or dropping a column is extremely dangerous.
- aasasd 7y ago
  > whip through a thousand vms less than a second with this method I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.
  - beebs93 7y ago
    Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for. One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.
    - sagichmal 7y ago
      > Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.) Strange terms. Isn't this just a canary?
      - beebs93 7y ago
        Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.
aasasd 7y ago
BTW, copying the entire app to a new directory did begin to take a while after a couple years, and I never got around to solving that. However, I believe it's quite possible to implement COW: hardlink the hell out of the current version, then beat rsync into replacing differing files instead of updating them in-place. The hardlinking part could be done before the deployment even begins—or you could copy the files instead of hardlinking to avoid altering them accidentally. Or could probably just use overlayfs or something like that. Note that, while containers can give you COW, they shouldn't really be necessary for the files of the app. And with php-fpm, changing the app dir is faster than restarting containers: it's actually done via changing the Nginx config.
- arpa 7y ago
  Hmmm... restarting container is usually on par with changing directory speed (i.e. there is no perceivable difference), unless you do some heavy lifting in your entrypoint which you shouldn't, at least in prod. We also used to do a symlink method, but proper kubernetes ci/cd setup is so much more better.

by aasasd 7y ago

wy35 7y ago

Since you mentioned symlink, does that mean you have each production version of the app on the same machine?

twunde 7y ago
Typically yes. This pattern was more common prior to containers and I think built into Ruby's Capistrano. The main benefit is that rollbacks were _quick_ especially compared to kubernetes or ecs
aasasd 7y ago
Not ‘each,’ but a few recent versions. Not sure what you mean by ‘on the same machine,’ though. It's not like we flip server roles between releases. Multiple backend servers all had those several versions of the backend code, and the machines serving the frontend stuff had corresponding versions of that.

ransom1538 7y ago

"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, and more dangerous with prod. "For the database, during a migration" There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point. Altering a column or dropping a column is extremely dangerous.

aasasd 7y ago
> whip through a thousand vms less than a second with this method I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.
- beebs93 7y ago
  Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for. One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.
  - sagichmal 7y ago
    > Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.) Strange terms. Isn't this just a canary?
    - beebs93 7y ago
      Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.

aasasd 7y ago

BTW, copying the entire app to a new directory did begin to take a while after a couple years, and I never got around to solving that. However, I believe it's quite possible to implement COW: hardlink the hell out of the current version, then beat rsync into replacing differing files instead of updating them in-place. The hardlinking part could be done before the deployment even begins—or you could copy the files instead of hardlinking to avoid altering them accidentally. Or could probably just use overlayfs or something like that. Note that, while containers can give you COW, they shouldn't really be necessary for the files of the app. And with php-fpm, changing the app dir is faster than restarting containers: it's actually done via changing the Nginx config.

arpa 7y ago
Hmmm... restarting container is usually on par with changing directory speed (i.e. there is no perceivable difference), unless you do some heavy lifting in your entrypoint which you shouldn't, at least in prod. We also used to do a symlink method, but proper kubernetes ci/cd setup is so much more better.