14 ms·

Stack Overflow: How We Do Deployment

428 points by Nick-Craver 10y ago

radicalbyte 10y ago
It's funny to see that stackoverflow came to exactly the same solution for database migrations on the Microsoft stack as my team did, even down to the test procedure. Simple, safe and very effective :)
- jdc0589 10y ago
  its also pretty much the same solution colleagues and I came up with a few years ago for a migration tool we were working on. It's kind of abandon-ware at this point, but this version is pretty far along: https://github.com/jdc0589/mite-node https://github.com/jdc0589/mite-node
ohitsdom 10y ago
This feels really clunky to me, but maybe I'm just not getting it. I'm trying to implement a more automated build/deploy process at my current place of employment and am basically modeling it off of Github's [0], which seems to have a better feel. Obviously the quality of the process needs to be high, but when it's effortless and "fun" then everybody wins. [0] http://githubengineering.com/deploying-branches-to-github-com/ http://githubengineering.com/deploying-branches-to-github-co...
richardwhiuk 10y ago
A developer is new, and early on we want code reviews A developer is working on a big (or risky) feature, and wants a one-off code review This implies you don't normally do code review??
- sklivvz1971 10y ago
  Yes, we don't normally code review stuff we don't need to. We trust each other.
  - scottaj2 10y ago
    I don't think doing code reviews implies a lack of trust. I trust everyone on my team but we code review everything. For that matter, the fact that we trust/have good relationships with everyone makes code reviews more effective because we can be more candid. We often find issues in code reviews like edge cases that weren't thought of, code that could be refactored to use an existing utility or patter the author wasn't aware of, etc.
    - gnaritas 10y ago
      Not a bad practice, but costly in dev time and a trade off many aren't willing to make especially in smaller companies where mistakes aren't as costly as all that extra developer time. Personally I do code reviews mostly to share knowledge and culture rather than looking for bugs. Occasionally a bug is found, but I don't generally have the time to review the logic, just the style.
    - numo16 10y ago
      I work for a company that averages around 50 production deployments per day for our customer-facing ERP, and we only do code reviews for new devs and changes to underlying framework changes for mostly the same reason sklivvz1971 mentions. We rollback very infrequently and a majority of our devs can deploy to prod with the push of a button as needed (this includes both application and database code). Not arguing that code review is unnecessary, just feel that with proper training and having devs with good judgement can help reduce the likely hood of breaking things when deploying small changes frequently.
      - scottaj2 10y ago
        We rarely catch things breaking in code reviews, I agree they are really bad at finding bugs. Automated tests and linters are better at finding stuff like that. The things we usually address in code reviews are architecture and code design issues, and occasionally edge and interaction issues that are outside the scope of what might have been considered when implementing. We also have frequent production deployments that everyone on the team can do, I view that as something that is independent of code review.
- darkhorn 10y ago
  Well, it is not an enterprice applucation. It is free web site. Yeah, not a very good practice but not very critical too.
  - aalear 10y ago
    I dunno that I'd say an enterprise app is inherently more valuable than Stack Overflow. ;) But to be clear - it's not that we never do reviews. It's more that we have an "ask for it when you need it" type of policy. New hires get regular reviews, so initial architecture/style concerns are addressed then... along with teaching the logistics of our code reviews (push to a branch & submit a merge request).
- manigandham 10y ago
  Code review is overrated. Over my career I've built 8/9 figure revenue platforms without code review or serious test setups. A lot of this cruft is unnecessary when compared to good domain knowledge and solid coding focus.
  - ohitsdom 10y ago
    > Over my career I've built 8/9 figure revenue platforms without code review or serious test setups. This does not diminish code reviews. Would you be a better engineer today if you had regularly participated in code reviews? Would your coworkers?
    - ralmeida 10y ago
      I believe s/he would, and likely knows it. They just said it was overrated, not useless. They just draw the tradeoff line differently.
    - meddlepal 10y ago
      One of the problems with code reviews is they're often done really poorly, for a number of reasons: 1. Reviewers are often poorly trained to provide good design reviews and default to nit-picky stuff a code linter should pickup. Human linting is just a poor use of time and money. 2. Nobody seems to ever have time for them to deep dive into the code. 3. Few engineers seem to ever actually want to do them. 4. Reviews can become hostile. Code reviews are probably really important in some fields, for example, medical equipment, aviation, etc, but for the vast number of projects where we're shoveling A bits to B bucket or transforming C bits into D bits it's overkill and companies would be better off investing the massive amount of wasted time in better CI/CD infrastructure.
    - manigandham 10y ago
      > Would you be a better engineer today if you had regularly participated in code reviews? Would your coworkers? Maybe, but probably not. It's not like I never see someone's code, it's right there when I'm working in the same code base and I can go through commits to see the high-level changes. There are lots of ways to become a better engineer and code reviews are pretty far down on the list in my view. They usually just turn into tedious ordeals that burn up actual productive time.
richardwhiuk 10y ago
Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for…itself. Oh wait, that’s not a fun fact — it’s actually a pain in the ass. Surely that should just be a very fast lookup in /etc/hosts?
- Nick-Craver 10y ago
  The problem here is that these services move - so if it's in /etc/hosts, our failover mechanisms (to a DR data center which has a replica server) are severely hindered. We're adding some local cache, but there are some nasty gotchas with subnet-local ordering on resolution. By this I mean, New York resolves the local /16 first, and Denver resolve's its local /16...instead BIND doesn't care (by default) and likes to auth against let's say: the London office. Good times!
  - KaiserPro 10y ago
    but thats what DNS scope is for surely? we had n datacenters each named after their city: ldn.$company.com, ny.$company.com etc etc. in the DHCP we pushed out the search order so that it would try and resolve locally, if that failed try a level up until something worked. This meant that you'd bind to service it would first look up service.$location.$company.com, if thats not there it'd try and find service.$company.com This cuts down the need for nasty split horizon DNS, moving VMs/services/machines between datacenters was simple and zero config. If you were taking a service out of commission in one datacenter, you'd CNAME service.$location.$company.com to a different datacenter, do a staged kick of the machines, and BOOM failed over with only one config change. On a side note, you can use SSSD or shudder NSLCD to cache DNS.
    - Nick-Craver 10y ago
      We do, but in the specific case of Active Directory, we want to fail over and auth against another data center if the primary is offline. This means for our domain, the local (to the /16) domain controllers are returned first and then the others. The problem is BIND locally doesn't preserve this order and applications are suddenly authenticating across the planet. DNS devolution isn't a good idea here, since the external domain is a wildcard. We'll be paying for that mistake from long ago until (if ever) we change the internal domain name. This is a pretty recent problem we're just now getting to because the DNS volume has been a back-burner issue - we'll look into permanent solutions for all Linux services after the CDN testing completes. Recommendations on the Linux DNS caching are much appreciated - we'll review each. It's something that just hasn't been an issue in the past so not experts on that particular area. I am surprised caching hasn't landed natively in most of the major distros yet though.
taurath 10y ago
How people manage git and source control tells you a lot of things about a company's culture. They said that most commits go directly onto master and this works for them, which indicates: - Good rapid communication about who is working where. People are generally not touching the same code or else you'd run into frequent collisions (solvable via rebasing of course but they would be doing more branching if it were a thing to happen very frequently I'd suspect) - The developers are given autonomy and have assumed some level of mastery over whatever their domain is. Trust in each developer's ability to commit well-formed and considered code. - They have a comprehensive test stack, which helps verify the above point and keep it sane
- fabian2k 10y ago
  The comprehensive test stack are mostly the users on meta, which gets the deployed code before the other sites. From what I heard so far, Stack Overflow and the Stack Exchange sites don't have a significant amount of automated tests.
- manacit 10y ago
  I found this very curious - by their own admission, this also means that most code _does not get reviewed_ before it lands in production. To me, this is quite scary, and I would be very hesitant to adopt this for any large-scale project or company. IMO, code review is a cornerstone of code quality and production stability - the number of dumb (and smart!) mistakes in my code that have been caught in CR are numerous, and it's a big portion of my workflow. There are times when I feel it's redundant (one-line changes, spelling mistakes, etc), but I wouldn't trade those slowdowns for a system where I only got review when I explicitly wanted it. Of course, for pre-production project and/or times when speed is of the utmost concern, dropping back to committing to master might make sense, but for an established and (I'm assuming) fairly large/complex codebase, I would think that it would be best for maintainability and stability to review code before it's deployed.
  - enraged_camel 10y ago
    Shouldn't the types of mistakes you're worried about be caught by the automated tests? IMO code review is more about bigger picture design, e.g. if a piece of code would yield the correct result but with sub-optimal performance (which can also be caught by automated tests but not always).
adontz 10y ago
I was always wondering if there is some way of database deployment which does not suck. And all I see - every professional team ends with bunch of 074-add-column-to-table.sql files. I mean, code deployment can be organized much better. You can make graceful restarts, transactional updates, etc. Actually, almost nobody backups old code version because deployment of new one may fail, but database upgrades are so fragile and making backups is a must not only because you afraid of upgrade process may be interrupted and leave database in inconsistent state, but because properly done upgrade may result in corrupted state.
- hansjorg 10y ago
  Have you checked out something like LiquiBase or Flyway? They offer a good improvement over the ad hoc solution. http://www.liquibase.org/ http://www.liquibase.org/ https://flywaydb.org/ https://flywaydb.org/
  - dlubarov 10y ago
    A lot of our apps use Flyway and it works pretty well, though it tries to do everything in a transaction. Normally that's desirable -- we don't want a migration to partially succeed -- but we would often use CREATE INDEX CONCURRENTLY in Postgres databases, which can't run in a transaction. In those cases we would need to manually run the migrations and update Flyway's schema_version table, which is annoyingly complex (11 columns, some with no obvious purpose).
- rossng 10y ago
  There definitely is some tooling available to help - Redgate, many ORMs - but I agree that it's somewhat lacking. I think the bigger problem is cultural - many programmers either don't really understand databases/data modelling or they don't care about it. After all, you don't really have to worry about it when you're just starting out - almost any schema will work. That is, right up until you have to modify it. By the time it becomes an issue, the culture has crystallised and changing the database is too risky. For some reason, a lot of companies are largely unwilling to spend money on good database management/migration tools - even if they're paying a stack of cash for SQL Server.
ngrilly 10y ago
The article says: > there is a slight chance that someone will get new static content with an old hash (if they hit a CDN miss for a piece of content that actually changed this build) Anyone has a solution to this problem?
- dpark 10y ago
  Without poking at their site to see how they behave, I assume they just append a cache breaking param onto the URL for a static entity. Assuming that assumption is correct, the fix is to make the hash meaningful to the process serving the static entities. This adds complexity as now your static site needs to serve requests based on the hash. This isn't conceptually complex but it means you must deploy at least two versions of your static resource (one for each hash in the wild). And you still have to do two phase deployment in this model (static resources first, then the "real" site). Or you can build redirect or proxying support from v2 to v1 and vice versa, which is a much uglier problem to solve, but eliminates the need for two phase deployment. Since they have short deployments, their solution is pretty elegant. If you have long deployments, the hash aware server becomes sensible. If you're a masochist, the two way proxying becomes attractive.
- windowsworkstoo 10y ago
  Nope, we solve it the same way, it's a pain. I did a presentation once for devs where I unraveled the up-to-9 different layers of caching between an end user and our website (when you take into account the browser and any tiered CDN caching) It's a pest of a problem but pre-deploying static assets is the best answer.
- mfontani 10y ago
  Push new static content first to all servers/cdn, _then_ bust the CDN cache / push a cache bust
  - dpark 10y ago
    This doesn't resolve the issue. The fundamental issue here is that two versions are running in parallel. * If you push static content and web pages together, you get V1 and V2 of both static and web, and you end up with incorrect static resources served in both directions. This approach is only reasonable if your deployment strategy is to take a service outage to upgrade all machines together. * If you push web first, you get the ugly scenario described in the article where V1 resources get served with V2 hashes and cached for 7 days. * If you push static content first, you still have V2 static content being served for V1 web pages. The "cache bust" doesn't matter. Somewhere a cache will expire and someone will get V2 static resources for a V1 page. You have to deal with the two versions somehow if you want to resolve the issue fully.
nxzero 10y ago
SE/SO is for it's size an amazingly high proformance team. It's rare to hear them say it, in fact, I don't recall ever hearing them say this. The importantance of the cohesion and trust amoung their team is critical to their deployments. In fact, I would say it's vital to how they're able to get away with minimal amounts of code reviews for example. It's dangerous to believe this is easy or reproducible. New teams needs extensive controls in place to make sure the quality of the deployments will not negatively impact the group.
daddykotex 10y ago
> If Gitlab pricing keeps getting crazier (note: I said “crazier”, not “more expensive”), we’ll certainly re-evaluate GitHub Enterprise again. Shots fired :P
gnahckire 10y ago
Did you choose polling over webhooks for a reason? Or was webhooks recently added as a feature to Gitlab?
- Nick-Craver 10y ago
  Webhooks didn't used to work well for many builds off a single repo, but I think this changed very recently in TeamCity. Thanks for the reminder - I'll take another look this week at adding web hooks. We'd still want the poll in case of any hook failures. At the moment, Gitlab knows nothing about our builds - and we'd want to keep it simple in that regard. If we can generically configure a hook to hit TeamCity to alert of any repo updates though, that's tractable...I need to see if that's possible now.
totally 10y ago
What's with all the backslashes?
- Nick-Craver 10y ago
  Yeah! What's wi...wait, what?
  - JdeBP 10y ago
    Enjoy https://news.ycombinator.com/item?id=11433139 https://news.ycombinator.com/item?id=11433139 .
- teknologist 10y ago
  Windows. The terminal window is Cmder [1] [1] http://cmder.net/ http://cmder.net/
goldbrick 10y ago
When did everybody decide that chatbots were the new hotness for deployments?
- Nick-Craver 10y ago
  If you mean pinbot - that's literally all it does. It takes a message and pins it, knocking the old one off the pins. The build messages build...that's also literally all it does. It simply puts handy notices in the chatroom. Why wouldn't you want that integration? Everyone going to look at the build screen and polling it to see what's up is a far less efficient system. A push style notification, no matter the medium, causes far less overhead. I doubt we'll ever build from chat directly for anything production at least, simply because those are 2 different user and authentication systems in play. It's too risky, IMO.
sakopov 10y ago
I think if you're deploying .net code you're almost certainly going to follow similar build architecture with TeamCity doing most of the grunt work. We have a very similar build structure but a bit more polished I think. Our TeamCity build agents build nuget packages, deploy them to Octopus, run unit and integration tests. Octopus handles promotions of code from dev to qa, to staging and all production environments. We also write migrations for database updates using FluentMigrator which works with just about any database driver. It's a joy deploying code on an environment like this.
- rjbwork 10y ago
  Agreed on the Octopus bit. TeamCity + Octopus is practically magical. Until literally yesterday, I'd yet to find something that didn't work with minimal effort between the two.
- Griever 10y ago
  Happy to see someone else using FluentMigrator. It's a fantastic library that doesn't get mentioned enough imo.
clio 10y ago
Additional anecdata: At my place of employment, after the required code review, we must write an essay about the change and have at least two coworkers approve it. Then we must email the essay to a mailing list of several hundred people. One of those people is the designated external reviewer, who must sign off. However, lately, this final reviewer has been requesting 10+ modifications to the deployment essay due to a managerial decision to "improve deployments". Moreover, deployments are not allowed 1/4th of the year unless a vice president has signed off. Any code change requires this process.
- TheRealDunkirk 10y ago
  What industry sector? Medical? Defense?
  - clio 10y ago
    Logistics
msane 10y ago
TeamCity is a great CI system.
infocollector 10y ago
Stack Overflow uses windows? Any particular reason to do this?
- throwaway_xx9 10y ago
  Joel was a former product manager at Microsoft.
- romanovcode 10y ago
  Written in C#
- sklivvz1971 10y ago
  The founders and first devs were proficient in it. It works well enough for our needs (e.g. C# has one of the best GCs on the market), and no one is a platform/language zealot, so we keep on using it. Stuff we added later runs on other platforms, as needed (e.g. we run redis and elastic search on CentOS, our server monitoring tool Bosun is written in go...)
NKCSS 10y ago
A few things strike me as odd/sub-optimal: - migration id's freeform in chat -> why not central db with an auto-increment column? - Use chat to prevent 'migration collisions' -> Same central db/msmq/whatever to report start/stop of migrations and lock based on that...
- GordonS 10y ago
  I guess this is because using a team chat is 'good enough' for them without adding another layer of tooling
JdeBP 10y ago
> Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for … itself. This is wrong in two ways, and isn't factual at all. First, the cause of the queries is nothing to do with whether DNS query answers are cached locally or not. There is no causal link here. What causes such queries is applications that repeatedly look up the same things, over and over again; not the DNS server arrangements. One could argue that this is poor design in the applications, and that they should remember the results of lookups. But there's a good counterargument to make that this is good design. Applications shouldn't all maintain their own private idiosyncratic DNS lookup result caches. History teaches us that applications attempting to cache their own DNS lookup results invariably do it poorly, for various reasons. (See Mozilla bug #162871, for one of several examples.) Good design is to hand that over to a common external subsystem shared by all applications. Which brings us to the second way in which this is wrong. A common Unix tradition is for all machines to have local proxy DNS servers. Linux operating systems have plenty of such server softwares that can be used: dnsmasq, pdnsd, PowerDNS, unbound ... One of the simplest, which does only caching and doesn't attempt to wear other hats simultaneously, is even named "dnscache". Set this up listening on the loopback interface, point its back-end at the relevant external servers, point the DNS client library at it, and -- voilà! -- a local caching proxy DNS server. * http://cr.yp.to/djbdns.html http://cr.yp.to/djbdns.html * http://cr.yp.to/djbdns/run-cache.html http://cr.yp.to/djbdns/run-cache.html * http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/djbdns-problems.html#wrong-icann-root http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/djb... * http://cr.yp.to/djbdns/install.html http://cr.yp.to/djbdns/install.html I run server machines that have applications that repeatedly look up the same domain names again and again. Each runs a local dnscache instance, which ameliorates this very well.
- TheRealDunkirk 10y ago
  I guess this goes to show that you can be a full-fledged expert in one development "world," and miss easy things in another. I see this a lot in the apparent expectations of people because of their ingrained presumptions of the OS platform. This article is of interest to HN because Mr. Craver is on a leading edge of .NET development and deployment, and this is an insightful oversight. I spent 20 years getting deep with PHP, and then Rails, on Linux, where, IMO, doing CI builds with something like Jenkins or Heroku is pretty straightforward. For the past couple years, I've been doing .NET, and, while I never really stopped doing ".NET" since the VB 3.0 days, I've had a lot to learn to get into doing serious, enterprise-level stuff. I just took my first steps in setting up auto-building with Visual Studio Team Services, and my experience has left me really disappointed. Dealing with changing variable values per environment is really hacky, no matter which of a handful of tricks you want to try. So much so, I left the whole thing hanging, and have gone back to doing releases by hand. Skimming the article, and seeing discussion about this topic in the comments, leads me to conclude that my experience wasn't just limited by my ignorance, and that this is still an area that is underserved in the .NET world. You'd think someone would have neatly sorted this by now. I keep looking for the oversight in MY case, but I'm not finding it.
streeter 10y ago
I was surprised they weren't using Kiln [https://www.fogcreek.com/kiln/ https://www.fogcreek.com/kiln/], a Fog Creek product. I know SO is independent from Fog Creek now, but still a bit surprised at it. I wonder if there was a migration off at some point.
- Nick-Craver 10y ago
  Yep...for the on-premise reasons listed in the article. Once upon a time a lot of projects were on Mercurial, hosted by Kiln. The Stack Overflow repo specifically has always been on an internal Mercurial and then Git server. Originally this was for speed, now it's for speed and reliability/dependency reduction.

428 points by Nick-Craver 10y ago

It's funny to see that stackoverflow came to exactly the same solution for database migrations on the Microsoft stack as my team did, even down to the test procedure. Simple, safe and very effective :)

jdc0589 10y ago
its also pretty much the same solution colleagues and I came up with a few years ago for a migration tool we were working on. It's kind of abandon-ware at this point, but this version is pretty far along: https://github.com/jdc0589/mite-node https://github.com/jdc0589/mite-node

ohitsdom 10y ago

This feels really clunky to me, but maybe I'm just not getting it. I'm trying to implement a more automated build/deploy process at my current place of employment and am basically modeling it off of Github's [0], which seems to have a better feel. Obviously the quality of the process needs to be high, but when it's effortless and "fun" then everybody wins. [0] http://githubengineering.com/deploying-branches-to-github-com/ http://githubengineering.com/deploying-branches-to-github-co...

richardwhiuk 10y ago

A developer is new, and early on we want code reviews A developer is working on a big (or risky) feature, and wants a one-off code review This implies you don't normally do code review??

sklivvz1971 10y ago
Yes, we don't normally code review stuff we don't need to. We trust each other.
- scottaj2 10y ago
  I don't think doing code reviews implies a lack of trust. I trust everyone on my team but we code review everything. For that matter, the fact that we trust/have good relationships with everyone makes code reviews more effective because we can be more candid. We often find issues in code reviews like edge cases that weren't thought of, code that could be refactored to use an existing utility or patter the author wasn't aware of, etc.
  - gnaritas 10y ago
    Not a bad practice, but costly in dev time and a trade off many aren't willing to make especially in smaller companies where mistakes aren't as costly as all that extra developer time. Personally I do code reviews mostly to share knowledge and culture rather than looking for bugs. Occasionally a bug is found, but I don't generally have the time to review the logic, just the style.
  - numo16 10y ago
    I work for a company that averages around 50 production deployments per day for our customer-facing ERP, and we only do code reviews for new devs and changes to underlying framework changes for mostly the same reason sklivvz1971 mentions. We rollback very infrequently and a majority of our devs can deploy to prod with the push of a button as needed (this includes both application and database code). Not arguing that code review is unnecessary, just feel that with proper training and having devs with good judgement can help reduce the likely hood of breaking things when deploying small changes frequently.
    - scottaj2 10y ago
      We rarely catch things breaking in code reviews, I agree they are really bad at finding bugs. Automated tests and linters are better at finding stuff like that. The things we usually address in code reviews are architecture and code design issues, and occasionally edge and interaction issues that are outside the scope of what might have been considered when implementing. We also have frequent production deployments that everyone on the team can do, I view that as something that is independent of code review.
darkhorn 10y ago
Well, it is not an enterprice applucation. It is free web site. Yeah, not a very good practice but not very critical too.
- aalear 10y ago
  I dunno that I'd say an enterprise app is inherently more valuable than Stack Overflow. ;) But to be clear - it's not that we never do reviews. It's more that we have an "ask for it when you need it" type of policy. New hires get regular reviews, so initial architecture/style concerns are addressed then... along with teaching the logistics of our code reviews (push to a branch & submit a merge request).
manigandham 10y ago
Code review is overrated. Over my career I've built 8/9 figure revenue platforms without code review or serious test setups. A lot of this cruft is unnecessary when compared to good domain knowledge and solid coding focus.
- ohitsdom 10y ago
  > Over my career I've built 8/9 figure revenue platforms without code review or serious test setups. This does not diminish code reviews. Would you be a better engineer today if you had regularly participated in code reviews? Would your coworkers?
  - ralmeida 10y ago
    I believe s/he would, and likely knows it. They just said it was overrated, not useless. They just draw the tradeoff line differently.
  - meddlepal 10y ago
    One of the problems with code reviews is they're often done really poorly, for a number of reasons: 1. Reviewers are often poorly trained to provide good design reviews and default to nit-picky stuff a code linter should pickup. Human linting is just a poor use of time and money. 2. Nobody seems to ever have time for them to deep dive into the code. 3. Few engineers seem to ever actually want to do them. 4. Reviews can become hostile. Code reviews are probably really important in some fields, for example, medical equipment, aviation, etc, but for the vast number of projects where we're shoveling A bits to B bucket or transforming C bits into D bits it's overkill and companies would be better off investing the massive amount of wasted time in better CI/CD infrastructure.
  - manigandham 10y ago
    > Would you be a better engineer today if you had regularly participated in code reviews? Would your coworkers? Maybe, but probably not. It's not like I never see someone's code, it's right there when I'm working in the same code base and I can go through commits to see the high-level changes. There are lots of ways to become a better engineer and code reviews are pretty far down on the list in my view. They usually just turn into tedious ordeals that burn up actual productive time.

richardwhiuk 10y ago

Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for…itself. Oh wait, that’s not a fun fact — it’s actually a pain in the ass. Surely that should just be a very fast lookup in /etc/hosts?

Nick-Craver 10y ago
The problem here is that these services move - so if it's in /etc/hosts, our failover mechanisms (to a DR data center which has a replica server) are severely hindered. We're adding some local cache, but there are some nasty gotchas with subnet-local ordering on resolution. By this I mean, New York resolves the local /16 first, and Denver resolve's its local /16...instead BIND doesn't care (by default) and likes to auth against let's say: the London office. Good times!
- KaiserPro 10y ago
  but thats what DNS scope is for surely? we had n datacenters each named after their city: ldn.$company.com, ny.$company.com etc etc. in the DHCP we pushed out the search order so that it would try and resolve locally, if that failed try a level up until something worked. This meant that you'd bind to service it would first look up service.$location.$company.com, if thats not there it'd try and find service.$company.com This cuts down the need for nasty split horizon DNS, moving VMs/services/machines between datacenters was simple and zero config. If you were taking a service out of commission in one datacenter, you'd CNAME service.$location.$company.com to a different datacenter, do a staged kick of the machines, and BOOM failed over with only one config change. On a side note, you can use SSSD or shudder NSLCD to cache DNS.
  - Nick-Craver 10y ago
    We do, but in the specific case of Active Directory, we want to fail over and auth against another data center if the primary is offline. This means for our domain, the local (to the /16) domain controllers are returned first and then the others. The problem is BIND locally doesn't preserve this order and applications are suddenly authenticating across the planet. DNS devolution isn't a good idea here, since the external domain is a wildcard. We'll be paying for that mistake from long ago until (if ever) we change the internal domain name. This is a pretty recent problem we're just now getting to because the DNS volume has been a back-burner issue - we'll look into permanent solutions for all Linux services after the CDN testing completes. Recommendations on the Linux DNS caching are much appreciated - we'll review each. It's something that just hasn't been an issue in the past so not experts on that particular area. I am surprised caching hasn't landed natively in most of the major distros yet though.

taurath 10y ago

How people manage git and source control tells you a lot of things about a company's culture. They said that most commits go directly onto master and this works for them, which indicates: - Good rapid communication about who is working where. People are generally not touching the same code or else you'd run into frequent collisions (solvable via rebasing of course but they would be doing more branching if it were a thing to happen very frequently I'd suspect) - The developers are given autonomy and have assumed some level of mastery over whatever their domain is. Trust in each developer's ability to commit well-formed and considered code. - They have a comprehensive test stack, which helps verify the above point and keep it sane

fabian2k 10y ago
The comprehensive test stack are mostly the users on meta, which gets the deployed code before the other sites. From what I heard so far, Stack Overflow and the Stack Exchange sites don't have a significant amount of automated tests.
manacit 10y ago
I found this very curious - by their own admission, this also means that most code _does not get reviewed_ before it lands in production. To me, this is quite scary, and I would be very hesitant to adopt this for any large-scale project or company. IMO, code review is a cornerstone of code quality and production stability - the number of dumb (and smart!) mistakes in my code that have been caught in CR are numerous, and it's a big portion of my workflow. There are times when I feel it's redundant (one-line changes, spelling mistakes, etc), but I wouldn't trade those slowdowns for a system where I only got review when I explicitly wanted it. Of course, for pre-production project and/or times when speed is of the utmost concern, dropping back to committing to master might make sense, but for an established and (I'm assuming) fairly large/complex codebase, I would think that it would be best for maintainability and stability to review code before it's deployed.
- enraged_camel 10y ago
  Shouldn't the types of mistakes you're worried about be caught by the automated tests? IMO code review is more about bigger picture design, e.g. if a piece of code would yield the correct result but with sub-optimal performance (which can also be caught by automated tests but not always).

adontz 10y ago

I was always wondering if there is some way of database deployment which does not suck. And all I see - every professional team ends with bunch of 074-add-column-to-table.sql files. I mean, code deployment can be organized much better. You can make graceful restarts, transactional updates, etc. Actually, almost nobody backups old code version because deployment of new one may fail, but database upgrades are so fragile and making backups is a must not only because you afraid of upgrade process may be interrupted and leave database in inconsistent state, but because properly done upgrade may result in corrupted state.

hansjorg 10y ago
Have you checked out something like LiquiBase or Flyway? They offer a good improvement over the ad hoc solution. http://www.liquibase.org/ http://www.liquibase.org/ https://flywaydb.org/ https://flywaydb.org/
- dlubarov 10y ago
  A lot of our apps use Flyway and it works pretty well, though it tries to do everything in a transaction. Normally that's desirable -- we don't want a migration to partially succeed -- but we would often use CREATE INDEX CONCURRENTLY in Postgres databases, which can't run in a transaction. In those cases we would need to manually run the migrations and update Flyway's schema_version table, which is annoyingly complex (11 columns, some with no obvious purpose).
rossng 10y ago
There definitely is some tooling available to help - Redgate, many ORMs - but I agree that it's somewhat lacking. I think the bigger problem is cultural - many programmers either don't really understand databases/data modelling or they don't care about it. After all, you don't really have to worry about it when you're just starting out - almost any schema will work. That is, right up until you have to modify it. By the time it becomes an issue, the culture has crystallised and changing the database is too risky. For some reason, a lot of companies are largely unwilling to spend money on good database management/migration tools - even if they're paying a stack of cash for SQL Server.

ngrilly 10y ago

The article says: > there is a slight chance that someone will get new static content with an old hash (if they hit a CDN miss for a piece of content that actually changed this build) Anyone has a solution to this problem?

dpark 10y ago
Without poking at their site to see how they behave, I assume they just append a cache breaking param onto the URL for a static entity. Assuming that assumption is correct, the fix is to make the hash meaningful to the process serving the static entities. This adds complexity as now your static site needs to serve requests based on the hash. This isn't conceptually complex but it means you must deploy at least two versions of your static resource (one for each hash in the wild). And you still have to do two phase deployment in this model (static resources first, then the "real" site). Or you can build redirect or proxying support from v2 to v1 and vice versa, which is a much uglier problem to solve, but eliminates the need for two phase deployment. Since they have short deployments, their solution is pretty elegant. If you have long deployments, the hash aware server becomes sensible. If you're a masochist, the two way proxying becomes attractive.
windowsworkstoo 10y ago
Nope, we solve it the same way, it's a pain. I did a presentation once for devs where I unraveled the up-to-9 different layers of caching between an end user and our website (when you take into account the browser and any tiered CDN caching) It's a pest of a problem but pre-deploying static assets is the best answer.
mfontani 10y ago
Push new static content first to all servers/cdn, _then_ bust the CDN cache / push a cache bust
- dpark 10y ago
  This doesn't resolve the issue. The fundamental issue here is that two versions are running in parallel. * If you push static content and web pages together, you get V1 and V2 of both static and web, and you end up with incorrect static resources served in both directions. This approach is only reasonable if your deployment strategy is to take a service outage to upgrade all machines together. * If you push web first, you get the ugly scenario described in the article where V1 resources get served with V2 hashes and cached for 7 days. * If you push static content first, you still have V2 static content being served for V1 web pages. The "cache bust" doesn't matter. Somewhere a cache will expire and someone will get V2 static resources for a V1 page. You have to deal with the two versions somehow if you want to resolve the issue fully.

nxzero 10y ago

SE/SO is for it's size an amazingly high proformance team. It's rare to hear them say it, in fact, I don't recall ever hearing them say this. The importantance of the cohesion and trust amoung their team is critical to their deployments. In fact, I would say it's vital to how they're able to get away with minimal amounts of code reviews for example. It's dangerous to believe this is easy or reproducible. New teams needs extensive controls in place to make sure the quality of the deployments will not negatively impact the group.

daddykotex 10y ago

> If Gitlab pricing keeps getting crazier (note: I said “crazier”, not “more expensive”), we’ll certainly re-evaluate GitHub Enterprise again. Shots fired :P

gnahckire 10y ago

Did you choose polling over webhooks for a reason? Or was webhooks recently added as a feature to Gitlab?

Nick-Craver 10y ago
Webhooks didn't used to work well for many builds off a single repo, but I think this changed very recently in TeamCity. Thanks for the reminder - I'll take another look this week at adding web hooks. We'd still want the poll in case of any hook failures. At the moment, Gitlab knows nothing about our builds - and we'd want to keep it simple in that regard. If we can generically configure a hook to hit TeamCity to alert of any repo updates though, that's tractable...I need to see if that's possible now.

totally 10y ago

What's with all the backslashes?

Nick-Craver 10y ago
Yeah! What's wi...wait, what?
- JdeBP 10y ago
  Enjoy https://news.ycombinator.com/item?id=11433139 https://news.ycombinator.com/item?id=11433139 .
teknologist 10y ago
Windows. The terminal window is Cmder [1] [1] http://cmder.net/ http://cmder.net/

goldbrick 10y ago

When did everybody decide that chatbots were the new hotness for deployments?

Nick-Craver 10y ago
If you mean pinbot - that's literally all it does. It takes a message and pins it, knocking the old one off the pins. The build messages build...that's also literally all it does. It simply puts handy notices in the chatroom. Why wouldn't you want that integration? Everyone going to look at the build screen and polling it to see what's up is a far less efficient system. A push style notification, no matter the medium, causes far less overhead. I doubt we'll ever build from chat directly for anything production at least, simply because those are 2 different user and authentication systems in play. It's too risky, IMO.

sakopov 10y ago

I think if you're deploying .net code you're almost certainly going to follow similar build architecture with TeamCity doing most of the grunt work. We have a very similar build structure but a bit more polished I think. Our TeamCity build agents build nuget packages, deploy them to Octopus, run unit and integration tests. Octopus handles promotions of code from dev to qa, to staging and all production environments. We also write migrations for database updates using FluentMigrator which works with just about any database driver. It's a joy deploying code on an environment like this.

rjbwork 10y ago
Agreed on the Octopus bit. TeamCity + Octopus is practically magical. Until literally yesterday, I'd yet to find something that didn't work with minimal effort between the two.
Griever 10y ago
Happy to see someone else using FluentMigrator. It's a fantastic library that doesn't get mentioned enough imo.

clio 10y ago

Additional anecdata: At my place of employment, after the required code review, we must write an essay about the change and have at least two coworkers approve it. Then we must email the essay to a mailing list of several hundred people. One of those people is the designated external reviewer, who must sign off. However, lately, this final reviewer has been requesting 10+ modifications to the deployment essay due to a managerial decision to "improve deployments". Moreover, deployments are not allowed 1/4th of the year unless a vice president has signed off. Any code change requires this process.

TheRealDunkirk 10y ago
What industry sector? Medical? Defense?
- clio 10y ago
  Logistics

msane 10y ago

TeamCity is a great CI system.

infocollector 10y ago

Stack Overflow uses windows? Any particular reason to do this?

throwaway_xx9 10y ago
Joel was a former product manager at Microsoft.
romanovcode 10y ago
Written in C#
sklivvz1971 10y ago
The founders and first devs were proficient in it. It works well enough for our needs (e.g. C# has one of the best GCs on the market), and no one is a platform/language zealot, so we keep on using it. Stuff we added later runs on other platforms, as needed (e.g. we run redis and elastic search on CentOS, our server monitoring tool Bosun is written in go...)

NKCSS 10y ago

A few things strike me as odd/sub-optimal: - migration id's freeform in chat -> why not central db with an auto-increment column? - Use chat to prevent 'migration collisions' -> Same central db/msmq/whatever to report start/stop of migrations and lock based on that...

GordonS 10y ago
I guess this is because using a team chat is 'good enough' for them without adding another layer of tooling

JdeBP 10y ago

> Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for … itself. This is wrong in two ways, and isn't factual at all. First, the cause of the queries is nothing to do with whether DNS query answers are cached locally or not. There is no causal link here. What causes such queries is applications that repeatedly look up the same things, over and over again; not the DNS server arrangements. One could argue that this is poor design in the applications, and that they should remember the results of lookups. But there's a good counterargument to make that this is good design. Applications shouldn't all maintain their own private idiosyncratic DNS lookup result caches. History teaches us that applications attempting to cache their own DNS lookup results invariably do it poorly, for various reasons. (See Mozilla bug #162871, for one of several examples.) Good design is to hand that over to a common external subsystem shared by all applications. Which brings us to the second way in which this is wrong. A common Unix tradition is for all machines to have local proxy DNS servers. Linux operating systems have plenty of such server softwares that can be used: dnsmasq, pdnsd, PowerDNS, unbound ... One of the simplest, which does only caching and doesn't attempt to wear other hats simultaneously, is even named "dnscache". Set this up listening on the loopback interface, point its back-end at the relevant external servers, point the DNS client library at it, and -- voilà! -- a local caching proxy DNS server. * http://cr.yp.to/djbdns.html http://cr.yp.to/djbdns.html * http://cr.yp.to/djbdns/run-cache.html http://cr.yp.to/djbdns/run-cache.html * http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/djbdns-problems.html#wrong-icann-root http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/djb... * http://cr.yp.to/djbdns/install.html http://cr.yp.to/djbdns/install.html I run server machines that have applications that repeatedly look up the same domain names again and again. Each runs a local dnscache instance, which ameliorates this very well.

TheRealDunkirk 10y ago
I guess this goes to show that you can be a full-fledged expert in one development "world," and miss easy things in another. I see this a lot in the apparent expectations of people because of their ingrained presumptions of the OS platform. This article is of interest to HN because Mr. Craver is on a leading edge of .NET development and deployment, and this is an insightful oversight. I spent 20 years getting deep with PHP, and then Rails, on Linux, where, IMO, doing CI builds with something like Jenkins or Heroku is pretty straightforward. For the past couple years, I've been doing .NET, and, while I never really stopped doing ".NET" since the VB 3.0 days, I've had a lot to learn to get into doing serious, enterprise-level stuff. I just took my first steps in setting up auto-building with Visual Studio Team Services, and my experience has left me really disappointed. Dealing with changing variable values per environment is really hacky, no matter which of a handful of tricks you want to try. So much so, I left the whole thing hanging, and have gone back to doing releases by hand. Skimming the article, and seeing discussion about this topic in the comments, leads me to conclude that my experience wasn't just limited by my ignorance, and that this is still an area that is underserved in the .NET world. You'd think someone would have neatly sorted this by now. I keep looking for the oversight in MY case, but I'm not finding it.

streeter 10y ago

I was surprised they weren't using Kiln [https://www.fogcreek.com/kiln/ https://www.fogcreek.com/kiln/], a Fog Creek product. I know SO is independent from Fog Creek now, but still a bit surprised at it. I wonder if there was a migration off at some point.

Nick-Craver 10y ago
Yep...for the on-premise reasons listed in the article. Once upon a time a lot of projects were on Mercurial, hosted by Kiln. The Stack Overflow repo specifically has always been on an internal Mercurial and then Git server. Originally this was for speed, now it's for speed and reliability/dependency reduction.