7 ms·

The architecture of Stack Overflow [video]

98 points by schmylan 13y ago

schmylan 13y ago
Before the title was moderated there was an important tidbit. StackOverflow doesn't unit-test. Fascinating.
- schmylan 13y ago
  tldw; He says he doesn't advocate it but they get away with it by having the community test it out for them in their meta site. Then the community writes up the bugs.
  - merak136 13y ago
    He actually says " I'm not advocating that you shouldn't put in tests. [ The reason we can get away with this ] is that we have a great community. " I take this to mean that he feels that StackOverflow doesn't need tests. Not that tests are useless.
    - kmontrose 13y ago
      That's an accurate read. - Stack Exchange employee
    - BrandonY 13y ago
      User community as testers presents some interesting pros and cons. Pros: * Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail. * Tests are unusually thorough * Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing. Cons: * Tests cannot be run offline. Feature must be committed and deployed before tests can be run. * Potentially large quantity of false positives (bad bug reports) * Potentially large quantity of false negatives (nobody notices particular bug, release considered good) * Does not work for non-user-visible features So basically you trade the reliability of your tests for a substantial build/release speedup. Some users experience each bug, but they are the users who are actively using the meta-community and have signed up to experience more bugs. Still, lack of pre-release unit testing must radically increase the importance of VERY careful code reviews. Not the decision I would have made, but definitely has the sorts of advantages that a small team of engineers drool would drool over.
      - sklivvz1971 13y ago
        Remember that our community writes bug reports but also vets bug reports. We rarely have to deal with bad reports. Interestingly, large quantities of false negatives are a non-issue.
        welegan 13y ago
        Presumably the same reason why they don't have a ton of bad questions on stack overflow: their community scoring would apply just as much to bug reports
carsongross 13y ago
The most important thing, technically, is having great developers who ship. For piths sake, I want to say "Everything else is noise" but that isn't true. Everything else can help or hurt, depending on the application and how doctrinaire the application of a given approach/methodology is, the organizational knock on effects (e.g. "Mr Tough Guy Testalot" holds up the release train or nukes your architecture to make it 'testable'), etc. but, seriously, "great developers who ship" is really what moves the needle.
- WestCoastJustin 13y ago
  Having a great Ops staff also helps ;) Of note is Thomas Limoncelli who wrote "The Practice of System and Network Administration" [1] and "Time Management for System Administrators" [2] works for Stack Exchange (formerly at Google). The Practice of System and Network Administration is basically the bible for most sysadmins, myself included. ps. I only singled Thomas Limoncelli out as an example just to highlight the caliber of their Ops staff. [1] http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668 http://www.amazon.com/Practice-System-Network-Administration... [2] http://www.amazon.com/Management-System-Administrators-Thomas-Limoncelli/dp/0596007833 http://www.amazon.com/Management-System-Administrators-Thoma...
  - carsongross 13y ago
    Violently agree.
    - skeletonjelly 13y ago
      Vehemently? Or do you want to punch someone?
      - carsongross 13y ago
        Violently. It's funnier.
- UK-AL 13y ago
esw 13y ago
Here are the slides for anyone who's interested: https://speakerdeck.com/sklivvz/the-architecture-of-stackoverflow-developer-conference-2013 https://speakerdeck.com/sklivvz/the-architecture-of-stackove...
merak136 13y ago
Some points that I find interesting: [1] StackOverflow has VERY FEW tests. He says that StackOverflow doesn't use many unit tests because of their active community and heavy usage of static code. [2] Most StackOverflow employees work remotely. This is very different than a lot of companies that are now trying to force employees back into an office. [3] Heavy usage of Static classes and methods. His main argument is that this gives them better performance than a more standard OO approach. [4] Caching even simple pages in order to avoid performance issues caused by garbage collection. [5] They don't worry about making a "Square Wheel". If their developers can write something more lightweight than an already developed alternative, they do! This is very different from the normal mindset of " don't reinvent the wheel ". [6] Always using multiple monitors. I love this. I feel like my productivity is nearly halved when I am working on one tiny screen. Overall, I was surprised at how few of the "norms" that they follow. Either way, seems like it could be a pretty cool place to work.
- tegeek 13y ago
  [7]. Millions of page views and just 25 servers for whole infrastructure, which includes everything including load balancing, cache, dbs' etc. etc. Seems very very optimized and cost effective. Its brilliant.
- thedufer 13y ago
  > StackOverflow employees work from home. Many do, but they have a fairly large office in NYC and a smaller one in London.
  - merak136 13y ago
    You are correct. I edited my post and actually found a good blog post on the subject. http://blog.stackoverflow.com/2013/02/why-we-still-believe-in-working-remotely/ http://blog.stackoverflow.com/2013/02/why-we-still-believe-i...
  - kmontrose 13y ago
    The Stack Overflow Q&A dev team has 2 people in New York, out of a team of 10 team. The Careers dev team is more New York heavy, 3 remote and 5 in New York. The sysadmin team is also quite remote, though I don't know the breakdown offhand. I believe at this point most new technical hires are remote. Our offices are mostly sales, Denver and London exclusively so.
y0ghur7_xxx 13y ago
I would love to know more about the Databases: - Are they used for different things on the sites? - Is data partitioned across tables? - Are they all SQL Server instances?
- zero1zero 13y ago
  I would like to know more about this as well. It sounds like they are all SQL Server instances. However, he made it seem like they are reproducing the schema once per site? I.e., a separate database per site rather than sharding the shared data to multiple hosts per site. Did I hear this right in the question/answer portion?
  - kmontrose 13y ago
    Stack Exchange has one database per-site, so Stack Overflow gets on, Super User gets one, Server Fault gets one, and so on. The schema for these is the same. There are a few wrinkles. There is one "network wide" database which has things like login credentials, and aggregated data (mostly exposed through stackexchange.com user profiles, or APIs). Careers Stack Overflow, stackexchange.com, and Area 51 all have their own unique database schema. All databases are MS SQL Server.
    - avemg 13y ago
      How do you manage schema changes with release deployments across across all of the databases that are meant to be standard?
      - sklivvz1971 13y ago
        All the schema changes are applied to all site databases at the same time. They need to be backwards compatible so, for example, if you need to rename a column - a worst case scenario - it's a multiple steps process: add a new column, add code which works with both columns, back fill the new column, change code so it works with the new column only, remove the old column.
notastartup 13y ago
is there an open source, self-hosted version of stack overflow that you can deploy on your own domain?
- robzienert 13y ago
  Yes. http://meta.stackoverflow.com/questions/2267/stack-overflow-clones http://meta.stackoverflow.com/questions/2267/stack-overflow-...
  - m_myers 13y ago
    To be clear: there is no version of the actual Stack Overflow code that is publicly available. There are, however, numerous open-source reimplementations of portions of the site code. Also (as the video perhaps mentioned), the Stack Overflow developers have often been able to spin off pieces of the code as open-source libraries. See http://blog.stackoverflow.com/2012/02/stack-exchange-open-source-projects/ http://blog.stackoverflow.com/2012/02/stack-exchange-open-so...
skittles 13y ago
He mentioned that they use the servicestack.text library. I've looked into servicestack recently (using the nuget packages), but then found the library to be pay-to-play. There's an older version (v3) that is BSD licensed that is being maintained. Do any of you have experience with it? I have grown tired of Microsoft pushing new solutions to the same problem (REST service with WCF and then Asp.net web api).
- sklivvz1971 13y ago
  We used it at the time I gave that talk, we don't anymore. We only used JSON serialization and we have rolled out our own free solution, Jil. https://github.com/kevin-montrose/Jil https://github.com/kevin-montrose/Jil
  - kmontrose 13y ago
    Technically we use Newtonsoft and Jil, Jil replacing Newtonsoft as we become increasingly confident in it. I wouldn't suggest anyone use Jil in a production role unless you're at Stack Overflow. It's too untested at the moment, and the typical person can't get me on the horn to fix whatever just broke.
    - guiomie 13y ago
      Why would I use Jil over Newtosoft ?
      - JasonPunyon 13y ago
        You wouldn't right now (Kevin doesn't recommend it). But in the end it you'll want to use it if JSON serialization is a performance bottleneck for you.
- dan_b 13y ago
  ServiceStack is just plain awesome when it comes to developing web services, though it's gone commercial for v4 onward. Nancy is another popular alternative - it's basically Sinatra for .Net. Every time I go back to WCF I want to stab myself in the face.
dlazerka 13y ago
I wouldn't trust Joel Spolsky's code expertise -- just look at Excel internals! Nevertheless, Stack Overflow is super cool. But that tells nothing about its architectural quality.
alexgartrell 13y ago
Dear any Stack Overflow Developers, Can you describe the network infrastructure in finer detail? Specifically what type of load balancer are you running? And what's peak RPS? Where are your network peaks? (I'm guessing major peak US Pacific and minor US Atlantic?)
- TacticalCoder 13y ago
  IIRC at first they had an entire Microsoft stack (I may be mistaken on that). But nowadays, from what I've read here on HN by SE devs in other threads, they're using lots and lots and lots of Linux: HAProxy, Redis, Nagios, etc. I just double-checked the slide and although I didn't notice it at first, you can see that 'HA Proxy' and 'Redis' are mentioned. The core Q&A is in C#/MS-SQL so that's probably not going to move to Linux anytime soon.
dimension64 13y ago
This might be a stackoverflow question, so what is a static code?

98 points by schmylan 13y ago

schmylan 13y ago

Before the title was moderated there was an important tidbit. StackOverflow doesn't unit-test. Fascinating.

schmylan 13y ago
tldw; He says he doesn't advocate it but they get away with it by having the community test it out for them in their meta site. Then the community writes up the bugs.
- merak136 13y ago
  He actually says " I'm not advocating that you shouldn't put in tests. [ The reason we can get away with this ] is that we have a great community. " I take this to mean that he feels that StackOverflow doesn't need tests. Not that tests are useless.
  - kmontrose 13y ago
    That's an accurate read. - Stack Exchange employee
  - BrandonY 13y ago
    User community as testers presents some interesting pros and cons. Pros: * Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail. * Tests are unusually thorough * Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing. Cons: * Tests cannot be run offline. Feature must be committed and deployed before tests can be run. * Potentially large quantity of false positives (bad bug reports) * Potentially large quantity of false negatives (nobody notices particular bug, release considered good) * Does not work for non-user-visible features So basically you trade the reliability of your tests for a substantial build/release speedup. Some users experience each bug, but they are the users who are actively using the meta-community and have signed up to experience more bugs. Still, lack of pre-release unit testing must radically increase the importance of VERY careful code reviews. Not the decision I would have made, but definitely has the sorts of advantages that a small team of engineers drool would drool over.
    - sklivvz1971 13y ago
      Remember that our community writes bug reports but also vets bug reports. We rarely have to deal with bad reports. Interestingly, large quantities of false negatives are a non-issue.
      - welegan 13y ago
        Presumably the same reason why they don't have a ton of bad questions on stack overflow: their community scoring would apply just as much to bug reports

carsongross 13y ago

The most important thing, technically, is having great developers who ship. For piths sake, I want to say "Everything else is noise" but that isn't true. Everything else can help or hurt, depending on the application and how doctrinaire the application of a given approach/methodology is, the organizational knock on effects (e.g. "Mr Tough Guy Testalot" holds up the release train or nukes your architecture to make it 'testable'), etc. but, seriously, "great developers who ship" is really what moves the needle.

WestCoastJustin 13y ago
Having a great Ops staff also helps ;) Of note is Thomas Limoncelli who wrote "The Practice of System and Network Administration" [1] and "Time Management for System Administrators" [2] works for Stack Exchange (formerly at Google). The Practice of System and Network Administration is basically the bible for most sysadmins, myself included. ps. I only singled Thomas Limoncelli out as an example just to highlight the caliber of their Ops staff. [1] http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668 http://www.amazon.com/Practice-System-Network-Administration... [2] http://www.amazon.com/Management-System-Administrators-Thomas-Limoncelli/dp/0596007833 http://www.amazon.com/Management-System-Administrators-Thoma...
- carsongross 13y ago
  Violently agree.
  - skeletonjelly 13y ago
    Vehemently? Or do you want to punch someone?
    - carsongross 13y ago
      Violently. It's funnier.
UK-AL 13y ago

esw 13y ago

Here are the slides for anyone who's interested: https://speakerdeck.com/sklivvz/the-architecture-of-stackoverflow-developer-conference-2013 https://speakerdeck.com/sklivvz/the-architecture-of-stackove...

merak136 13y ago

Some points that I find interesting: [1] StackOverflow has VERY FEW tests. He says that StackOverflow doesn't use many unit tests because of their active community and heavy usage of static code. [2] Most StackOverflow employees work remotely. This is very different than a lot of companies that are now trying to force employees back into an office. [3] Heavy usage of Static classes and methods. His main argument is that this gives them better performance than a more standard OO approach. [4] Caching even simple pages in order to avoid performance issues caused by garbage collection. [5] They don't worry about making a "Square Wheel". If their developers can write something more lightweight than an already developed alternative, they do! This is very different from the normal mindset of " don't reinvent the wheel ". [6] Always using multiple monitors. I love this. I feel like my productivity is nearly halved when I am working on one tiny screen. Overall, I was surprised at how few of the "norms" that they follow. Either way, seems like it could be a pretty cool place to work.

tegeek 13y ago
[7]. Millions of page views and just 25 servers for whole infrastructure, which includes everything including load balancing, cache, dbs' etc. etc. Seems very very optimized and cost effective. Its brilliant.
thedufer 13y ago
> StackOverflow employees work from home. Many do, but they have a fairly large office in NYC and a smaller one in London.
- merak136 13y ago
  You are correct. I edited my post and actually found a good blog post on the subject. http://blog.stackoverflow.com/2013/02/why-we-still-believe-in-working-remotely/ http://blog.stackoverflow.com/2013/02/why-we-still-believe-i...
- kmontrose 13y ago
  The Stack Overflow Q&A dev team has 2 people in New York, out of a team of 10 team. The Careers dev team is more New York heavy, 3 remote and 5 in New York. The sysadmin team is also quite remote, though I don't know the breakdown offhand. I believe at this point most new technical hires are remote. Our offices are mostly sales, Denver and London exclusively so.

y0ghur7_xxx 13y ago

I would love to know more about the Databases: - Are they used for different things on the sites? - Is data partitioned across tables? - Are they all SQL Server instances?

zero1zero 13y ago
I would like to know more about this as well. It sounds like they are all SQL Server instances. However, he made it seem like they are reproducing the schema once per site? I.e., a separate database per site rather than sharding the shared data to multiple hosts per site. Did I hear this right in the question/answer portion?
- kmontrose 13y ago
  Stack Exchange has one database per-site, so Stack Overflow gets on, Super User gets one, Server Fault gets one, and so on. The schema for these is the same. There are a few wrinkles. There is one "network wide" database which has things like login credentials, and aggregated data (mostly exposed through stackexchange.com user profiles, or APIs). Careers Stack Overflow, stackexchange.com, and Area 51 all have their own unique database schema. All databases are MS SQL Server.
  - avemg 13y ago
    How do you manage schema changes with release deployments across across all of the databases that are meant to be standard?
    - sklivvz1971 13y ago
      All the schema changes are applied to all site databases at the same time. They need to be backwards compatible so, for example, if you need to rename a column - a worst case scenario - it's a multiple steps process: add a new column, add code which works with both columns, back fill the new column, change code so it works with the new column only, remove the old column.

notastartup 13y ago

is there an open source, self-hosted version of stack overflow that you can deploy on your own domain?

robzienert 13y ago
Yes. http://meta.stackoverflow.com/questions/2267/stack-overflow-clones http://meta.stackoverflow.com/questions/2267/stack-overflow-...
- m_myers 13y ago
  To be clear: there is no version of the actual Stack Overflow code that is publicly available. There are, however, numerous open-source reimplementations of portions of the site code. Also (as the video perhaps mentioned), the Stack Overflow developers have often been able to spin off pieces of the code as open-source libraries. See http://blog.stackoverflow.com/2012/02/stack-exchange-open-source-projects/ http://blog.stackoverflow.com/2012/02/stack-exchange-open-so...

skittles 13y ago

He mentioned that they use the servicestack.text library. I've looked into servicestack recently (using the nuget packages), but then found the library to be pay-to-play. There's an older version (v3) that is BSD licensed that is being maintained. Do any of you have experience with it? I have grown tired of Microsoft pushing new solutions to the same problem (REST service with WCF and then Asp.net web api).

sklivvz1971 13y ago
We used it at the time I gave that talk, we don't anymore. We only used JSON serialization and we have rolled out our own free solution, Jil. https://github.com/kevin-montrose/Jil https://github.com/kevin-montrose/Jil
- kmontrose 13y ago
  Technically we use Newtonsoft and Jil, Jil replacing Newtonsoft as we become increasingly confident in it. I wouldn't suggest anyone use Jil in a production role unless you're at Stack Overflow. It's too untested at the moment, and the typical person can't get me on the horn to fix whatever just broke.
  - guiomie 13y ago
    Why would I use Jil over Newtosoft ?
    - JasonPunyon 13y ago
      You wouldn't right now (Kevin doesn't recommend it). But in the end it you'll want to use it if JSON serialization is a performance bottleneck for you.
dan_b 13y ago
ServiceStack is just plain awesome when it comes to developing web services, though it's gone commercial for v4 onward. Nancy is another popular alternative - it's basically Sinatra for .Net. Every time I go back to WCF I want to stab myself in the face.

dlazerka 13y ago

I wouldn't trust Joel Spolsky's code expertise -- just look at Excel internals! Nevertheless, Stack Overflow is super cool. But that tells nothing about its architectural quality.

alexgartrell 13y ago

Dear any Stack Overflow Developers, Can you describe the network infrastructure in finer detail? Specifically what type of load balancer are you running? And what's peak RPS? Where are your network peaks? (I'm guessing major peak US Pacific and minor US Atlantic?)

TacticalCoder 13y ago
IIRC at first they had an entire Microsoft stack (I may be mistaken on that). But nowadays, from what I've read here on HN by SE devs in other threads, they're using lots and lots and lots of Linux: HAProxy, Redis, Nagios, etc. I just double-checked the slide and although I didn't notice it at first, you can see that 'HA Proxy' and 'Redis' are mentioned. The core Q&A is in C#/MS-SQL so that's probably not going to move to Linux anytime soon.

dimension64 13y ago

This might be a stackoverflow question, so what is a static code?