7 ms·

It always puts me on edge when people choose to write their web-app in a particular language or framework 'because performance'. The C# VM is by no means slow b

by nfm 12y ago

It always puts me on edge when people choose to write their web-app in a particular language or framework 'because performance'. The C# VM is by no means slow but the fact that StackExchange is running off 9 front-end servers under pretty light load is testament to the fact that for regular web app workloads, it's just premature optimization to pull out the 'performance' card.

juliangregorian 12y ago
I wonder why they don't use more varnish or other cache servers instead. I've worked on a number of content-driven sites on the same scale as SO and that's a much more common pattern.
- patelsan 12y ago
  Maybe because most of their stack seems to be running on windows and varnish doesn't run on windows?
  - juliangregorian 12y ago
    So? They use haproxy. This also doesn't mean they can't use whatever the windows equivalent is; I simply don't know what that would be.
melling 12y ago
Maybe they only need so few servers because they are using a performant language like C#? Would they need a lot more servers if they used Rails, Django, or PHP? In other words, an interpreted language. I'm not so sure your conclusion can be derived from their stack.
- nfm 12y ago
  Yeah that's my point - if they were using a slower language they'd be clocking in at say 20 or 30 servers. Not remotely worth optimizing for saving that many servers at their scale.
  - juliangregorian 12y ago
    Not necessarily, because code is seldom the bottleneck in these types of scenarios, it's network, file IO, or DB transactions. Not to mention that most websites of this nature typically deploy some type of caching layer long before they reach 9 app servers.
  - alexchamberlain 12y ago
    Sorry if I misunderstood, but I don't believe they chose C# for its performance characteristics: I believe that they chose it as it was the best choice at the time; a lot has changed since the advent of Stack Overflow. Similar reason that a lot of Amazon is C and/or C++. I can't find any references for this, however.
    - codingthebeach 12y ago
      I don't think they would've chosen C# at all were it not for ASP.NET MVC. It was the MVC, more than anything else, that made lean-and-mean WISA stacks possible. C# with ASP.NET MVC under recent versions of Visual Studio is tooled to the Nth degree, performant, and yet and manages to stay clean. Not so under ASP.NET WebForms and even less so under classic ASP.
      - mattmanser 12y ago
        They used c# because Jeff Atwood was a c# developer. 8 or 9 years ago it was a lot less common to be effortlessly multilingual, changing stack was a lot of effort. Mainly because SO didn't exist! So you had to hunt through reams of text to find answers to simple questions, that now you google and the exact question has been asked on SO and answered. I can't emphasise how painful switching languages used to be. I suspect if asp.net MVC hadn't been out, it would have been written in webforms. I seem to remember asp.net MVC had only been out a couple of months when they started writing it or not even fully released, I was learning django at the time because MVC so was so obviously better than webforms. But because c# was so much faster than either ruby or python, when asp.net MVC turned up, and it was very, very good for a v1, I stuck with c#, as did many other .net programmers I suspect. At the time SO was written rails and django were still new and slowwwwww, php was king and still mainly written by page without frameworks, Java who knows, and the MVC revolution was just starting. I'm not sure what you think he would have written it in? I have a feeling if you hunt through coding horror you'll probably find a post from the SO start time period saying "use what you know". In Jeff Atwood's case that's C#. I know that Joel's company had written their own language that I think at the time was still in use, so they wouldn't use the fog creek stack. Which was still based on Microsoft stuff anyway.
andrea_s 12y ago
Well, the system is kind of over-dimensioned, but this is just good engineering, isn't it? If they are capable of rendering any question page in 33ms, it stands to reason they could use just 3 web servers and still serve content in roughly 100ms (which is fast enough that most people wouldn't notice). My point is, without knowing what is the bottleneck for the web servers (we only know it's not CPU - I suspect it should be I/O) it's hard to make this kind of point. Aside from that, I think I remember the original development team was already familiar with C#/.NET, and this drove the decision for the original stack.
- organsnyder 12y ago
  > If they are capable of rendering any question page in 33ms, it stands to reason they could use just 3 web servers and still serve content in roughly 100ms (which is fast enough that most people wouldn't notice). It doesn't work that way. With less servers, requests would still take 33ms (until the servers are overloaded, at which point they may start to take longer unless they have rate-limiting to reject requests above their capacity), but their overall capacity would be lower.
  - andrea_s 12y ago
    It definitely doesn't work that way, of course... That was meant to provide a reasonable upper bound for discussion. We know from the provided data that their architecture can compile and serve a page in about 33ms. This includes everything that is not web servers, so we are certain that the actual time spent in the webserver will be lower. Aside from that, we know that their webservers are nowhere near full CPU usage (they claim 15-20%). This also goes in the direction of not invalidating the use of 33ms as an upper bound. We would need to know more about why they have 9 webservers in the first place to get a better idea of what's going on (mostly how the I/O is behaving, I'd say). Putting this assumptions together, we can safely assume that by shrinking the number of webservers from 9 to 3 wouldn't cause the page loading time to increase to more than 100ms (and it's likely going to be much smaller in most conditions). Of course, this may be overlooking some piece of information we may be missing (the prime example still being I/O-related). edit: I see now that in the original post I stated "roughly" 100ms, while it would have been better to say "at maximum".
blakecaldwell 12y ago
I believe most pages are served almost entirely from cache. In those paths, the code should be doing nothing more than authenticating and talking with Redis. Language/stack performance becomes way more important for situations that rely on CPU.
revelation 12y ago
Nobody is optimizing anything or pulling out the performance card. C# was chosen because that is what they were comfortable with when building the site. It's not about optimization at all.
- nfm 12y ago
  Sorry, I don't think I was very clear. That's exactly what I meant. They didn't make a performance based decision when choosing their stack, and now that they're a huge site they're doing just fine - no need to go rewriting in the latest 'performant' stack.
saosebastiao 12y ago
This points me to the exact opposite conclusion. Why wouldn't I want to write my webapp in an efficient language and framework? If a tiny bit of extra effort means I can keep my app on a single EC2/RDS pair for a long enough to last until my first hire, I think that's worth it. It means I don't have to worry about setting up and tuning and maintaining load balancers, memcached servers, etc. Sysadmin/Devops work is a large enough distraction from the product that I would prefer not to have it at all. Efficient software frameworks/languages give me the breathing room I need to focus on the product.
- stouset 12y ago
  Any choice of language is going to satisfy your requirement of one app server prior to your first hire (for most everything not involving huge amounts of data processing). If your code performs poorly enough that this isn't the case, the problem isn't the language — it's you, and no language would have saved you. That said, there are so many benefits of running multiple servers behind a firewall off the bat, I can't fathom why people don't do this. It more or less enables trivial failover for many classes of problem, and deploys are no longer as large a source of instability: deploy by launching a new box, route a small amount of traffic to it at first, then route all of it. If there's an issue, route to the previous app servers. Compared to the time wasted by getting the architecture wrong off the bat (and months of pain while you wait for there to be enough time to do the work, which never happens), this is a massive win.

by nfm 12y ago

juliangregorian 12y ago

I wonder why they don't use more varnish or other cache servers instead. I've worked on a number of content-driven sites on the same scale as SO and that's a much more common pattern.

patelsan 12y ago
Maybe because most of their stack seems to be running on windows and varnish doesn't run on windows?
- juliangregorian 12y ago
  So? They use haproxy. This also doesn't mean they can't use whatever the windows equivalent is; I simply don't know what that would be.

melling 12y ago

Maybe they only need so few servers because they are using a performant language like C#? Would they need a lot more servers if they used Rails, Django, or PHP? In other words, an interpreted language. I'm not so sure your conclusion can be derived from their stack.

nfm 12y ago
Yeah that's my point - if they were using a slower language they'd be clocking in at say 20 or 30 servers. Not remotely worth optimizing for saving that many servers at their scale.
- juliangregorian 12y ago
  Not necessarily, because code is seldom the bottleneck in these types of scenarios, it's network, file IO, or DB transactions. Not to mention that most websites of this nature typically deploy some type of caching layer long before they reach 9 app servers.
- alexchamberlain 12y ago
  Sorry if I misunderstood, but I don't believe they chose C# for its performance characteristics: I believe that they chose it as it was the best choice at the time; a lot has changed since the advent of Stack Overflow. Similar reason that a lot of Amazon is C and/or C++. I can't find any references for this, however.
  - codingthebeach 12y ago
    I don't think they would've chosen C# at all were it not for ASP.NET MVC. It was the MVC, more than anything else, that made lean-and-mean WISA stacks possible. C# with ASP.NET MVC under recent versions of Visual Studio is tooled to the Nth degree, performant, and yet and manages to stay clean. Not so under ASP.NET WebForms and even less so under classic ASP.
    - mattmanser 12y ago
      They used c# because Jeff Atwood was a c# developer. 8 or 9 years ago it was a lot less common to be effortlessly multilingual, changing stack was a lot of effort. Mainly because SO didn't exist! So you had to hunt through reams of text to find answers to simple questions, that now you google and the exact question has been asked on SO and answered. I can't emphasise how painful switching languages used to be. I suspect if asp.net MVC hadn't been out, it would have been written in webforms. I seem to remember asp.net MVC had only been out a couple of months when they started writing it or not even fully released, I was learning django at the time because MVC so was so obviously better than webforms. But because c# was so much faster than either ruby or python, when asp.net MVC turned up, and it was very, very good for a v1, I stuck with c#, as did many other .net programmers I suspect. At the time SO was written rails and django were still new and slowwwwww, php was king and still mainly written by page without frameworks, Java who knows, and the MVC revolution was just starting. I'm not sure what you think he would have written it in? I have a feeling if you hunt through coding horror you'll probably find a post from the SO start time period saying "use what you know". In Jeff Atwood's case that's C#. I know that Joel's company had written their own language that I think at the time was still in use, so they wouldn't use the fog creek stack. Which was still based on Microsoft stuff anyway.

andrea_s 12y ago

Well, the system is kind of over-dimensioned, but this is just good engineering, isn't it? If they are capable of rendering any question page in 33ms, it stands to reason they could use just 3 web servers and still serve content in roughly 100ms (which is fast enough that most people wouldn't notice). My point is, without knowing what is the bottleneck for the web servers (we only know it's not CPU - I suspect it should be I/O) it's hard to make this kind of point. Aside from that, I think I remember the original development team was already familiar with C#/.NET, and this drove the decision for the original stack.

organsnyder 12y ago
> If they are capable of rendering any question page in 33ms, it stands to reason they could use just 3 web servers and still serve content in roughly 100ms (which is fast enough that most people wouldn't notice). It doesn't work that way. With less servers, requests would still take 33ms (until the servers are overloaded, at which point they may start to take longer unless they have rate-limiting to reject requests above their capacity), but their overall capacity would be lower.
- andrea_s 12y ago
  It definitely doesn't work that way, of course... That was meant to provide a reasonable upper bound for discussion. We know from the provided data that their architecture can compile and serve a page in about 33ms. This includes everything that is not web servers, so we are certain that the actual time spent in the webserver will be lower. Aside from that, we know that their webservers are nowhere near full CPU usage (they claim 15-20%). This also goes in the direction of not invalidating the use of 33ms as an upper bound. We would need to know more about why they have 9 webservers in the first place to get a better idea of what's going on (mostly how the I/O is behaving, I'd say). Putting this assumptions together, we can safely assume that by shrinking the number of webservers from 9 to 3 wouldn't cause the page loading time to increase to more than 100ms (and it's likely going to be much smaller in most conditions). Of course, this may be overlooking some piece of information we may be missing (the prime example still being I/O-related). edit: I see now that in the original post I stated "roughly" 100ms, while it would have been better to say "at maximum".

blakecaldwell 12y ago

I believe most pages are served almost entirely from cache. In those paths, the code should be doing nothing more than authenticating and talking with Redis. Language/stack performance becomes way more important for situations that rely on CPU.

revelation 12y ago

Nobody is optimizing anything or pulling out the performance card. C# was chosen because that is what they were comfortable with when building the site. It's not about optimization at all.

nfm 12y ago
Sorry, I don't think I was very clear. That's exactly what I meant. They didn't make a performance based decision when choosing their stack, and now that they're a huge site they're doing just fine - no need to go rewriting in the latest 'performant' stack.

saosebastiao 12y ago

This points me to the exact opposite conclusion. Why wouldn't I want to write my webapp in an efficient language and framework? If a tiny bit of extra effort means I can keep my app on a single EC2/RDS pair for a long enough to last until my first hire, I think that's worth it. It means I don't have to worry about setting up and tuning and maintaining load balancers, memcached servers, etc. Sysadmin/Devops work is a large enough distraction from the product that I would prefer not to have it at all. Efficient software frameworks/languages give me the breathing room I need to focus on the product.

stouset 12y ago
Any choice of language is going to satisfy your requirement of one app server prior to your first hire (for most everything not involving huge amounts of data processing). If your code performs poorly enough that this isn't the case, the problem isn't the language — it's you, and no language would have saved you. That said, there are so many benefits of running multiple servers behind a firewall off the bat, I can't fathom why people don't do this. It more or less enables trivial failover for many classes of problem, and deploys are no longer as large a source of instability: deploy by launching a new box, route a small amount of traffic to it at first, then route all of it. If there's an issue, route to the previous app servers. Compared to the time wasted by getting the architecture wrong off the bat (and months of pain while you wait for there to be enough time to do the work, which never happens), this is a massive win.