44 ms·

StackOverflow Update: 560M Pageviews a Month, 25 Servers

479 points by quicksilver03 12y ago

jonhmchan 12y ago
Relatively new stack dev here. I came in on the other side of the fence of a lot of these technologies (bread and butter is Python in Flask with Mongo on Heroku on a mac) but since I started here, I've been constantly and pleasantly surprised by how performant everything here has been despite my biases. It's mighty fun.
chiph 12y ago
I've recently started using their micro-ORM Dapper, and I like it a lot. I get the performance of hand-coded SQL, but without the tedious mapping from SqlDataReader to my entity.
- Aaronontheweb 12y ago
  Dapper is great - I promptly dumped NHibernate and Entity Framework for it. It's the only ORM I've found that makes it trivially easy (as it should be) to use stored procedures and views in queries.
  - atonse 12y ago
    +1 for Dapper - I'm a big fan. It is very fast and we use it to serve our live (mostly read-only) traffic, while using something like Entity Framework for the backend Administrative UI, which has a lot of inserts and updates.
    - chiph 12y ago
      There's a couple of user-contrib projects that do similar things for inserts & updates. Dapper Extensions is the one I'm using. https://github.com/tmsmith/Dapper-Extensions https://github.com/tmsmith/Dapper-Extensions
      - Aaronontheweb 12y ago
        I wrote a Dapper extension for working with SQL Server's geospatial queries and types a couple of years ago - have they added anything like that yet? Otherwise I'd be happy to add it.
        Nick-Craver 12y ago
        Marc Gravell added in the ability to put in pretty much any custom type without adding any dependency weight about a month ago. You can see the commit here: https://github.com/StackExchange/dapper-dot-net/commit/e26ee0abe5bdff561ff59fded87cabb9f5d983a1 https://github.com/StackExchange/dapper-dot-net/commit/e26ee... Look towards the end at the tests for example of how to hook up a custom type (it's pretty simple).
        Aaronontheweb 12y ago
        Cool - the one I wrote was years ago. Looks like it's not necessary any more.
- teh_klev 12y ago
  I've been using Dapper for around three years, "it just works" and is very fast. I've been around the block a few times with ORM's - LINQ to SQL, NHibernate, Entity Framework, Massive, SubSonic. For small to medium size projects there's no contest, provided you don't mind hand crafting your SQL. Hat's off to Marc, Sam and team for building this.
- Akkuma 12y ago
  Another awesome one which I've used on a previous project is SqlFu, https://github.com/sapiens/SqlFu https://github.com/sapiens/SqlFu
- pbz 12y ago
  Dapper is nice, but if you want to have the compiler check your queries, and still run fast, I recommend LINQ to DB: https://github.com/linq2db/linq2db https://github.com/linq2db/linq2db (not to be confused with LINQ to SQL from Microsoft)
  - Nick-Craver 12y ago
    To be fair, the compiler is checking against what you told it the database looks like. This can help with lots of things, but still has the fundamental disconnect problem you'll get at runtime with either approach. A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example we (and by "we" I mean, "I made Marc Gravell do it") added literal replacement to Dapper to help with query parameterization which allows you to use things like filtered indexes. In dapper we also intercept the SQL calls to dapper and add add exactly where it came from. Here's what that looks like, replicate for the other methods: public static IEnumerable<T> Query<T>(this DataContext db, SqlBuilder.Template template, bool buffered = true, int? commandTimeout = null, IDbTransaction transaction = null, [CallerFilePath]string fromFile = null, [CallerLineNumber]int onLine = 0, string comment = null) { using (db.Connection.EnsureOpen()) { return SqlMapper.Query<T>(db.Connection, MarkSqlString(template.RawSql, fromFile, onLine, comment), template.Parameters as object, transaction ?? db.Transaction, buffered, commandTimeout); } } And here's the our marking method tailored for our code, but you get the idea: private static string MarkSqlString(string sql, string path, int lineNumber, string comment) { if (path.IsNullOrEmpty() || lineNumber == 0) { return sql; } var commentWrap = " "; var i = sql.IndexOf(Environment.NewLine); // if we didn't find \n, or it was the very end, go to the first space method if (i < 0 || i == sql.Length - 1) { i = sql.IndexOf(' '); commentWrap = Environment.NewLine; } if (i < 0) return sql; // Grab one directory and the file name worth of the path // this dodges problems with the build server using temp dirs // but also gives us enough info to uniquely identify a queries location var split = path.LastIndexOf('\\') - 1; if (split < 0) return sql; split = path.LastIndexOf('\\', split); if (split < 0) return sql; split++; // just for Craver var sqlComment = " /* " + path.Substring(split) + "@" + lineNumber + (comment.HasValue() ? " - " + comment : "") + " */" + commentWrap; return sql.Substring(0, i) + sqlComment + sql.Substring(i); } This results in a comment at the top of the query like this: /* Models\Post.LinkedQuestions.cs@105 */ select top (@top) p.Id , p.Title , p.Score It's such a simple approach using the caller member attributes, but it saves so much time tracking things down.
- j_s 12y ago
  Another library worth mentioning in this space is Massive, which uses .NET dynamics a lot (focusing slightly less on performance than ease of use). https://github.com/robconery/massive https://github.com/robconery/massive
lemcoe9 12y ago
Makes me wonder why smaller website teams need dozens of engineers to keep their infinitesimally smaller app running.
- edgyswingset 12y ago
  Rigid adherence to certain technologies could be a culprit. "We're a Ruby shop!" is fine for problems where Ruby is great, but the limitations of the technology are bound to creep up somewhere. Not using a polyglot approach would be bad.
  - nostrademons 12y ago
    It can also be caused by too much embrace of polyglot tools. "We use the best tools for the job" can easily mean an architecture with Redis, Memcached, Mongo, MySQL, Angular, JQuery, Node, and several dozen third-party libraries, all glued together with Thrift or JSON-RPC. If all you're doing is a messaging app, you might be much better off with a single in-process Java app. Usually you want your solution to be just right for the problem; solve the essential complexity but don't introduce any accidental complexity.
    - swah 12y ago
      I wish more people considered this alternative when recommending the usage of queues etc. I always remember of this post: http://williamedwardscoder.tumblr.com/post/18065079081/cogs-bad http://williamedwardscoder.tumblr.com/post/18065079081/cogs-...
      - nostrademons 12y ago
        Hacker News also runs on a single box using only in-language hashtables and lists. ViaWeb did too, and PlentyOfFish.
- daigoba66 12y ago
  I personally think it's due to overly complex applications and systems with many moving (and breaking) parts. Also StackOverflow often practices "scale-up" instead of "scale-out".
tomblomfield 12y ago
"One problem is not many tests. Tests aren’t needed because there’s a great community... If users find any problems with it they report the bugs that they’ve found." I'm often surprised at the paucity of test-coverage in relatively large companies. http://nerds.airbnb.com/testing-at-airbnb/ http://nerds.airbnb.com/testing-at-airbnb/
- aturek 12y ago
  That bullet point surprised me. It comes right after "110K lines of code. A small number given what it does"; to me this reads like they didn't write tests because that would add complexity. Can any SO devs give us more details on "not many tests"? Or how many bug reports get filed vs rate of change of software?
  - jc4p 12y ago
    The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange). You can look at reported bugs here: http://meta.stackexchange.com/questions/tagged/bug http://meta.stackexchange.com/questions/tagged/bug -- it's pretty frequent, at least one every couple of hours or so. The last commit in our "Tests" project was 14 days ago, where as the last commit in the code base was 4 minutes ago.
    - jacques_chester 12y ago
      You should come down our way and do a tech talk about it. If only for the wailing, gnashing of teeth and rending of garments.
      - jc4p 12y ago
        Your people would literally murder me. Are you back from west coast / toronto?
habosa 12y ago
> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.
- x1798DE 12y ago
  It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive. I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system. 1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#physical_memory_limits_windows_server_2012 http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...
  - mrb 12y ago
    $4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160. Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months. A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).
    - ZachPruckowski 12y ago
      Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.
programminggeek 12y ago
One smart thing they are doing is putting different sites on different databases. It effectively as a kind of partitioning to allow for horizontal scaling if needed. If nothing else, it keeps less data in each table, so queries should be faster due to smaller datasets. Sounds like they know what they are doing.
- beachstartup 12y ago
  beware that you can over-optimized this. for example, do not put every single customer of an saas solution into a separate database.
  - csharrison 12y ago
    I'm curious, why not? If your saas was small enough and had few enough customers with large data needs, separating them each out into a separate database seems like a viable solution.
    - beachstartup 12y ago
      yeah, if you have an experienced team, you can easily refactor your one-customer-per-db model into something that will handle 1,000 new customers signing up every month. given a good enough team, you can even do this while doing other sort-of important things like customer support, bug fixes, and new development. however, inexperienced developers and teams are not good enough, or fast enough, to do that, and end up painting themselves into a corner when they have thousands of live customers to support and need to change their entire application architecture + database schema when their backup, replication, and housekeeping tasks choke on 1,000+ databases. and oftentimes, one-customer-per-db also means one-instance-of-application-per-customer, another anti pattern to avoid. we see this kind of stuff ALL the time. it happens a lot - people make terrible decisions and then are stuck with them 5 years down the line and are looking at a monumental cost to redo. not everyone is experience enough to work their way out of an awful situation like that.
UVB-76 12y ago
Anyone else not quite expecting StackExchange to be using a Microsoft stack?
- ufo 12y ago
  Anyone who knows Joel Spolsky would expect a MS stack :)
  - Alupis 12y ago
    It's difficult to take pragmatic advice from someone who took the least pragmatic (and most expensive) route for a startup web company.
    - rstupek 12y ago
      I'm not sure I agree. Pragmatic choices based on using what you know and not just choosing a toolset because everyone else is using it.
    - WorldWideWayne 12y ago
      Why does everyone in Silicon Valley run with expensive Apple hardware when Linux can run on any CPU?
    - Amezarak 12y ago
      If you have extensive Microsoft experience and take advantage of something like BizSpark* (although one suspects with their connections they wouldn't have needed that) it's hard to see how it wouldn't be plenty pragmatic and inexpensive. * Currently three years of free licenses, all software you download during that time is free forever, and discounted MSDN subs thereafter.
    - jacquesm 12y ago
      Programmer time is a factor that comes into play when you build something and re-training your people to use a stack they are not familiar with + the associated mistakes can get expensive in a hurry. If you happen to be familiar with the MS stack and are able to factor the license fees into your business plan there is no strong reason against and many for using that particular stack. MS programmers tend to be a bit more expensive than people working with one of the many Linux offerings but that's not such a huge difference that it would become the deciding factor and if anything the MS stack is more performant than the Linux one on the same hardware (and I write that as an anti-MS guy, I strongly believe that politics and tech should be kept separate when it comes to discussing systems relative to each other). So even if there are lots of reasons why I would not have made that particular choice I can completely understand why the SO people made their choice the other way.
tjdetwiler 12y ago
Can anyone provide context on "Heavy usage of static classes and methods, for simplicity and better performance." Is performance _really_ an issue here?
- Griever 12y ago
  I could be mistaken, but I think this is in regards to that post Sam Saffron wrote a few years back about getting SO to do Gen.2 garbage collection less often, and using static classes and structs were one of the ways that they fixed it.
- MichaelGG 12y ago
  I find the opposition to static classes odd. Unless you need the data encapsulation of an object, why force people to create an object to call functions? I find too much C# code loves creating objects for no real purpose other than that's how it works. For SO, if we guess they max out around 2000 req/sec, then if a bunch of objects are being allocated for each request, there could be added GC pressure. From my own experience developing much higher-scale managed code, allocating objects is a real pain and people will take a lot of effort to avoid it.
  - Griever 12y ago
    I think the real issue (although it doesn't seem to be an issue for SO) is testability. AFAIK, static classes cannot be mocked, and you are pretty much boxing yourself into only doing integration tests for everything. Not necessarily a bad way to go, just lots more setup per test.
    - MrBuddyCasino 12y ago
      Yep, that used to be one of the main reasons to have DI in Java. But now that everything (even static methods) can be mocked, that reason is no longer valid. Is this not possible in .NET?
    - mattmanser 12y ago
      It completely depends if you do this: public static decimal DiscountValue(Order order) Or this: public static decimal DiscountValue(decimal orderValue, decimal orderDiscount) Or even this, which would farm the work out to SQL: public static decimal DiscountValue(int orderId) The first would require a mock, the 2nd wouldn't and the third isn't unit testable. It's a bit of a contrived example, as you're more likely to do the third method if you're trying to get some sort of summary data about an order out which would mean you don't really need/want to load the whole order object.
nmjohn 12y ago
> AWS would cost a fortune. Isn't that comparing apples to oranges? Taking any application built for large servers and putting the same application onto a cloud-based architecture will be more expensive. Cloud-based architecture requires ground up differences in how the application is built. Now whether or not it is better to use a cloud-based approach or traditional bare metal is highly subjective and isn't my point.
- incision 12y ago
  >'Isn't that comparing apples to oranges?' On another site, in another context, it probably would be, but here it's really presented as a contrast of scale-up versus scale-out - something the regular audience of highscalability will certainly grok. In context... 'Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t been a problem because they’ve been quite successful at sizing their system correctly.' It's an acknowledgement of their relatively unique strategy and the short list of caveats that make it possible.
sz4kerto 12y ago
Extremely interesting read, especially because it goes against many fashionable engineering practices. They see (correctly) that abstractions, DI have their own compromises. Also, they're not religious (about technology), that's also quite rare.
kbenson 12y ago
> The cost of inefficient code can be higher than you think. Efficient code stretches hardware further, reduces power usage, makes code easier for programmers to understand. I'm curious what the reasoning is for "Efficient code ... makes code easier for programmers to understand". To my mind, efficient code (in this case, I assume coding to the hardware, as they mention elsewhere), has many benefits, but making it easier to understand is not one of them. A useful comment by the code may help, but that's not a result of efficient coding, that's a result of good commenting practice.
- hueving 12y ago
  Definitely. Some of the most efficient C code I have seen is incomprehensible to anyone but a C expert and even then it takes several minutes to understand all of the nuances of what might only be a 4-line function.
  - mkr-hn 12y ago
    I experienced exactly this when I cracked open the source to a GNU tool to see if I could figure it out.
- itry 12y ago
  users=db.get("SELECT name FROM users WHERE group='admin'") is more efficient and understandable then: users=[]; tmp=db.get("SELECT name,group FROM users") for (i=0;i<tmp.count();i++) { if (tmp[i]['group']=='admin') users[]=tmp[i][name]; }
  - CodeCube 12y ago
    This is where a good abstraction layer can pay dividends. Of course, a bad abstraction layer (as evidenced by Spolsky's "leaky abstraction" post (http://www.joelonsoftware.com/articles/LeakyAbstractions.html) http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) on the opposite hand can add immeasurable cost.
  - abritishguy 12y ago
lnanek2 12y ago
Really hate the separate sites thing. I use a half dozen of them and they are computer related, so it is a pain in the ass always having to register and not being allowed to comment/answer at first, etc.. They have a few improvements now like importing your profile from other sites, but they shouldn't even have so many separate sites in the first place, just a tag or a category or something.
- recursive 12y ago
  You get +100 rep bonus for linking your accounts, so you shouldn't have a problem being allowed to comment.
  - JelteF 12y ago
    You do need to have more than 200 rep on one site though.
    - r00fus 12y ago
      Which one? I'm a full member on all the StackExchange sites I care about for now with the +100 cross-membership bonus.
      - iancarroll 12y ago
        In order to get the +100 bonus on all sites, you must have one profile on the network with >200 reputation.
  - rayshan 12y ago
    How? I can't seem to find this option in account admin panels.
    - zo1 12y ago
      It happens automatically, if I remember correctly.
    - 12y ago
epsteinbargraph 12y ago
Soooo... a average of 200 pageviews a second - I suppose that doesn't sound as cool. Or, to put it another way; 1 server per 8 pageviews per second Does not seem that efficient when you put it like that. Even a resource hog like a wordpress install can eclipse that level. *made correction- still not impressed
- NietTim 12y ago
  Well yeah, but these aren't just plain old text file of course. Database queries, assets, async clients, the whole lot.
- jre 12y ago
  I think you're missing a 0 somewhere : 560000000 / (60 * 60 * 24 * 31) = 209
  - deleted 12y ago
    [deleted]
- cruise02 12y ago
  You're off by a factor of 10. I have it at just over 200 page views per second. Also, that's the average load over a month. The same hardware needs to be able to handle peak load.
  - tiernano 12y ago
    and they also mention that they could possibly run on a lot less hardware... 7-8 machines + network gear (7 if the load balancer is an appliance, 8 if its a server).
MichaelGG 12y ago
So this is what, a 2000 request/sec peak? Over 11 servers, that's like 200 requests/sec peak per frontend? The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their transactions in RAM and keep it there (replicated). I know that's what VoltDB is counting on.
- sklivvz1971 12y ago
  Yes, the idea is to scale up as much as economically makes sense and only then scale out, but due to Moore's law we're still scaling up. Not everyone has google-like problems that are betters solved by a battery of cheap boxes.
  - threeseed 12y ago
    Actually most of the websites today are better solved by a battery of cheap boxes. You simply put a load balancer (Nginx/ELB/HAProxy etc) in front of a fleet of smaller web/application servers that dynamically scale depending on traffic. That way it is cost effective, far more reliable, easier to scale and you can tolerate DC outages better.
    - mwsherman 12y ago
      As with most systems, the data store is the hard thing to scale out. That requires intelligence at the application level, and that might include CAP issues. So for us, the SQL data store is the real “up” part of the equation. We have a fair amount of headroom there, so if we can keep sharding (and other data strategies) out of the codebase, so much the better. (Load-balancing HTTP requests “out” is not a big deal and we are doing that.)
    - sklivvz1971 12y ago
      Marco/Stack Overflow here. Simply put, this is not true at all. Most of the websites today run happily on a single box and the choice of architecture is frankly irrelevant because of the abysmal amount of traffic they get. Large, and by large I mean top 100 sites, are large and typically complex systems. Each one is different and bespoke. Making broad generalizations doesn't really work, because typically they have parts that need to be scaled up, parts that need to be scaled out, etc. Finally, if you read the article you would know that we already use a load balancer with multiple web servers. What we are discussing here is whether it's better to have 100s of cheap/cloud boxes versus a few big ones. In our use case (we want fast response times) the second solution is demonstrably better.
josephscott 12y ago
"with just 25 servers" This really needs to stop. Go to stackexchange.com and you'll find that more than half of the HTTP requests are to cdn.sstatic.net. Looking up the IP addresses for cdn.sstatic.net returned five entries for me, all owned by CloudFlare. None of the CloudFlare servers that they are using seem to be in that 25 count. Sure, these are all for static assets, that isn't the point. There are way more than 25 servers being used to serve the StackOverflow sites.
- tiernano 12y ago
  Stack Exchange dont own their CDN. If you really start couting external servers, you would be adding DNS, possible client side proxy servers, client side network infrastructure, etc... these are things that OTHER people do, not SO.
  - duderific 12y ago
    I would say though, CDN doesn't fall in the same category as DNS. Serving static assets off other networks does take a huge load off the SE infrastructure and should probably be mentioned in the article.
    - mschuster91 12y ago
      Only in terms of bandwidth, though. The CPU load by processing is the interesting figure, not side noise by images/js/css.
- sklivvz1971 12y ago
  It's how many servers you would need to provision to do the same, with the same code base. It's basically the number of directly active servers our code runs on. Sure, we use a CDN, but not for CPU or I/O load, but to make you find your answers faster.
- kondro 12y ago
  Sort of true, but static assets are also trivial to serve. What they get by using CloudFlare is getting their assets as close to end users as possible to reduce latency. This isn't a complex problem to solve (for a single site), just a really expensive one that very few single app companies could ever justify solving themselves. I believe Google and Amazon are the only ones with their own significant CDN infrastructure, everyone else outsources it.
Nux 12y ago
"Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there’s no compelling reason to change." <- there's your problem :-)
- serge2k 12y ago
  54th ranked site in the world with a small number of servers and apparently very maintainable and stable systems. I would love that problem.
rakoo 12y ago
What I'd like to see is how much cache hit they add on each level of cache. I remember some presentation of the facebook images stack where less than 10% (if I remember well) of the requests actually hit the disks; it would be interesting to see the patterns for the whole SE galaxy.
- sklivvz1971 12y ago
  It totally depends on what we are caching (e.g. the community bullettins or hot meta questions get a gazillion hits...)
- skeletonjelly 12y ago
  As someone mentioned above, SQL Server hits memory first, and as they have 384GB of memory, most of the requests would sit there. That's just on the DB server. In the linked article it all authenticated requests hit the DB, with anon users getting a cached copy.
Perceptes 12y ago
I'm grateful to them them for being so transparent about how their systems work. It's rare to get this level of detail on how a large, successful site operates.
phkahler 12y ago
I haven't been able to use SO since they shut off myopenid last year. I can't log in, so can't change my credentials ;-( Any ideas HN?
- GeorgeBeech 12y ago
  Email team@stackexchange.com they should be able to help you out. (I'm an SE SysAdmin )
  - phkahler 12y ago
    Thanks, I followed the recovery link provided by another SE person.
- jseutter 12y ago
  http://meta.stackexchange.com/questions/115223/my-openid-provider-already-shut-down-is-there-any-way-for-me-to-access-my-accou http://meta.stackexchange.com/questions/115223/my-openid-pro...
- Shog9 12y ago
  You can email us (as George said) or you can just punch your email address into the account recovery tool and get an automated email telling you what to click within the space of a few minutes: http://stackoverflow.com/users/account-recovery http://stackoverflow.com/users/account-recovery We... probably need to make that link a bit more obvious.
  - phkahler 12y ago
    Thanks, that was easy. So now I just have a password instead of linking to another account @facebook or google+ I guess that works ;-)
AlisdairO 12y ago
Stack Overflow is the example I'm forever using when arguing against premature scale-out. For non-trivial applications scale-out has substantial complexity costs attached to it, and the overwhelming majority of applications will never truly need it. It's frustrating to see time wasted obsessing over trying to maintain eventual consistency (or chasing bugs when you failed to do so) on systems that could quite happily run on a single, not that beefy machine.
- samstave 12y ago
  >For non-trivial applications scale-out has substantial complexity costs attached to it Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out. From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of their forum is not an extraordinarily complex problem.
  - AlisdairO 12y ago
    By nontrivial I mean 'application which, in your document database, requires multi-document updates to perform some individual logical operations'. This is a rather low bar :-). The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course there are real reasons to use scale out - I simply advocate thinking hard about whether your problem will ever truly need it before taking the substantial complexity hit of coding for it.
    - AlisdairO 12y ago
      Er, I don't seem to be able to edit, but I guess I should specify that I'm really talking about scale-out of writes here. Read scalability is an obviously easier problem..
ternaryoperator 12y ago
"SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods." I don't understand this at all. What does TDD have to do with reducing garbage collection?
- numbsafari 12y ago
  My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.
  - bjourne 12y ago
    IME that always happens when you try to perform tdd in combination with Javaesque encapsulation. The good solution to the problem is to not be so afraid of classes seeing each others internals. The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.
    - mistermann 12y ago
      > The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable. Not doing this will cause much self-righteous snickering from some.
    - edwinnathaniel 12y ago
      You do realize that most efficient DI framework only inject the dependency once. Separating Controller, Repository, and Services are good practices as well and let's be honest, we're looking at 3 methods layer at most. Here's what happened in Java: 1. When you deploy your WAR, the DI will inject necessary component _once_ (and these components only instantiated _once_ for the whole web-app so there you go, Singleton without hardcoding). 2. A request comes in, gets processed by the pumped-up Servlet (already DI-ed). If the servlet has not been instantiated, it will be instantiated _once_ and the instance of the Servlet is kept in memory. 3. Another request comes in, gets processed by the same pumped-up Servlet that is already instantiated and has already been injected with the component (no more injection, no more instantiation, no more Object to create...) So I've got to ask this question: what GC problem we're trying to solve here? Some of the static methods are understandable but if Component A requires Component B and both of them have already been instantiated _once_ and have been properly wired up together, we have 2 Objects for the lifetime of the application. I'd pay for a wee bit extra hardware for the cost of maintainable code ;) Discipline (and knowledge of object graph) definitely help to reach to that point.
guiomie 12y ago
Can someone help me understand what they mean by: "Garbage collection driven programming. SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods. While extreme, the result is highly performing code."
- louthy 12y ago
  I assume it means using C# more as a functional language than an OO language, so that most garbage collection is of short-lifetime objects and therefore cheaper to collect (generation 0).
  - virmundi 12y ago
    I take that statement to mean C# as C. Functional languages are terrible for garbage collection. Static methods are great since they are static, loaded once and probably in-lineable by the compiler/JIT.
    - thomasz 12y ago
      The usual wisdom in clr-land is that the generational garbage collector can cope just fine with a lot of small, short lived objects. It absolutely doesn't like big, long lived objects that reach G1, G2 or the large object heap and must be traversed for a full gc. This means that normally, you want to avoid data structures that introduce a lot of objects trees, linked lists and stuff like that. In some circumstances you might lose some speed, but win big time when the full GC comes. Additionally, It may be worthwhile to consider avoiding heap allocations by organizing your data as value types instead of ordinary objects.
      - Nick-Craver 12y ago
        It depends on your definitions of "a lot" is. It can scale pretty high by default, but when you're doing hundreds of millions of objects in a short window, you can actually measure pauses in the app domain while GC runs. These have a pretty decent impact on requests that hit up against that block. The vast majority of programmers need not worry about this, but if you're putting millions of objects through a single app domain in a short time, then it's a concern you should be aware of. This applies to pretty much any managed language.
ck2 12y ago
This is the fascinating part to me, their SSD have not failed: Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.
- dekhn 12y ago
  I would agree that hundreds of drives is a large enough sample that you would expect to see some failures. That said, all the SSDs I've had have taken repeated pounding with no complaints. I accidentally swapped a few terabytes to swap on an SSD, and was simply surprised that the job I was doing finished faster.
- Nick-Craver 12y ago
  Yep, still true. We lost one Intel 910 drive (PCIe SSD), and that was very abnormal - died so soon it was almost DOA. We hooked up directly with Intel for them to The replacement is still going strong as is another 910 we have. All of those 2.5" Intels though, still trucking along! We're looking at some P3700s PCIe NVMe drives now, blog post coming about that.
- kmontrose 12y ago
  There's a real difference between consumer and enterprise grade SSDs. For comparison, for a while we nearly had a 1 to 1 dev to killed (consumer) SSD in work machines (see: http://blog.codinghorror.com/the-hot-crazy-solid-state-drive-scale/ http://blog.codinghorror.com/the-hot-crazy-solid-state-drive...). Granted, we did adopt them (see: http://blog.codinghorror.com/the-state-of-solid-state-hard-drives/ http://blog.codinghorror.com/the-state-of-solid-state-hard-d...) for local dev use much earlier than we started putting them in servers. I've killed 2 in my time here, though I do horrible big-ish data things that are either not SSD friendly or actively hostile.
fideloper 12y ago
I love that all the traffic is running through a single HAProxy box - a software load balancer (vs a hardware LB like F5). And they've moved SSL termination to it. That's a great quality product, and easy to setup. Edit: I work I'm software that supports MySQL,PG and SqlServer. SqlServerr seems to be the most stable and consistent in performance - they're hard to kill! One of my few liked MS products :D
zobzu 12y ago
So doing it right works. Nice :) I like these articles as it gives good examples/help supporting decisions when you dont want the crappy hack, but the do it right instead.
bikamonki 12y ago
I never ever ever never use SO's search feature: I land on the result through Google. So I guess they should give Google some credit for the claimed performance?
- cruise02 12y ago
  Google isn't serving Stack Exchange's pages to you. They published stats on pages served by their own servers. Google has nothing to do with it.
  - bikamonki 12y ago
    You miss the point: I never use the search feature of the site, I go to the answer/subject directly from Google's results. I do not think Google is running my keywords against SO's server in 'real-time' and returning the answers as results, Google returns indexed/cached copies. I am sure that many users behave like I do, no? They might as well keep a static copy of all pages in RAM and only update when new comments/answers are posted.
yblu 12y ago
> Redis has 2 slaves, SQL has 2 replicas, tag engine has 3 nodes, elastic has 3 nodes - any other service has high availability as well (and exists in both data centers) Does this mean the 2 Redis slaves, 2 SQL replicates etc. are in a secondary data center?
- Nick-Craver 12y ago
  We have 2 SQL Clusters. In the primary data center (New York) there is usually 1 master and 1 read-only replica in each cluster. There's also 1 read-only replica (async) in the DR data center (Oregon). If we're running in Oregon then the primary is there and both of the New York replicas are read-only and async. For redis both data centers have 2 servers, the chain would look like this typically: ny-redis01 - ny-redis02 - or-redis01 - or-redis02 Not everything is slaved between data centers (very temporary cache data we don't need to eat bandwidth by syncing, etc.) but the big items are so we don't come up without any shared cache in case of a hard down in the active data center. We can start without a cache if we need to, but it isn't very graceful. Here's a screenshot of our redis dashboard just a moment ago (counts are low because we upgraded to 2.8.12 last week, which needed a restart): http://i.stack.imgur.com/IgaBU.png http://i.stack.imgur.com/IgaBU.png Nick Craver - Stack Exchange Sysadmin & Developer
  - yblu 12y ago
    Great info, Nick. Thanks for sharing.
doczoidberg 12y ago
I would love to see comparing costs of their solution against cloud solutions as Azure. They say that hardware is cheaper than programmers. Wouldn't that speak for the cloud solution (software based scaling instead of admins)?
- Nick-Craver 12y ago
  Hardware is cheaper than developers and efficient code. But you're only as fast as your slowest bottleneck and all the current cloud solutions have fundamental performance or capacity limits we run into. Could you do it well if building for the cloud from day one? Yeah sure, I think so. Could you consistency render all your pages performing several up to date queries and cache fetches across that cloud network you don't control and getting sub 50ms render times? Well that's another matter, and unless you're talking about substantially higher cost (at least 3-4x), the answer is no - it's still more economical for us to host our own servers. And yes I realize that's a fun claim to make without detail. As soon as I get a free day (I think there's one scheduled in 2017 somewhere), I'll get with George to do another estimate of what AWS and Azure would cost and what bottlenecks they each have. Nick Craver - Stack Exchange Sysadmin & Developer
noelwelsh 12y ago
StackOverflow shows just how powerful using a fast language can be. Compare the TechEmpower benchmarks for, say, Java vs Rails on a very simple JSON serialization benchmark (we can assume .Net would achieve comparable performance to the JVM): http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ The Java servers get around 900K requests/s on beefy hardware while Rails squeezes out 6K. That's a 150x difference! Any real application will be slower than this, and on cloud hardware you can expect to be a order of magnitude slower as well. You just don't have much headroom if you use languages like Ruby before you have to scale out. And once you scale out you have to worry about sys. admin. and all the problems of a distributed system. It's a one off cost to learn an efficient language, but it pays returns forever.
- skndr 12y ago
  Conversely, you might be able to iterate faster with a 'web' language and create a product people really use in a shorter time frame and worry about scaling later!
  - noelwelsh 12y ago
    5 years ago it might have been true that RoR was that much more productive it was worth using, but that doesn't match my experience today using RoR and Scala. Update: To try to avoid turning this into a language war, there are other good reasons to use something like RoR. Ease of hiring is one. Using known tools is another, if you're not building anything you expect to get load.
  - pjc50 12y ago
    Strange, to me Java and ASP.NET (as used by SE) are web languages.
    - brixon 12y ago
      No, they are general purpose languages that can be used for the web. Languages like ColdFusion and PHP are web languages that don't make much sense outside of the web.
ksec 12y ago
It is fascinating to see how well its server it holding out. With Plenty of headroom to spare!. With Higher Capacity DDR4, and Haswell or even Broadwell Xeon, and PCIe SSD getting cheaper and faster. It is not hard to imagine they could handle 3 - 4 times the load when they upgrade their server two years down the line. But i would love to see Joel's take on it though. Since he is a Ruby guy now, and I dont think you could even achieve 20% of SO performance with RoR.
- farresito 12y ago
  It would definitely be several orders of magnitude slower. That's the price you pay when you are using a language like ruby: it's very productive and fast to iterate in, but way less performant. At the end of the day, what you use depends on your needs.
  - ksec 12y ago
    Arh... I actually meant Jeff, not Joel. Hardware is cheap, Programmers are expensive! But i am sure there is a line where this crosses over though.
    - judk 12y ago
      The point of this article is that it costs more programmers to maintain a cloud than to maintain a single master database
ilaksh 12y ago
So funny because the most important application for programmers uses the complete opposite of almost every web development trend out there. Also the SSDs are funny to me because I recently had someone who was supposed to be an expert say that SSDs were a fatal mistake with a high likelihood of massive failure.

479 points by quicksilver03 12y ago

Relatively new stack dev here. I came in on the other side of the fence of a lot of these technologies (bread and butter is Python in Flask with Mongo on Heroku on a mac) but since I started here, I've been constantly and pleasantly surprised by how performant everything here has been despite my biases. It's mighty fun.

chiph 12y ago

I've recently started using their micro-ORM Dapper, and I like it a lot. I get the performance of hand-coded SQL, but without the tedious mapping from SqlDataReader to my entity.

Aaronontheweb 12y ago
Dapper is great - I promptly dumped NHibernate and Entity Framework for it. It's the only ORM I've found that makes it trivially easy (as it should be) to use stored procedures and views in queries.
- atonse 12y ago
  +1 for Dapper - I'm a big fan. It is very fast and we use it to serve our live (mostly read-only) traffic, while using something like Entity Framework for the backend Administrative UI, which has a lot of inserts and updates.
  - chiph 12y ago
    There's a couple of user-contrib projects that do similar things for inserts & updates. Dapper Extensions is the one I'm using. https://github.com/tmsmith/Dapper-Extensions https://github.com/tmsmith/Dapper-Extensions
    - Aaronontheweb 12y ago
      I wrote a Dapper extension for working with SQL Server's geospatial queries and types a couple of years ago - have they added anything like that yet? Otherwise I'd be happy to add it.
      - Nick-Craver 12y ago
        Marc Gravell added in the ability to put in pretty much any custom type without adding any dependency weight about a month ago. You can see the commit here: https://github.com/StackExchange/dapper-dot-net/commit/e26ee0abe5bdff561ff59fded87cabb9f5d983a1 https://github.com/StackExchange/dapper-dot-net/commit/e26ee... Look towards the end at the tests for example of how to hook up a custom type (it's pretty simple).
        Aaronontheweb 12y ago
        Cool - the one I wrote was years ago. Looks like it's not necessary any more.
teh_klev 12y ago
I've been using Dapper for around three years, "it just works" and is very fast. I've been around the block a few times with ORM's - LINQ to SQL, NHibernate, Entity Framework, Massive, SubSonic. For small to medium size projects there's no contest, provided you don't mind hand crafting your SQL. Hat's off to Marc, Sam and team for building this.
Akkuma 12y ago
Another awesome one which I've used on a previous project is SqlFu, https://github.com/sapiens/SqlFu https://github.com/sapiens/SqlFu
pbz 12y ago
Dapper is nice, but if you want to have the compiler check your queries, and still run fast, I recommend LINQ to DB: https://github.com/linq2db/linq2db https://github.com/linq2db/linq2db (not to be confused with LINQ to SQL from Microsoft)
- Nick-Craver 12y ago
  To be fair, the compiler is checking against what you told it the database looks like. This can help with lots of things, but still has the fundamental disconnect problem you'll get at runtime with either approach. A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example we (and by "we" I mean, "I made Marc Gravell do it") added literal replacement to Dapper to help with query parameterization which allows you to use things like filtered indexes. In dapper we also intercept the SQL calls to dapper and add add exactly where it came from. Here's what that looks like, replicate for the other methods: public static IEnumerable<T> Query<T>(this DataContext db, SqlBuilder.Template template, bool buffered = true, int? commandTimeout = null, IDbTransaction transaction = null, [CallerFilePath]string fromFile = null, [CallerLineNumber]int onLine = 0, string comment = null) { using (db.Connection.EnsureOpen()) { return SqlMapper.Query<T>(db.Connection, MarkSqlString(template.RawSql, fromFile, onLine, comment), template.Parameters as object, transaction ?? db.Transaction, buffered, commandTimeout); } } And here's the our marking method tailored for our code, but you get the idea: private static string MarkSqlString(string sql, string path, int lineNumber, string comment) { if (path.IsNullOrEmpty() || lineNumber == 0) { return sql; } var commentWrap = " "; var i = sql.IndexOf(Environment.NewLine); // if we didn't find \n, or it was the very end, go to the first space method if (i < 0 || i == sql.Length - 1) { i = sql.IndexOf(' '); commentWrap = Environment.NewLine; } if (i < 0) return sql; // Grab one directory and the file name worth of the path // this dodges problems with the build server using temp dirs // but also gives us enough info to uniquely identify a queries location var split = path.LastIndexOf('\\') - 1; if (split < 0) return sql; split = path.LastIndexOf('\\', split); if (split < 0) return sql; split++; // just for Craver var sqlComment = " /* " + path.Substring(split) + "@" + lineNumber + (comment.HasValue() ? " - " + comment : "") + " */" + commentWrap; return sql.Substring(0, i) + sqlComment + sql.Substring(i); } This results in a comment at the top of the query like this: /* Models\Post.LinkedQuestions.cs@105 */ select top (@top) p.Id , p.Title , p.Score It's such a simple approach using the caller member attributes, but it saves so much time tracking things down.
j_s 12y ago
Another library worth mentioning in this space is Massive, which uses .NET dynamics a lot (focusing slightly less on performance than ease of use). https://github.com/robconery/massive https://github.com/robconery/massive

lemcoe9 12y ago

Makes me wonder why smaller website teams need dozens of engineers to keep their infinitesimally smaller app running.

edgyswingset 12y ago
Rigid adherence to certain technologies could be a culprit. "We're a Ruby shop!" is fine for problems where Ruby is great, but the limitations of the technology are bound to creep up somewhere. Not using a polyglot approach would be bad.
- nostrademons 12y ago
  It can also be caused by too much embrace of polyglot tools. "We use the best tools for the job" can easily mean an architecture with Redis, Memcached, Mongo, MySQL, Angular, JQuery, Node, and several dozen third-party libraries, all glued together with Thrift or JSON-RPC. If all you're doing is a messaging app, you might be much better off with a single in-process Java app. Usually you want your solution to be just right for the problem; solve the essential complexity but don't introduce any accidental complexity.
  - swah 12y ago
    I wish more people considered this alternative when recommending the usage of queues etc. I always remember of this post: http://williamedwardscoder.tumblr.com/post/18065079081/cogs-bad http://williamedwardscoder.tumblr.com/post/18065079081/cogs-...
    - nostrademons 12y ago
      Hacker News also runs on a single box using only in-language hashtables and lists. ViaWeb did too, and PlentyOfFish.
daigoba66 12y ago
I personally think it's due to overly complex applications and systems with many moving (and breaking) parts. Also StackOverflow often practices "scale-up" instead of "scale-out".

tomblomfield 12y ago

"One problem is not many tests. Tests aren’t needed because there’s a great community... If users find any problems with it they report the bugs that they’ve found." I'm often surprised at the paucity of test-coverage in relatively large companies. http://nerds.airbnb.com/testing-at-airbnb/ http://nerds.airbnb.com/testing-at-airbnb/

aturek 12y ago
That bullet point surprised me. It comes right after "110K lines of code. A small number given what it does"; to me this reads like they didn't write tests because that would add complexity. Can any SO devs give us more details on "not many tests"? Or how many bug reports get filed vs rate of change of software?
- jc4p 12y ago
  The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange). You can look at reported bugs here: http://meta.stackexchange.com/questions/tagged/bug http://meta.stackexchange.com/questions/tagged/bug -- it's pretty frequent, at least one every couple of hours or so. The last commit in our "Tests" project was 14 days ago, where as the last commit in the code base was 4 minutes ago.
  - jacques_chester 12y ago
    You should come down our way and do a tech talk about it. If only for the wailing, gnashing of teeth and rending of garments.
    - jc4p 12y ago
      Your people would literally murder me. Are you back from west coast / toronto?

habosa 12y ago

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

x1798DE 12y ago
It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive. I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system. 1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#physical_memory_limits_windows_server_2012 http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...
- mrb 12y ago
  $4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160. Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months. A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).
  - ZachPruckowski 12y ago
    Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.

programminggeek 12y ago

One smart thing they are doing is putting different sites on different databases. It effectively as a kind of partitioning to allow for horizontal scaling if needed. If nothing else, it keeps less data in each table, so queries should be faster due to smaller datasets. Sounds like they know what they are doing.

beachstartup 12y ago
beware that you can over-optimized this. for example, do not put every single customer of an saas solution into a separate database.
- csharrison 12y ago
  I'm curious, why not? If your saas was small enough and had few enough customers with large data needs, separating them each out into a separate database seems like a viable solution.
  - beachstartup 12y ago
    yeah, if you have an experienced team, you can easily refactor your one-customer-per-db model into something that will handle 1,000 new customers signing up every month. given a good enough team, you can even do this while doing other sort-of important things like customer support, bug fixes, and new development. however, inexperienced developers and teams are not good enough, or fast enough, to do that, and end up painting themselves into a corner when they have thousands of live customers to support and need to change their entire application architecture + database schema when their backup, replication, and housekeeping tasks choke on 1,000+ databases. and oftentimes, one-customer-per-db also means one-instance-of-application-per-customer, another anti pattern to avoid. we see this kind of stuff ALL the time. it happens a lot - people make terrible decisions and then are stuck with them 5 years down the line and are looking at a monumental cost to redo. not everyone is experience enough to work their way out of an awful situation like that.

UVB-76 12y ago

Anyone else not quite expecting StackExchange to be using a Microsoft stack?

ufo 12y ago
Anyone who knows Joel Spolsky would expect a MS stack :)
- Alupis 12y ago
  It's difficult to take pragmatic advice from someone who took the least pragmatic (and most expensive) route for a startup web company.
  - rstupek 12y ago
    I'm not sure I agree. Pragmatic choices based on using what you know and not just choosing a toolset because everyone else is using it.
  - WorldWideWayne 12y ago
    Why does everyone in Silicon Valley run with expensive Apple hardware when Linux can run on any CPU?
  - Amezarak 12y ago
    If you have extensive Microsoft experience and take advantage of something like BizSpark* (although one suspects with their connections they wouldn't have needed that) it's hard to see how it wouldn't be plenty pragmatic and inexpensive. * Currently three years of free licenses, all software you download during that time is free forever, and discounted MSDN subs thereafter.
  - jacquesm 12y ago
    Programmer time is a factor that comes into play when you build something and re-training your people to use a stack they are not familiar with + the associated mistakes can get expensive in a hurry. If you happen to be familiar with the MS stack and are able to factor the license fees into your business plan there is no strong reason against and many for using that particular stack. MS programmers tend to be a bit more expensive than people working with one of the many Linux offerings but that's not such a huge difference that it would become the deciding factor and if anything the MS stack is more performant than the Linux one on the same hardware (and I write that as an anti-MS guy, I strongly believe that politics and tech should be kept separate when it comes to discussing systems relative to each other). So even if there are lots of reasons why I would not have made that particular choice I can completely understand why the SO people made their choice the other way.

tjdetwiler 12y ago

Can anyone provide context on "Heavy usage of static classes and methods, for simplicity and better performance." Is performance _really_ an issue here?

Griever 12y ago
I could be mistaken, but I think this is in regards to that post Sam Saffron wrote a few years back about getting SO to do Gen.2 garbage collection less often, and using static classes and structs were one of the ways that they fixed it.
MichaelGG 12y ago
I find the opposition to static classes odd. Unless you need the data encapsulation of an object, why force people to create an object to call functions? I find too much C# code loves creating objects for no real purpose other than that's how it works. For SO, if we guess they max out around 2000 req/sec, then if a bunch of objects are being allocated for each request, there could be added GC pressure. From my own experience developing much higher-scale managed code, allocating objects is a real pain and people will take a lot of effort to avoid it.
- Griever 12y ago
  I think the real issue (although it doesn't seem to be an issue for SO) is testability. AFAIK, static classes cannot be mocked, and you are pretty much boxing yourself into only doing integration tests for everything. Not necessarily a bad way to go, just lots more setup per test.
  - MrBuddyCasino 12y ago
    Yep, that used to be one of the main reasons to have DI in Java. But now that everything (even static methods) can be mocked, that reason is no longer valid. Is this not possible in .NET?
  - mattmanser 12y ago
    It completely depends if you do this: public static decimal DiscountValue(Order order) Or this: public static decimal DiscountValue(decimal orderValue, decimal orderDiscount) Or even this, which would farm the work out to SQL: public static decimal DiscountValue(int orderId) The first would require a mock, the 2nd wouldn't and the third isn't unit testable. It's a bit of a contrived example, as you're more likely to do the third method if you're trying to get some sort of summary data about an order out which would mean you don't really need/want to load the whole order object.

nmjohn 12y ago

> AWS would cost a fortune. Isn't that comparing apples to oranges? Taking any application built for large servers and putting the same application onto a cloud-based architecture will be more expensive. Cloud-based architecture requires ground up differences in how the application is built. Now whether or not it is better to use a cloud-based approach or traditional bare metal is highly subjective and isn't my point.

incision 12y ago
>'Isn't that comparing apples to oranges?' On another site, in another context, it probably would be, but here it's really presented as a contrast of scale-up versus scale-out - something the regular audience of highscalability will certainly grok. In context... 'Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t been a problem because they’ve been quite successful at sizing their system correctly.' It's an acknowledgement of their relatively unique strategy and the short list of caveats that make it possible.

sz4kerto 12y ago

Extremely interesting read, especially because it goes against many fashionable engineering practices. They see (correctly) that abstractions, DI have their own compromises. Also, they're not religious (about technology), that's also quite rare.

kbenson 12y ago

> The cost of inefficient code can be higher than you think. Efficient code stretches hardware further, reduces power usage, makes code easier for programmers to understand. I'm curious what the reasoning is for "Efficient code ... makes code easier for programmers to understand". To my mind, efficient code (in this case, I assume coding to the hardware, as they mention elsewhere), has many benefits, but making it easier to understand is not one of them. A useful comment by the code may help, but that's not a result of efficient coding, that's a result of good commenting practice.

hueving 12y ago
Definitely. Some of the most efficient C code I have seen is incomprehensible to anyone but a C expert and even then it takes several minutes to understand all of the nuances of what might only be a 4-line function.
- mkr-hn 12y ago
  I experienced exactly this when I cracked open the source to a GNU tool to see if I could figure it out.
itry 12y ago
users=db.get("SELECT name FROM users WHERE group='admin'") is more efficient and understandable then: users=[]; tmp=db.get("SELECT name,group FROM users") for (i=0;i<tmp.count();i++) { if (tmp[i]['group']=='admin') users[]=tmp[i][name]; }
- CodeCube 12y ago
  This is where a good abstraction layer can pay dividends. Of course, a bad abstraction layer (as evidenced by Spolsky's "leaky abstraction" post (http://www.joelonsoftware.com/articles/LeakyAbstractions.html) http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...) on the opposite hand can add immeasurable cost.
- abritishguy 12y ago

lnanek2 12y ago

Really hate the separate sites thing. I use a half dozen of them and they are computer related, so it is a pain in the ass always having to register and not being allowed to comment/answer at first, etc.. They have a few improvements now like importing your profile from other sites, but they shouldn't even have so many separate sites in the first place, just a tag or a category or something.

recursive 12y ago
You get +100 rep bonus for linking your accounts, so you shouldn't have a problem being allowed to comment.
- JelteF 12y ago
  You do need to have more than 200 rep on one site though.
  - r00fus 12y ago
    Which one? I'm a full member on all the StackExchange sites I care about for now with the +100 cross-membership bonus.
    - iancarroll 12y ago
      In order to get the +100 bonus on all sites, you must have one profile on the network with >200 reputation.
- rayshan 12y ago
  How? I can't seem to find this option in account admin panels.
  - zo1 12y ago
    It happens automatically, if I remember correctly.
  - 12y ago

epsteinbargraph 12y ago

Soooo... a average of 200 pageviews a second - I suppose that doesn't sound as cool. Or, to put it another way; 1 server per 8 pageviews per second Does not seem that efficient when you put it like that. Even a resource hog like a wordpress install can eclipse that level. *made correction- still not impressed

NietTim 12y ago
Well yeah, but these aren't just plain old text file of course. Database queries, assets, async clients, the whole lot.
jre 12y ago
I think you're missing a 0 somewhere : 560000000 / (60 * 60 * 24 * 31) = 209
- deleted 12y ago
  [deleted]
cruise02 12y ago
You're off by a factor of 10. I have it at just over 200 page views per second. Also, that's the average load over a month. The same hardware needs to be able to handle peak load.
- tiernano 12y ago
  and they also mention that they could possibly run on a lot less hardware... 7-8 machines + network gear (7 if the load balancer is an appliance, 8 if its a server).

MichaelGG 12y ago

So this is what, a 2000 request/sec peak? Over 11 servers, that's like 200 requests/sec peak per frontend? The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their transactions in RAM and keep it there (replicated). I know that's what VoltDB is counting on.

sklivvz1971 12y ago
Yes, the idea is to scale up as much as economically makes sense and only then scale out, but due to Moore's law we're still scaling up. Not everyone has google-like problems that are betters solved by a battery of cheap boxes.
- threeseed 12y ago
  Actually most of the websites today are better solved by a battery of cheap boxes. You simply put a load balancer (Nginx/ELB/HAProxy etc) in front of a fleet of smaller web/application servers that dynamically scale depending on traffic. That way it is cost effective, far more reliable, easier to scale and you can tolerate DC outages better.
  - mwsherman 12y ago
    As with most systems, the data store is the hard thing to scale out. That requires intelligence at the application level, and that might include CAP issues. So for us, the SQL data store is the real “up” part of the equation. We have a fair amount of headroom there, so if we can keep sharding (and other data strategies) out of the codebase, so much the better. (Load-balancing HTTP requests “out” is not a big deal and we are doing that.)
  - sklivvz1971 12y ago
    Marco/Stack Overflow here. Simply put, this is not true at all. Most of the websites today run happily on a single box and the choice of architecture is frankly irrelevant because of the abysmal amount of traffic they get. Large, and by large I mean top 100 sites, are large and typically complex systems. Each one is different and bespoke. Making broad generalizations doesn't really work, because typically they have parts that need to be scaled up, parts that need to be scaled out, etc. Finally, if you read the article you would know that we already use a load balancer with multiple web servers. What we are discussing here is whether it's better to have 100s of cheap/cloud boxes versus a few big ones. In our use case (we want fast response times) the second solution is demonstrably better.

josephscott 12y ago

"with just 25 servers" This really needs to stop. Go to stackexchange.com and you'll find that more than half of the HTTP requests are to cdn.sstatic.net. Looking up the IP addresses for cdn.sstatic.net returned five entries for me, all owned by CloudFlare. None of the CloudFlare servers that they are using seem to be in that 25 count. Sure, these are all for static assets, that isn't the point. There are way more than 25 servers being used to serve the StackOverflow sites.

tiernano 12y ago
Stack Exchange dont own their CDN. If you really start couting external servers, you would be adding DNS, possible client side proxy servers, client side network infrastructure, etc... these are things that OTHER people do, not SO.
- duderific 12y ago
  I would say though, CDN doesn't fall in the same category as DNS. Serving static assets off other networks does take a huge load off the SE infrastructure and should probably be mentioned in the article.
  - mschuster91 12y ago
    Only in terms of bandwidth, though. The CPU load by processing is the interesting figure, not side noise by images/js/css.
sklivvz1971 12y ago
It's how many servers you would need to provision to do the same, with the same code base. It's basically the number of directly active servers our code runs on. Sure, we use a CDN, but not for CPU or I/O load, but to make you find your answers faster.
kondro 12y ago
Sort of true, but static assets are also trivial to serve. What they get by using CloudFlare is getting their assets as close to end users as possible to reduce latency. This isn't a complex problem to solve (for a single site), just a really expensive one that very few single app companies could ever justify solving themselves. I believe Google and Amazon are the only ones with their own significant CDN infrastructure, everyone else outsources it.

Nux 12y ago

"Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there’s no compelling reason to change." <- there's your problem :-)

serge2k 12y ago
54th ranked site in the world with a small number of servers and apparently very maintainable and stable systems. I would love that problem.

rakoo 12y ago

What I'd like to see is how much cache hit they add on each level of cache. I remember some presentation of the facebook images stack where less than 10% (if I remember well) of the requests actually hit the disks; it would be interesting to see the patterns for the whole SE galaxy.

sklivvz1971 12y ago
It totally depends on what we are caching (e.g. the community bullettins or hot meta questions get a gazillion hits...)
skeletonjelly 12y ago
As someone mentioned above, SQL Server hits memory first, and as they have 384GB of memory, most of the requests would sit there. That's just on the DB server. In the linked article it all authenticated requests hit the DB, with anon users getting a cached copy.

Perceptes 12y ago

I'm grateful to them them for being so transparent about how their systems work. It's rare to get this level of detail on how a large, successful site operates.

phkahler 12y ago

I haven't been able to use SO since they shut off myopenid last year. I can't log in, so can't change my credentials ;-( Any ideas HN?

GeorgeBeech 12y ago
Email team@stackexchange.com they should be able to help you out. (I'm an SE SysAdmin )
- phkahler 12y ago
  Thanks, I followed the recovery link provided by another SE person.
jseutter 12y ago
http://meta.stackexchange.com/questions/115223/my-openid-provider-already-shut-down-is-there-any-way-for-me-to-access-my-accou http://meta.stackexchange.com/questions/115223/my-openid-pro...
Shog9 12y ago
You can email us (as George said) or you can just punch your email address into the account recovery tool and get an automated email telling you what to click within the space of a few minutes: http://stackoverflow.com/users/account-recovery http://stackoverflow.com/users/account-recovery We... probably need to make that link a bit more obvious.
- phkahler 12y ago
  Thanks, that was easy. So now I just have a password instead of linking to another account @facebook or google+ I guess that works ;-)

AlisdairO 12y ago

Stack Overflow is the example I'm forever using when arguing against premature scale-out. For non-trivial applications scale-out has substantial complexity costs attached to it, and the overwhelming majority of applications will never truly need it. It's frustrating to see time wasted obsessing over trying to maintain eventual consistency (or chasing bugs when you failed to do so) on systems that could quite happily run on a single, not that beefy machine.

samstave 12y ago
>For non-trivial applications scale-out has substantial complexity costs attached to it Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out. From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of their forum is not an extraordinarily complex problem.
- AlisdairO 12y ago
  By nontrivial I mean 'application which, in your document database, requires multi-document updates to perform some individual logical operations'. This is a rather low bar :-). The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course there are real reasons to use scale out - I simply advocate thinking hard about whether your problem will ever truly need it before taking the substantial complexity hit of coding for it.
  - AlisdairO 12y ago
    Er, I don't seem to be able to edit, but I guess I should specify that I'm really talking about scale-out of writes here. Read scalability is an obviously easier problem..

ternaryoperator 12y ago

"SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods." I don't understand this at all. What does TDD have to do with reducing garbage collection?

numbsafari 12y ago
My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.
- bjourne 12y ago
  IME that always happens when you try to perform tdd in combination with Javaesque encapsulation. The good solution to the problem is to not be so afraid of classes seeing each others internals. The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.
  - mistermann 12y ago
    > The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable. Not doing this will cause much self-righteous snickering from some.
  - edwinnathaniel 12y ago
    You do realize that most efficient DI framework only inject the dependency once. Separating Controller, Repository, and Services are good practices as well and let's be honest, we're looking at 3 methods layer at most. Here's what happened in Java: 1. When you deploy your WAR, the DI will inject necessary component _once_ (and these components only instantiated _once_ for the whole web-app so there you go, Singleton without hardcoding). 2. A request comes in, gets processed by the pumped-up Servlet (already DI-ed). If the servlet has not been instantiated, it will be instantiated _once_ and the instance of the Servlet is kept in memory. 3. Another request comes in, gets processed by the same pumped-up Servlet that is already instantiated and has already been injected with the component (no more injection, no more instantiation, no more Object to create...) So I've got to ask this question: what GC problem we're trying to solve here? Some of the static methods are understandable but if Component A requires Component B and both of them have already been instantiated _once_ and have been properly wired up together, we have 2 Objects for the lifetime of the application. I'd pay for a wee bit extra hardware for the cost of maintainable code ;) Discipline (and knowledge of object graph) definitely help to reach to that point.

guiomie 12y ago

Can someone help me understand what they mean by: "Garbage collection driven programming. SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods. While extreme, the result is highly performing code."

louthy 12y ago
I assume it means using C# more as a functional language than an OO language, so that most garbage collection is of short-lifetime objects and therefore cheaper to collect (generation 0).
- virmundi 12y ago
  I take that statement to mean C# as C. Functional languages are terrible for garbage collection. Static methods are great since they are static, loaded once and probably in-lineable by the compiler/JIT.
  - thomasz 12y ago
    The usual wisdom in clr-land is that the generational garbage collector can cope just fine with a lot of small, short lived objects. It absolutely doesn't like big, long lived objects that reach G1, G2 or the large object heap and must be traversed for a full gc. This means that normally, you want to avoid data structures that introduce a lot of objects trees, linked lists and stuff like that. In some circumstances you might lose some speed, but win big time when the full GC comes. Additionally, It may be worthwhile to consider avoiding heap allocations by organizing your data as value types instead of ordinary objects.
    - Nick-Craver 12y ago
      It depends on your definitions of "a lot" is. It can scale pretty high by default, but when you're doing hundreds of millions of objects in a short window, you can actually measure pauses in the app domain while GC runs. These have a pretty decent impact on requests that hit up against that block. The vast majority of programmers need not worry about this, but if you're putting millions of objects through a single app domain in a short time, then it's a concern you should be aware of. This applies to pretty much any managed language.

ck2 12y ago

This is the fascinating part to me, their SSD have not failed: Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.

dekhn 12y ago
I would agree that hundreds of drives is a large enough sample that you would expect to see some failures. That said, all the SSDs I've had have taken repeated pounding with no complaints. I accidentally swapped a few terabytes to swap on an SSD, and was simply surprised that the job I was doing finished faster.
Nick-Craver 12y ago
Yep, still true. We lost one Intel 910 drive (PCIe SSD), and that was very abnormal - died so soon it was almost DOA. We hooked up directly with Intel for them to The replacement is still going strong as is another 910 we have. All of those 2.5" Intels though, still trucking along! We're looking at some P3700s PCIe NVMe drives now, blog post coming about that.
kmontrose 12y ago
There's a real difference between consumer and enterprise grade SSDs. For comparison, for a while we nearly had a 1 to 1 dev to killed (consumer) SSD in work machines (see: http://blog.codinghorror.com/the-hot-crazy-solid-state-drive-scale/ http://blog.codinghorror.com/the-hot-crazy-solid-state-drive...). Granted, we did adopt them (see: http://blog.codinghorror.com/the-state-of-solid-state-hard-drives/ http://blog.codinghorror.com/the-state-of-solid-state-hard-d...) for local dev use much earlier than we started putting them in servers. I've killed 2 in my time here, though I do horrible big-ish data things that are either not SSD friendly or actively hostile.

fideloper 12y ago

I love that all the traffic is running through a single HAProxy box - a software load balancer (vs a hardware LB like F5). And they've moved SSL termination to it. That's a great quality product, and easy to setup. Edit: I work I'm software that supports MySQL,PG and SqlServer. SqlServerr seems to be the most stable and consistent in performance - they're hard to kill! One of my few liked MS products :D

zobzu 12y ago

So doing it right works. Nice :) I like these articles as it gives good examples/help supporting decisions when you dont want the crappy hack, but the do it right instead.

bikamonki 12y ago

I never ever ever never use SO's search feature: I land on the result through Google. So I guess they should give Google some credit for the claimed performance?

cruise02 12y ago
Google isn't serving Stack Exchange's pages to you. They published stats on pages served by their own servers. Google has nothing to do with it.
- bikamonki 12y ago
  You miss the point: I never use the search feature of the site, I go to the answer/subject directly from Google's results. I do not think Google is running my keywords against SO's server in 'real-time' and returning the answers as results, Google returns indexed/cached copies. I am sure that many users behave like I do, no? They might as well keep a static copy of all pages in RAM and only update when new comments/answers are posted.

yblu 12y ago

> Redis has 2 slaves, SQL has 2 replicas, tag engine has 3 nodes, elastic has 3 nodes - any other service has high availability as well (and exists in both data centers) Does this mean the 2 Redis slaves, 2 SQL replicates etc. are in a secondary data center?

Nick-Craver 12y ago
We have 2 SQL Clusters. In the primary data center (New York) there is usually 1 master and 1 read-only replica in each cluster. There's also 1 read-only replica (async) in the DR data center (Oregon). If we're running in Oregon then the primary is there and both of the New York replicas are read-only and async. For redis both data centers have 2 servers, the chain would look like this typically: ny-redis01 - ny-redis02 - or-redis01 - or-redis02 Not everything is slaved between data centers (very temporary cache data we don't need to eat bandwidth by syncing, etc.) but the big items are so we don't come up without any shared cache in case of a hard down in the active data center. We can start without a cache if we need to, but it isn't very graceful. Here's a screenshot of our redis dashboard just a moment ago (counts are low because we upgraded to 2.8.12 last week, which needed a restart): http://i.stack.imgur.com/IgaBU.png http://i.stack.imgur.com/IgaBU.png Nick Craver - Stack Exchange Sysadmin & Developer
- yblu 12y ago
  Great info, Nick. Thanks for sharing.

doczoidberg 12y ago

I would love to see comparing costs of their solution against cloud solutions as Azure. They say that hardware is cheaper than programmers. Wouldn't that speak for the cloud solution (software based scaling instead of admins)?

Nick-Craver 12y ago
Hardware is cheaper than developers and efficient code. But you're only as fast as your slowest bottleneck and all the current cloud solutions have fundamental performance or capacity limits we run into. Could you do it well if building for the cloud from day one? Yeah sure, I think so. Could you consistency render all your pages performing several up to date queries and cache fetches across that cloud network you don't control and getting sub 50ms render times? Well that's another matter, and unless you're talking about substantially higher cost (at least 3-4x), the answer is no - it's still more economical for us to host our own servers. And yes I realize that's a fun claim to make without detail. As soon as I get a free day (I think there's one scheduled in 2017 somewhere), I'll get with George to do another estimate of what AWS and Azure would cost and what bottlenecks they each have. Nick Craver - Stack Exchange Sysadmin & Developer

noelwelsh 12y ago

StackOverflow shows just how powerful using a fast language can be. Compare the TechEmpower benchmarks for, say, Java vs Rails on a very simple JSON serialization benchmark (we can assume .Net would achieve comparable performance to the JVM): http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ The Java servers get around 900K requests/s on beefy hardware while Rails squeezes out 6K. That's a 150x difference! Any real application will be slower than this, and on cloud hardware you can expect to be a order of magnitude slower as well. You just don't have much headroom if you use languages like Ruby before you have to scale out. And once you scale out you have to worry about sys. admin. and all the problems of a distributed system. It's a one off cost to learn an efficient language, but it pays returns forever.

skndr 12y ago
Conversely, you might be able to iterate faster with a 'web' language and create a product people really use in a shorter time frame and worry about scaling later!
- noelwelsh 12y ago
  5 years ago it might have been true that RoR was that much more productive it was worth using, but that doesn't match my experience today using RoR and Scala. Update: To try to avoid turning this into a language war, there are other good reasons to use something like RoR. Ease of hiring is one. Using known tools is another, if you're not building anything you expect to get load.
- pjc50 12y ago
  Strange, to me Java and ASP.NET (as used by SE) are web languages.
  - brixon 12y ago
    No, they are general purpose languages that can be used for the web. Languages like ColdFusion and PHP are web languages that don't make much sense outside of the web.

ksec 12y ago

It is fascinating to see how well its server it holding out. With Plenty of headroom to spare!. With Higher Capacity DDR4, and Haswell or even Broadwell Xeon, and PCIe SSD getting cheaper and faster. It is not hard to imagine they could handle 3 - 4 times the load when they upgrade their server two years down the line. But i would love to see Joel's take on it though. Since he is a Ruby guy now, and I dont think you could even achieve 20% of SO performance with RoR.

farresito 12y ago
It would definitely be several orders of magnitude slower. That's the price you pay when you are using a language like ruby: it's very productive and fast to iterate in, but way less performant. At the end of the day, what you use depends on your needs.
- ksec 12y ago
  Arh... I actually meant Jeff, not Joel. Hardware is cheap, Programmers are expensive! But i am sure there is a line where this crosses over though.
  - judk 12y ago
    The point of this article is that it costs more programmers to maintain a cloud than to maintain a single master database

ilaksh 12y ago

So funny because the most important application for programmers uses the complete opposite of almost every web development trend out there. Also the SSDs are funny to me because I recently had someone who was supposed to be an expert say that SSDs were a fatal mistake with a high likelihood of massive failure.