6 ms·

Before anyone wants to criticize Stack Overflow, consider their server footprints as of 2016 [1]: 4 Microsoft SQL Servers (new hardware for 2 of them) 11 I

• kjgkjhfkjf 5y ago
  If they'd used open-source software rather than the Microsoft stack, they could have scaled horizontally without having to worry about buying more software licenses.
  • mrweasel 5y ago
    Personally I'd rather manage a small number of large servers, compared to managing a large number of smaller ones. The licensing costs associated with Stack Overflow also can't been that large. Sure Microsoft software isn't cheap, but it's also not that expensive. The licensing cost is easily offset by staying on a technology stack their employees know.
    • kjgkjhfkjf 5y ago
      > The licensing cost is easily offset by staying on a technology stack their employees know. Stack Overflow wasn't developed by an established software company. It was a completely new startup, founded by two high-profile (at the time) software commentators. They could have hired people to use any stack they wanted. While the state of web development wasn't what it is today, OSS stacks such as the LAMP stack were fairly well-understood and popular.
      • mrweasel 5y ago
        That's a reasonable point, but still is the cost of the licens higher than the cost of one or two additional developers? Jeff Atwood was/is if I recall a pretty talented C# developer, so why onboard extra people. It's more than just their salary, there's a cost beyond that associated with hiring people. I'm not arguing that the licens isn't a concern, I just question how much of a concern it really is, compared to everything else.
      • marcosdumay 5y ago
        The MS stack is affordable to developers working on developing countries and free for small products (with a lot of strings attached). Of course, the OSS stack costs nothing to acquire, is cheaper to admin, and will give you more functionality and easiness of use, but the MS one isn't some unaffordable monster like other proprietary ones.
      • georgemcbay 5y ago
        Stack Overflow existed for a couple of years before taking any funding and it was developed by three people who were friends (Jeff Atwood, Jarrod Dixon and Geoff Dalgas) none of which were even getting paid regularly at the start and who were all primarily familiar with the .NET stack from previous jobs. It makes total sense why the tech is what it is. One could argue they could have shifted away from that stack since then, but one could also easily argue (very convincingly to anyone who has ever been involved with any significant project rewrite) "but... why?".
    • darkwater 5y ago
      > The licensing costs associated with Stack Overflow also can't been that large Because we went with the reliability of a full Microsoft stack—.NET, C#, and MSSQL—our costs grew with the number of instances. Each server required a new license They literally mention license costs as an architecture driving constraint.
      • wwefwe2323g 5y ago
        It's probably only costs for SQL Server, because anything else (ASP.NET, .NET, C#, etc.) is free and open source.
        • reificator 5y ago
          Stack Overflow predates any of those being open source.
        • severino 5y ago
          You couldn't run ASP.NET in Linux back then, so you had to pay for a Windows license, too.
        • forty 5y ago
          Wouldn't they have to pay for IIS as well? (Not mentioning Windows itself?)
          • mrweasel 5y ago
            Windows server yes, you pay for that, but IIS comes bundled.
      • mrweasel 5y ago
        Sorry, I wasn't making that clear. It's not that it isn't a factor, they clearly think it is. The question is how big a factor is it really? My experience is that the licens is less of an issue, compared to the time and resources you'd otherwise need to redesign everything. When Jeff Atwood and Joel Spolsky started Stack Overflow they had a podcast. In that podcast Joel repeatably argued that hardware wasn't that expensive, compared to Jeffs time. His argument was that buying a larger server and scaling that way was much more cost-effective. I think that's still true, to a point, even if you didn't have the licensing cost.
  • pjmlp 5y ago
    Software licenses are usually peanuts versus developer salaries and overall infrastructure costs of the whole company.
    • tomnipotent 5y ago
      Agreed. It's always the people not paying for things that seem to be complaining about paying for things.
    • 5e92cb50239222b 5y ago
      Depends on where you live. Certainly not peanuts here. Edit: I know that if you spent your whole life in California it can be hard to believe that in some places junior developers make $250 a month (senior more than that, but typically less than $1k a month), so a single Windows Server license buys you four months of work of a single junior dev (however unproductive it may be), or 1-1.5 months of a senior dev's work. In my current place of work a few years back I started a strong push towards Linux and as much FOSS as possible, and over some time (as new projects were written and some legacy stuff re-written) it resulted in a massive reduction of total operation costs. I didn't expect any other reaction, though. You understand our realities better than us.
  • p_l 5y ago
    They also commented that the licenses paid for themselves in reduced hw spend, as AOT compiled code plus very performant SQL Server allowed them to run in fraction of the size comparable sites operated.
• fabian2k 5y ago
  https://stackexchange.com/performance https://stackexchange.com/performance There are some more details on this page, including the typical CPU usage on those servers.
• amelius 5y ago
  Would 10x the amount of hardware be a problem for them in any way? Did they aggressively optimize their setup because of hardware costs?
  • habibur 5y ago
    they were losing money in the first few years. struggling to find way to monetize it. at one point were selling programming books with amazon affiliation. these types of cost cuts make the difference between life and death in early stage startups.
    • asddubs 5y ago
      i believe they still do that
• 3np 5y ago
  I hope this is obvious, but best practices can often intentionally sacrifice performance for the sake of other aspects. The fact that breaking them can give performance gains shouldn't come as a surprise and can be the right thing to do. My reasoning is that they should only be broken if you understand the reason for them, how it applies to your situation and the implications (or if nothing's at stake, which can be a great learning experience as to why those recommendations are made).
  • karmakaze 5y ago
    The part that's not obvious to those applying best practices is doing so without such a similar understanding. It is a shallow appeal to authority of the 'best'.
  • Quarrelsome 5y ago
    performance has almost always been about violating best practices because a simple happy path isn't usually suitable for hot traffic (e.g. bulk insert vs insert into). Back in the early 2000s when the traffic just started to hit people were de-normalising databases just to make sure stuff could even work.
  • smoe 5y ago
    Even before breaking them, I think one of the bigger problems with best practices in practice is, that many people have forgotten or never tried to understand why they exist in the first place, what problems they are trying to solve and how, their scope and what the trade-offs are you are making by implementing them. Instead they get treated as magic wands swung around promising to solve all the worlds problems if you just believe in them strongly enough.
• jasfi 5y ago
  If you read further in the article there appear to be 6 SQL Server machines (across 2 clusters). Those are some highly spec'd servers with huge amounts of RAM and CPUs. With decent caching the SQL Server machines wouldn't be hit too often either, hence the Redis and Elasticsearch servers. Fewer, but larger DB servers would typically mean much less maintenance and dev ops headaches. As others have said, skipping best practices is typically reasoned out really well before proceeding. Not everyone needs every best practice.
  • simooooo 5y ago
• cpach 5y ago
  What I don’t understand is how they can stand dealing with IIS. IME it’s so extremely frustrating to use, especially compared to Nginx.
  • mrweasel 5y ago
    It wasn't really an option when they started. Nginx couldn't and still can't run ASP.NET application. You could maybe put the application in a container, or run it as directly with the new versions of .Net, but that requires reworking the entire application. There's not really a lot to gain, not if you already know IIS.
    • cpach 5y ago
      Ah. That makes sense. I wonder to what degree they have automated the configuration of IIS…? From my POV it seems really tricky to automate. But I am by no means an IIS expert, I just have some clients that use it.
      • mrweasel 5y ago
        You can automate IIS configuration with Ansible, that's what we do. Mostly the issue is documentation, you sort of have to know what things are called, but you can get around that by using win_iis_webapppool and get the configuration from a running IIS server and then work out which properties you can set. There might be things you can not automate, we don't use that many advanced features. Edit: I wasn't being fair. Ansible, is actually pretty good at managing Windows servers. It's just different from managing Linux servers.
  • jaywalk 5y ago
    I'm guessing you just don't have much experience with it? As someone who's been using IIS for 20+ years, it's all very straightforward to me.
    • vanusa 5y ago
      Anything is "straightforward" after doing it for 20+ years. Just ask a COBOL programmer.
• heurisko 5y ago
  On the other hand, I worked for a company that seemed to follow Stack Overflow's article (sparse tests, unmockable static fields) that made everything a nightmare and slowed velocity almost to a halt, and provided a brittle customer experience. I can quite believe that caching SQL results saved time (hardly not a "best practice"). The waters become murkier when you say you can't afford basic abstractions to make things testable. They provide no statistics on how much time their "optimisations" actually saved. Likely, their optimisations wouldn't have made a difference, as they would have been JIT'd away anyway. I am very much opinion that if you're writing code without some sort of test, you're writing legacy code.
  • loldot_ 5y ago
    While they do not provide any benchmarks in the summarizing article, I can guarantee that they have done some (read: alot) profiling. I follow both Nick Craver and Marc Gravell on Twitter, and they regularly post benchmarks for optimizations done at SO, in the .net world they are some of the most knowledgable people on performance optimizations.
  • commandlinefan 5y ago
    > (sparse tests, unmockable static fields) that made everything a nightmare I had the same thought. I've noticed over the past few decades that most developers migrate toward static fields (i.e. making everything global) not for performance reasons, but just because it's requires less up-front thought. And when the only thing that shows up on your year-end performance review is "did you hit the date?" this is almost understandable. Maybe the folks at SO really are that good - they can code everything right the first time and there's no particular need for unit tests, but I'm used to finding bugs that have been in the code for YEARS that would have been trivially caught if the code had just been run even one time under observation. I'm also used to code that's so monolithic (statics that import other statics that import other statics) that it takes months for new hires to get any sort of proficiency in - and that's if they're really experienced to begin with.
• namelosw 5y ago
  It's no small feat, but the nature of SO is very cache-friendly (content focused, reads >> writes). If you're building a modern SaaS, dealing with the same amount of traffic as SO, you might find a hard time with the same setup.
  • naasking 5y ago
    Reads are much more frequent than writes in the vast, vast majority of programs.
• desktopninja 5y ago
  Am a firm believer that a billion dollar company (Saas, etc) can be run with very sane and simple stack: - 1x web-server [$FOSS_or_COSS, iis] # COSS => commerical open source e.g caddy. nginx is both - 1x application-server [.net, java, python, php, and all other langs. heck PERL too!] - 1x database-server [$FOSS_or_COSS, sql-server, oracle, etc] Virtualized or baremetal. Containers not necessary! Increase it by 2 or 3 for HA if you so desire. Engineering teams should aspire to advocate "We run this entire thing on these few resources vs our application runs on 26k nodes" The latter just means we've lost the plot and throwing money to solve a problem. Forgive my sarcasm here, but well done k8s&&containers! Don't see any reason why FB can't run on the same comparative hardware as SO. I.e. replace the CCSS (commercial closed source software) bits with the FOSS/COSS ones. And when we take it in as a whole, this is much more greener/sustainable to work with. Also, there is a legend out there that I heard of many moons ago :) ... THERE is a billion dollar company running on a single sql-server and single iis web-server