8 ms·
On Stack Overflow's recent battles with the .NET Garbage Collector
- jroseattle 15y agoTo SamS and the SO team: what drove the choice to Free Text Search? Any other options evaluated at the time?
- Griever 15y agoFantastic read. Thanks for posting it! After reading this article I felt extremely humbled. As a developer doing relatively "simple" work day to day, seeing this level of programming proficiency is very impressive. There isn't a chance I'd be able to figure out a problem such as this. At least certainly not within the timeframe that the SO team fixed it in. It's articles like this that make me want to continue pushing my limits as a developer.
- bhrgunatha 15y ago> There isn't a chance I'd be able to figure out a problem such as this. Don't sell yourself short. You generally don't know what your actual limits are until you really push hard against them or break through them. I've had to hunt down and fix some very unusual problems - with few or no references or no results from google. Those kind of mind-bending problems actually give a great deal of satisfaction. I'm serious that you never really know your limits until you break them. I think the trick is to keep growing and expanding your abilities and knowledge.
- Griever 15y agoVery true. This is generally why I appended that with "at least not within the timeframe that the SO team did". I think that I have gotten to where I am by constantly pushing myself, but reading articles like this just reminds me that I still have a ways to go.
- cookiecaper 15y agoI wonder whether all those gymnastics were more prudent than simply writing a C module to filter questions and feeding the results back to the C# web app. Opinions?
- jcheng 15y agoI was thinking the same thing. This problem seems extremely well suited for C/C++, and C#/.NET makes it very easy to call directly into native libraries.
- sams99 15y agoI agree this is a totally valid option, trouble is we have no c experts on core team so it would raise the cost of maintaining. Also the code is fairly complex, moving to structs was fairly simple compared to a c port.
- cookiecaper 15y agoSounds like the perfect opportunity for someone to become a C expert. Something like this just throws up a lot of red flags. If Microsoft changes some of the methodology in its GC or further demotes structs, your solution may fall apart. It just sounds like something that requires tight manual memory management to get right and remain stable.
- melling 15y agoJeff doesn't know C. Should he learn C was an ongoing topic in Jeff and Joel's StackExchange podcast, which was actually somewhat entertaining. Time for another episode.
- smiler 15y agoI think stackoverflow is the first high volume .net site that have gone in depth publically with how they are solving their performance issues. Microsoft should be paying them for the continued great PR
- riffraff 15y agoit's great that they share their findings and part of their code, even if I'm not a .NET dev I find the blog posts really interesting and their approach/tools inspirational. But wasn't slashdot the first high volume site sharing their solutions to performance issues?
- smiler 15y agoNotice I said .NET. You can find plenty of presentations / blogs on php performance, ruby / rails performance, python / django performance but not many on .NET - I guess simply because most high-volume sites don't use .NET
- riffraff 15y agoOops, sorry I missed it
- noselasd 15y agoI'd say it's rather because most high volume sites using .NET are more commercialized , and often run by companies that's typically not been very open about anything they do internally.
- dbattaglia 15y agoAlso StackOverflow are one of the few .Net sites hackers/devs flock to and care about. I work on a large scale .Net web app, but I don't think many hacker news readers would really care about the application itself (and, as you pointed out, my company would probably kill me if I started blogging about the internals!).
- 15y ago
- mattmanser 15y agoLess a stop watch in the brain, more likely a case of simply pressing F12 in the browser. All of the consoles have profilers these days. I must admit I am a little surprised that tags are stored as a free text field.
- sams99 15y agoThe issue popped up once in a hundred times, in general people do not notice. Tags are stored both in a posttags table and free text. Fts is faster for many queries.
- sams99 15y agoI do admit it surprised me that we stored tags in an fts field when I started, but after a few months it made quite a bit of sense, the queries are just way faster.
- ableal 15y agoThese two phrases, early in the article, could give someone an interesting problem to solve: Our initial tag engine design was a GC nightmare. The worst possible type memory allocation. Lots of objects with lots of references that lived just enough time to get into generation 2 (which was collected every couple of minutes on our web servers). Interestingly, the worst possible scenario for the GC is quite common in our environment. We quite often cache information for 1 to 5 minutes. (Curiously, this is under a header mentioning "abuse of the garbage collector". I'd put it the other way around ...)
- david_a_r_kemp 15y agoLet me prefix this rant by saying I Love C# and .Net. I think it's a really good platform to build on, as demonstrated by it's growth outside of the Microsoft world. This article highlights why (good) C/C++ have such a hard time moving to C#/.Net. In .Net memory management is, 99.9999999999% of the time, fire and forget, whereas a big chunk of learning and programming C/C++ is about memory management. Most C, and some C++, programmers would have baulked at having a massive object graph in memory - releasing it becomes a massive headache, and that's before you even get into synchronisation issues, and the .Net team have done a really good job of taking this headache away (yes, I realise they 're not the first). However, we're now at the point where there are a lot of C# programmers who've never heard of a linker (by default, csc compiles and links in one go), have little concept of static vs dynamic linking, and, more importantly, don't have the first clue about memory management. This manifests itself in things like not using IDisposable properly/consistently, using the string.+ operator rather than using StringBuilder/StringWriter, and developers have no idea about the difference between class and struct, especially when it comes to parameter handling. Even books like (the excellent) C# in Depth are pretty light on the memory management side of things, deferring this to books about the CLI and DNR. Whereas, the truth is, if you're going to write a big application, you're going to have some concept of this.
- sams99 15y agoI agree with your criticism, many times developers of managed platforms (Java, .NET and even the less complex GCs like Ruby) forget that memory needs to be allocated and deallocated. Often we think the magic memory fairy is going to make everything OK, it is easy to fall in this trap even if you understand the underlying concepts, this is clearly not the case at high scale.
- MatthewPhillips 15y agoCould you explain what you mean about string + vs StringBuilder? I'm guilty of this one.
- marcgravell 15y agoActually, there's absolutely wrong with using + for individual expressions; it is actually the correct way of doing it (a + b + c + d compiles to String.Concat(a,b,c,d) ) - the problem is if you are doing this in a loop, where you get a lot of throwaway strings from intermediate concatenations. In a loop, you should concatenate via StringBuilder.
- MatthewPhillips 15y agoSlightly off topic, for any SO employees here, have you all ever experimented with using mono-fastcgi-server for any part of your sites and what were your findings? Assuming any of your code compiles under mono.
- sams99 15y agoWe have not, has anyone ran any benchmarks/tests comparing the mono GC to the Microsoft GC? I am pretty sure we would need to change large chunks of our system to run it in mono.
- kmontrose 15y agoMy understanding (from discussions with some Fog Creek devs who've done the mono dance for FogBugz) is that ASP.NET on mono is an awful lot of trouble, and doesn't run under IIS. http://www.mono-project.com/FAQ:_ASP.NET http://www.mono-project.com/FAQ:_ASP.NET Seems to validate the IIS bit.
- bluelu 15y agoWhy not write a script that periodically takes a machine of the loadbalancer, forces a GC on each of the servers, and then adds it back again. I think this would be a cleaner approach than changing the entire code base.
- nick_craver 15y agoThis wasn't a change of an entire codebase, only one portion of it (the tag engine). But more importantly, when only 6 web severs are running stackoverflow.com there's a decent chance a non-triggered GC happens stalling the servers in rotation anyway, granted this lessens the chance of a user-facing stall overall. Also, it complicates the build processes since that queue of rotating servers out gets involved, for example if 3-4 are in a GC rotation then 1-2 are starting the build loop, you're running only on 5 and 6, while that's fine and doesn't even hit 50% CPU, it's a little risky. Worse is you have the very real possibility of taking ALL servers down during a build and GC combo, which we'd never want to happen.
- scott_s 15y agoI disagree. Your solution attempts to fix the symptom. Their solution fixes the root cause of the problem.
- ajross 15y agoIt does? What is the "root cause" though? You seem to take it to mean the generation of references that must be traversed in gen2, but that sounds like just as much a hack to me: they modify the code in ways that don't represent the semantics of the problem to work around a limitation of the implementation. Hell, you could argue the "root cause" is the use of a garbage-collected environment in the first place. All popular GC implementations have latency issues like this. All of them. If you can't deal with occasional high latencies, you should identify that requirement before choosing Java or C#. All the solutions provided are just patching around that issue. I don't see anything in the post that looks like a root cause, and the GP's post has the advantage of being much simpler to implement.
- 15y ago
- steve8918 15y agoI'm not one to hate on C#, or be a programming language snob but this certainly looks like a case where poor performance and a lot of extra work were created simply because of a deficiency in a language. Languages like C# are advertised as being simple to use, and for the most part they are. They fit the bill in a lot of different use cases. But it certainly seems like to me that if you want to build a high-performance, scalable piece of software, you don't want to have to bang your head against wall and fight against deficiencies within the language implementation itself in order to get your software working properly.
- mwsherman 15y agoYou are conflating the language with the runtime, first. And second, we were experiencing the occasional page taking maybe one second, which we considered unacceptable but 99% of the world wouldn’t notice. (NB, languages like Ruby and PHP are lovely, but are over 10x slower than C# and Java.)
- steve8918 15y agoYes, sorry, I meant implementation of the language, ie. CLR. That being said, having to rewrite an entire engine because of the underbellies of the CLR seems to be dubious, especially in the case of a development environment that purports to make life easier for the developer. I have nothing against C#, one of the trading frameworks I use uses it exclusively (Ninjatrader) and it's pretty nice. It's just that I would hate to have invested weeks/months into a software project, using what I imagine would be tried-and-true software techniques, and then have to rework them to get over the particulars of the implementation. That being said, I guess all languages have similar problems. For example, when I was working on a backend server back in 2001, we found that our product was "leaking" memory on HP-UX, even though none of memory leak detectors found any problems, and the memory leak problem didn't occur in Solaris, AIX and Windows (we had a cross-platform server). It turns out on high-load systems, our app was allocating a bunch of small strings all over the place, and the base memory allocator for HP-UX was causing memory fragmentation over time. By switching to another memory allocator that handled this situation, it fixed the problem. I guess there aren't any C/C++ books that I've come across that say "don't allocate a millions of small memory blocks because your memory might fragment, even though you free them properly." So maybe I misspoke, I guess to some degree it does manifest itself in all implementations of languages.
- iam 15y agoIf they used an array of structs they would've cut down on the GC's graph traversal significantly. Another option I was surprised they didn't mention was to run GC.Collect periodically to clear out gen2.
- mrgoldenbrown 15y agoRunning Collect periodically probably won't help much in this scenario - the huge data structure isn't actually being collected during the GC.Collect. But the GC doesn't know this until it spikes the CPU while it walks a bajillion references. Since nothing is actually getting collected, if you ran GC.Collect periodically, you would just be incurring more lag with no benefit. I think your idea might help more in a different situation, one where you do have large amounts of memory getting reclaimed with each GC sweep.
- Sukotto 15y agoI can not stress how important it is to have queryable web logs. I'd love to read some (actionable) articles on how to set up some of the benchmarking / logging stuff that you guys have. For sure I'll check out .NET Memory Profiler and mini profiler linked in the article, but I'd be grateful if anyone wants to post more comprehensive info on where to start.
- platonichvn 15y agoGreat post. I always learn something new when the SO guys share their experiences with .NET in a large scale web application. Thanks for the link to the memory profiler. Hopefully it will be better than struggling with windbg.