11 ms·
You Got Your Web Browser in my Compiler
- octo_t 13y agoAwesome, in depth writeup (especially for someone who isn't at all familiar with Visual Studio / development on Windows in general). From your report this is 'use a builtin XML parser' and that xml parser has a dependency on IE?
- jjaredsimpson 13y agoshoutouts to xperf. it is truly an amazing tool.
- Flow 13y agoYears ago, I've used procmon.exe and saw VB6 load project files by doing tons of 1 byte reads... I wonder if the behavior in the article also applies to C# compiles?
- josteink 13y agoMSBuild has come a long way since the VB6 days. C# compilation is seriously fast compared to C++ and the likes (due to its proper staticly typed nature) and this allows the compiler to assert certain things quickly and not waste any more time on this. Given how .NET developers loves separating things into tons of DLLs and libraries (and how .NET makes this easy), I would be surprised if this part of the compilation wasn't as optimized as the rest of the process.
- jacobparker 13y ago> due to its proper staticaly typed nature I'm not sure what you mean by this - C++ compilation isn't slow because of a lack of static typing :) Here is some actual info on the subject: http://www.drdobbs.com/cpp/c-compilation-speed/228701711 http://www.drdobbs.com/cpp/c-compilation-speed/228701711
- raverbashing 13y agoCorrect, the speed has nothing to do with static typing or not. Go is fast to compile as well and statically typed See the post on the C++ preprocessor that was on HN some days ago as well
- Danieru 13y agoMy friend who I'd call the closest thing to a template programmer summarized the situation as such: "The core tenant of C++ has been to move work from runtime to to compile time." Both RAII (avoiding use of new & delete) and generics are strengths of C++ but also require heavy compile time work. RAII needs decent analysis and optimization or your program will thrash copying struts by value. Generics and templates force your compiler to run a second compilation in your frontend. This on top of C's copy/paste based include system and macros gives you what should be one of the slowest to compile procedural languages designable.
- 72deluxe 13y agoRAII doesn't necessarily mean avoiding new and delete. It just means cleaning up in destructors when you use new in the constructor or initialiser list or elsewhere. In particular, RAII means grabbing what you need in the constructor rather than haphazardly creating items in member functions. That's what I thought anyway?
- bhouston 13y agoC# compilation is fast in part because of the removal of headers and the replacement of them by well defined assemblies with interfaces. C# compilation is fast for the same reason Java compilation is fast. :) C++ compilation is sort of screwy in that to compile a new C++ file the compiler actually has re-compile all the definitions of the libraries you are using. It is insanely inefficient compared to just using explicitly defined and immutable API interfaces on Assemblies/Modules.
- noblethrasher 13y agoWhat's the advantage of doing it the C++ way?
- diek 13y agoNone really. It just inherited the C model of compilation.
- Locke1689 13y agoC# is fast because 1) we try really hard to make it fast :) and 2) it's not an optimizing compiler.
- Locke1689 13y agoC# doesn't have a static analyzer :) Right now the closest thing is FxCop or a third-party tool.
- bhouston 13y agoYou need to reduce the parallelism in your build. If it is possible in your build setup to be runing 144 concurrent builds you are trashing your caches. This isn't an optimization. You need to reduce the number of projects being built concurrently (or even the number of total projects.) While I advocate a bit of over allocation, you have it in the extreme and this is really not optimal at all. Also you need to get using the precompiled headers if you can -- sometimes this requires structuring your projects right to avoid lots of little projects. Even with large projects I usually have really low compilation times for C++. Here is a blog post I wrote about the right way to do this: http://exocortex.com/blog/speeding_up_cpp_builds_on_windows http://exocortex.com/blog/speeding_up_cpp_builds_on_windows
- ygra 13y agoI think Bruce already knows plenty about making builds fast ;-) The linked article demonstrates some ignorance as well by noting that command-line MSBuild builds are slower than from Visual Studio. I'd hazard a guess and say that they didn't find the /m option.
- taspeotis 13y agoTo my knowledge VS caches the MSBuild projects (.targets etc) while it's running. So it's possible that, even if VS and MSBuild have the same degree of parallelism, not having to parse the MSBuild XML gives VS an edge.
- 13y ago
- bazzargh 13y agoThis looks like the same bug, back in 2010: http://social.msdn.microsoft.com/Forums/fr-FR/155b51f7-60ea-41c2-a369-2e53c3d16a6f/msxml4dll-pops-up-a-warning-messagebox?forum=xmlandnetfx http://social.msdn.microsoft.com/Forums/fr-FR/155b51f7-60ea-... (ie that MSXML is calling URLMon, with unpredictable results). There's a comment at the bottom of that thread that suggests how to fix the bug if it's in your own code, but not if it's in Visual Studio... edited to add: related bugs crop up in many programs that use xml parsers, not just MS. Some combination of disabling validation, loading of external entities, or adding an entity resolver is usually needed (see eg these options for Xerces http://xerces.apache.org/xerces-j/features.html http://xerces.apache.org/xerces-j/features.html). The symptom there was usually errors in production that didn't occur on the developer's box because in production the network connection to grab the schema didn't work...or worse, a SOAP service that will fetch external entities in messages.
- acqq 13y agoIt looks it's urlmon.dll there, whereas Bruce observes mshtml.dll loaded.
- yuhong 13y agoIn this case, I can tell from the call stack that it is calling http://msdn.microsoft.com/en-us/library/ms762722%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/ms762722%28v=vs.85%2... with a res:// URL. One solution would be to load the resource manually using Win32 APIs, convert to BSTR, and then calling http://msdn.microsoft.com/en-us/library/ms754585%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/ms754585%28v=vs.85%2... instead. And yes don’t forget to disable loading of external entities too.
- prayerslayer 13y agoI wouldn't say that VS loads "Internet Explorer", that's a bit of an exaggeration. iexplore.exe is Internet Explorer, mshtml.dll holds the DOM implementation it's using [1,2]. Now that's still close to a web browser, but one may also use Trident to work e.g. with generic XML documents. I don't know if VS does that and whether this is a good idea or not though. [1] N. Zakas: High Performance Javascript, p. 35 [2] http://en.wikipedia.org/wiki/Trident_(layout_engine) http://en.wikipedia.org/wiki/Trident_(layout_engine)
- pjc50 13y agoHe does claim that it creates a window, even if that's a conceptual one rather than a physical one. Hence the lock contention in the desktop window manager.
- danielweber 13y agoThe question “why is my computer unresponsive when doing dozens of simultaneous compiles” is difficult to answer Doctor, doctor, it hurts when I do this!
- fafner 13y agoAh, the wonders of XML. Somehow somebody decided to use XML to store some text data. Because after all you already have a parser for this in the approved tools. So why bother using something different?! And look it does all the cool stuff like namespaces and validation and amazingly it can even fetch remote DTDs by loading IE...
- simias 13y agoI hate XML as much as the next guy but in this case I would blame a poorly designed (or maybe very misused) API. There's no reason why any file parsing library would end up fetching remote data without being explicitly asked to do so. Actually, it shouldn't even be the library's concern to fetch those files, an XML library has no business with networking. It's a security concern and a maintenance hell.
- acqq 13y ago> There's no reason why any file parsing library would end up fetching remote data It's not the API, it is the part of many of the specs, as it was thought to be a good idea once, specifically many standardizations involved additional definitions located at the http servers, example: http://msdn.microsoft.com/en-us/library/aa468557.aspx http://msdn.microsoft.com/en-us/library/aa468557.aspx Which was "poised to play a central role in the future of XML processing, especially in Web services where it serves as one of the fundamental pillars that higher levels of abstraction are built upon." So you had to implement it to be "conforming," and then to avoid overheads as "optimizations." Ironically, "/optimize" feature isn't optimized. The reason this it's not discovered earlier is that the Visual Studios which contain that option were priced more thousands of dollars (I don't know the what the currently cheapest version containing "/optimize" is -- anybody knows?).
- pwg 13y ago> There's no reason why any file parsing library would end up fetching remote data without being explicitly asked to do so. I would agree. But the XML spec. authors clearly disagreed with both of us: External Entities: XML 1.0: http://www.w3.org/TR/2008/REC-xml-20081126/#sec-external-ent http://www.w3.org/TR/2008/REC-xml-20081126/#sec-external-ent XML 1.1: http://www.w3.org/TR/2006/REC-xml11-20060816/#sec-external-ent http://www.w3.org/TR/2006/REC-xml11-20060816/#sec-external-e... This is "XML speak" for an "include" statement to something like cpp, with the exception that this "include" could end up performing remote network fetches to acquire that which is being included. So, technically, to be a proper, standards compliant XML parser, the parser has to at least submit requests to "fetch" these entities to the higher level code using the library, and let that code decide what to do about the "includes". As to why Microsoft's implementation is the way it is, absent a Raymond Chen blog post explaining the why, we can only guess.
- psuter 13y agoI am not sure whether it is still the case, but the Scala compiler used to have a dependency on Swing. The reason was that one of the potent -Y flags gave you access to an AST browser after the phase of your choice. Very convenient tool for compiler plugin developers, but a questionable dependency for most. (Edit: seems to have been taken away at least as of 2.10)
- jfim 13y agoscalac -Ybrowse:typer Foo.scala still works on 2.10.3. It's also pretty convenient when writing macros, not only for compiler plugin developers.
- rohansingh 13y agoThe unfortunate excerpts from this post that stood out for me: > Without access to a lot of source code I can’t tell exactly what is going on, but... > Until Microsoft fixes their compiler to not load their web browser it seems impossible to avoid this problem when doing lots of parallel builds. In my past life developing applications and services on Windows, this scenario was too common — something is broken, you can't quite tell why because you don't have the source, and you can't do anything but wait for it to be fixed or find some hacky workarounds (which is tougher to do without the source). This has been the real win on switching to an open source stack for me: you can take your tools apart, look inside them, and fix any issues. I know Microsoft has come further in this in the past few years, but it still sucks to be stuck in that kind of situation.
- duncan_bayne 13y agoVery true. Being able to fix tools and libraries is such a change. I'll never go back to Microsoft (or any other closed-source, proprietary) development if I can help it.
- ksk 13y agoMay I ask what kind of F/OSS software bugs have you found and fixed (as a user) which you were unable to with proprietary software? I don't need a list, I want to know more about the description of the bug.
- deleted 13y ago[deleted]
- duncan_bayne 13y agoThis is the specific example that happened not long after I abandoned Windows development altogether (I'd been using RoR since 2007, and Linux since 1995, but hadn't gone full time with both except through my own company): https://github.com/sparklemotion/nokogiri/pull/554 https://github.com/sparklemotion/nokogiri/pull/554 Gave me the warm fuzzies, even though I broke everything temporarily :)
- bitwize 13y agoThis is one of the things about Microsoft APIs that gets me. They all depend on each other in weird ways that have less to do with practicality than they do with getting people on board to use the various Microsoft APIs. For example, DirectX depends on COM. It doesn't have to, but they're both Microsoft technologies so why not? You're going to need to learn how to use COM anyway; it's the future. Systemd is designed this way too; that's why I hate it, despite the fact that it won. It seems architected with a "getting people on board with using certain tools" mindset rather than orthogonally providing functionality. But this... this takes the dependency tangle to whole new levels of comedy.
- Negitivefrags 13y agoCOM isn't really a API like DirectX is. It's a model for making APIs. Thus it makes sense that they would use it when creating an API. I can understand your argument in the general case, certainly, but it doesn't apply very well to COM.
- benched 13y agoLong time Microsoft and Windows app developer here. I just thought somebody should point out the obvious: that this is not even remotely unusual for a Microsoft app on a Microsoft operating system. You're just describing the normal architecture of the OS and the way apps work.
- jwise0 13y agoI think the true enlightenment in this article (for me, at least) is that Windows's tracing framework is far more powerful than I had realized. Being able to capture a system-level trace is an incredibly valuable tool. Until recently, I had thought that the only real tools out there to do this kind of analysis lived only on Solaris (DTrace). (As I recall, SystemTap gets part of the way, but does not reach far enough into userspace to really get a good view of what's going on -- is that still true?) It would be very interesting to augment Windows's performance tracing tools with some of the other things that DTrace has to offer -- pervasive low-overhead trace scripting seems like it would have made some of the other subsequent analyses that Bruce was working on much easier.
- wging 13y ago>Until recently, I had thought that the only real tools out there to do this kind of analysis lived only on Solaris (DTrace). FYI, there are ports of dtrace to systems other than Solaris.
- scintill76 13y agoSounds like it might be worth it to write a drop-in replacement DLL that doesn't parse XML, or at least in a way that doesn't require mshtml. The XML files that are being parsed appear to be static resources within the analyze DLL itself. A third party might even be able to patch the DLL to have pre-parsed resources instead of XML.
- muricula 13y agoIf true, this implies that recent versions of IE compiled with this compiler needed older versions of IE to compile themselves. IE is probably the first browser to have the honor of bootstrapping itself.