13 ms·
Joel on Solid State Disks
- inklesspen 18y agoAh, the cost of premature optimization: a couple thousand bucks, and a couple days of monkeying around. Better to first figure out where the bottleneck is (and maybe listen to your developer, since he thinks parallelization will help.)
- palish 18y agoBingo! Beat me to it.
- pkaler 18y agoListen to the last Stack Overflow podcast. Every single month that they ship FogBugz earlier is an extra $200k in revenue share for the developers. The developer may know how to solve the problem in code. But, Joel is the CEO. He has a better idea of how this affects the Balance Sheet and Income Statement. It's kinda, sorta his job.
- inklesspen 18y agoExcept that (a) he spent a couple days of his time being wrong about it, and (b) he then wrote a blog entry about how he was wrong about it. Imagine if instead he'd said to the guy "okay, spend three hours profiling the build process and get me some suggestions with time estimates", and they'd found some likely prospects, and three days from now he gets to post about how they'll be shipping the next release a month earlier because of the improvements they made to the build process.
- pkaler 18y agoListen to the Stack Overflow podcast. The developer estimated that it would take "a few weeks". A day and a half of Joel's time sees like a fair trade to do this experiment. Listen to the podcast. The conversation starts at about 16:00 mark if that helps. I am seeing a ton of comments that don't understand the context of the article. http://blog.stackoverflow.com/2009/03/podcast-47/ http://blog.stackoverflow.com/2009/03/podcast-47/
- jrockway 18y agoBut the developer wasn't going to take this brain-dead approach; he was going to do something that would actually help. Joel spent time and got nothing; the developer wanted to spend time doing something that would probably speed up the build.
- blahblahblah 18y agoExcept that he didn't "get nothing" out of the exercise. As he expected it would, the new drive did provide significant speedups for a variety of tasks and will undoubtedly improve his productivity enough to have been worth the time and money he spent on it. As it happens, it didn't improve the build speed specifically, but so what?
- icefox 18y agoYah, but I spent 10 minutes and read the ars article on the drives and got what took him several days to learn.
- sfg 18y agoHe got: snappier computers for everyone and a story to write about (key to his marketing).
- spolsky 18y agoNo matter how much you speed up a 30 second build, you're not going to save a month on the release process. Be realistic, now. SSDs provide so many other performance benefits--even just launching apps--that they're going to make our developers a lot happier anyway. And my time is far less valuable than a developer who is in the critical path to shipping.
- calambrac 18y agoA 15 second build over a 30 second build saves way more time than 15 seconds a build. It means that I'm more likely to hit compile after a smaller change before moving on to the next thing. It means I can iterate more rapidly on my approach to fixing a bug, keeping the issue hotter in my mind. It means I have half the time to get distracted by something shiny. Any time you can make the build perceptibly faster, you win big.
- Timothee 18y agoPlaying the devil's advocate, I would say that you can also look at it the other way and say that a developer might not build as often but instead make sure his/her code is right before building, thus being more careful about what s/he's writing. (just for the sake of argument ;))
- calambrac 18y agoRight, because I was advocating typing line noise until it makes it past the compiler. I can't count the number of times I used to make a tiny change that I didn't think was worth running the build for, only to have it be the first thing to pop up as wrong next time I compile (now I always run a build, because we made it super fast). And iterating over a bug, that can involve a lot of little changes, it can involve writing a ton of unit tests trying to duplicate the problem, it can involve subtle interactions that all seem right until you figure out what the issue is. Yes, you have to think, but sometimes you just need to churn through it, too, and frankly it's ridiculous to suggest that having a faster build process wouldn't help this. And I, personally, get distracted pretty easily. This comment courtesy the 5 minute test suite I'm plowing through in the background right now.
- mmelin 18y agoI think this was just an excuse for Joel to get SSD's for his laptop and workstation - and a good one at that!
- cubicle67 18y agoExactly. "Hmm... Love to get one of those new SSD drives. Just need to justify it somehow"
- spolsky 18y agoYeah, otherwise my boss never would have let me spend the money ;/
- mmelin 18y agoYes - I love everything about being self-employed except for the fact that my boss is an asshole.
- wglb 18y agoSame here. And he wants me to work all the time.
- Timothee 18y agoWell, in a few months, looking back at Fog Creek's spendings, you might have wondered who the hell authorized this spending... (http://www.joelonsoftware.com/items/2008/12/10.html http://www.joelonsoftware.com/items/2008/12/10.html) ;)
- pbz 18y agoWhat about compiling with VS? Any improvements, and if so by how much?
- wheels 18y agoWhat language is your compiler written in? Most languages have an off the shelf implementation of map / map-reduce or similar these days, which would seem well-suited to compiler tasks and greatly simplifies parallelizing jobs. We use map and map-reduce patterns in C++ heavily which is quite nice since if you've got that relatively small core of code thread safe you can mostly ignore thread safety in the major portions. You also can often speed up I/O that way since you can saturate both your I/O and CPU bandwidth, rather than alternating between waiting on them. I also take it that you're not compiling to individual object files since usually compilation since usual compile parallelization is based on farming out single object file compilations to multiple machines or processes. I wrote some small portions of "icecream" which is used for compilation parallelization by the guys at SUSE and there it would actually transmit over the network a full chroot environment so that even different Linux installs working on compatible architectures (later cross-compiler support was added as well) could execute the compiler jobs of rather different OSes.
- mynameishere 18y agoSeriously? Compiling? What language are they using that requires actual, regular, compile processes?
- blasdel 18y agoTheir bullshit private VB dialect that was someone's weekend project: Wasabi http://www.joelonsoftware.com/items/2006/09/01b.html http://www.joelonsoftware.com/items/2006/09/01b.html
- jcdreads 18y agoAre you angry that they used/wrote something called wasabi? Joel force you to read stuff that makes you mad? I mean, seriously, what's with all the random ire in the HN comments on this story?
- blasdel 18y ago1. Joel is trolling us, and we love it. 2. He is describing a stupid decision he made, which is framed around an unacknowledged stupid decision he made years ago (see 3). 3. His company uses a private language they wrote themselves, for totally ridiculous reasons, despite having written essays about how "you shouldn't let your programmers use any non-mainstream languages".
- wglb 18y agoWhen I first read about Wasabi, I kind of wondered. But then in a previous lifetime, I was with a company that wrote its own compiler. A few years after that, the CEO said "thank god for that compiler" as it relieved pressure on a target chip decision. Similarly, Wasabi has targeted a completely different end architecture with apparently little fuss. This is not a trivial result.
- jrockway 18y agoI think people are upset that Joel markets it as a good idea when, in fact, it is probably not that good of an idea. (Yes, he gets that 1% of the market that can't run whatever language they pick, but he has to maintain his own in-house programing language and toolchain, and train new developers to use a language that exists nowhere else in the world.)
- smanek 18y agoI'm a big fan of incremental builds - no point in recompiling 1M LOC when only a few source files have changed. Image based systems (like Common Lisp) are particularly nice because you can just recompile the functions that have changed and have the changes instantly take effect on a running image. Unfortunately, ASDF (the most common build/packaging system) doesn't have particularly good support for incremental builds yet, but it's an area of active developement (particularly with XCVB and other new build systems). But don't GCC, Java, and Visual Studio all support parallel builds pretty easily? I've never looked into it, but I know I've read articles about it. What sort of language/env. does FogCreek use? I vaguely remember hearing that they had written their own compiler in Java that transformed ASP to PHP or some other such ludicrous thing. Edit: Some googling showed that they decided their software had to run on Windows and Unix boxes, and that the best way to do that was to support ASP (pre .NET) on Windows and PHP on *nix. They already had a large ASP code base, so they wrote a compiler (named Thistle) that could translate a subset of ASP into PHP. Then, it seems like MS deprecated ASP (and they started to realize how much it sucked) so they wrote their own language (Wasabi) that is based on VBScript and can be compiled into ASP, PHP, or JS. My God. I would gouge my eyes out if I had to work there. From the fragments of code I've seen, it frankly seems horrible. See http://www.fogcreek.com/FogBugz/blog/category/Wasabi.aspx http://www.fogcreek.com/FogBugz/blog/category/Wasabi.aspx Although, I guess I can't really judge. I recently just wrote some ridiculously convoluted Java code that could automatically and transparently trick the JVM into making certain recursive functions Tail Call Optimizing (i.e., use constant space on the stack, regardless of how deep the recursion is) by transforming the function into automatically issuing/catching Exceptions to manually unwind the stack when I want to. Ah, I miss Common Lisp - I never knew how good I had it.
- thwarted 18y agoIf you define your dependencies correctly, make can do parallel builds (using the -j option). Of course you will have to use Makefiles in that case, which a lot of people don't like.
- smanek 18y agoYep, right you are. That seems like a much more reasonable way to go. Parallelize the build - not the compiler.
- blasdel 18y agoHe doesn't mention that the "compiler" that they're bottlenecking on is the one they wrote themselves, for "Wasabi" -- compiling their private language into VB or PHP. On top of that, you don't fucking parallelize compilers! You use a goddamn build system to do that (make, et. al.). Of course, that requires that your compiler be decent enough to be able to compile modules independently, and not have to recompile unmodified source. What do you want to bet that their compiler is just ridiculously awful?
- jrockway 18y agoThere’s an open source app called Clonezilla, which, I have to say, is only free if your time is worthless. What about dd if=/dev/old-disk of=/dev/new-disk bs=16M ?
- andrewtj 18y agoIt's trivial to folks familiar with nix-a-likes but for others it is too unlike the abstractions they're used to for it to be entertained as anything other than some sort of bizarre incantation.
- spolsky 18y agoIf disk sizes are different, which they almost always are, AFAIK dd doesn't quite hack it, otherwise I would have just booted off a Ubuntu live CD and done that.
- wmf 18y agoYes, after the dd you have to run parted to adjust the partition table and then possibly use some other tool to grow the filesystem.
- jrockway 18y agoIndeed. Without this, though, I think his system still would have booted. Incremental progress is certainly better than wasting two days pressing buttons in a GUI, at least IMHO. (Once the system boots, you can worry about using the extra bits on the disk.)
- blasdel 18y agoIf the new volume is smaller (which it almost always is when switching to a SSD), you'd need to shrink the filesystem before you block-copied it with dd.
- blasdel 18y agopartimage was the tool you were looking for. It natively understands NTFS, FAT, extX, Reiser, XFS, etc. and can resize as it images. By default it pops up a somewhat annoying ncurses interface, but it's quite a solid tool.
- tsally 18y agoSuggestion: check whether a process is IO bound first before spending hundreds of dollars trying to make it faster with SSDs.
- rgoddard 18y agoMy guess is that this was the cheaper option. It only cost Joel a couple of days and several hundred on the hard drive. What it did not do is interrupt his programmers. As Joel has already described, he is now a grunt worker. His job is to make it as easy for others to do their jobs. Their time is probably worth 10 to 100 times his, because they are the one actually making something that directly results in more sales and money for the company. So all Joel had to do was save a couple of hours of work for the programmers and he would come out ahead. Plus these additional side benefits: The programmer who was complaining does not feel like he is being ignored. Joel is now going to upgrade most of their computers, making it better for the entire company. So lots of warm fuzzies to go around, leading to happier programmers, leading to a better product.
- litewulf 18y agoThe cheaper option would be to run a compile and open any of the many different programs in Windows that tells you whether you're using lots of CPU, RAM or IO-waiting. Then you go AHA! I'll spend money on CPUs or making my compiler/build parallel.
- ighost 18y agoYou probably don't even need to check. If it's a compiler, chances are it won't be IO bound.
- wglb 18y agoNot necessarily, I think. If you have lots of memory, a second compile where stuff gets cached might be measurably faster.
- ilaksh 18y agoWith the current abundance of incremental and/or parallel tools, the fact that the compilation process takes up a significant amount of time is your first clue that there is something fundamentally wrong with the technology selection and overall thought process at this shop. Also, he could have saved $350 dollars and bought the 80 GB drive (unless he just HAS to store 40+ feature-length pirated movies on his boot/app drive?).
- deleted 18y ago[deleted]
- chiffonade 18y agoSo... he concludes that CPU bound tasks are CPU bound, and that parallelizing compilation (technically it would be the dispatch of compilation) is better? Every time I read this guy I fail to get what his allure to developers is.
- CRASCH 18y agoI can say my build times went down considerably. However I moved to a completely new machine. I didn't plan on writing an article so I don't have any official bench marks. I'm guessing they dropped from over a minute and a half for a full recompile to under 30 seconds. Incremental builds used to take about 20 seconds and now take between 2 and ten seconds. I went from a athlon X64 at 2.2Ghz and 2GB Ram with a 74GB raptor and another 74GB raptor for code to a Core i7 Extreme at 4Ghz and 12GB Ram with 2x80GB intel SSDs in raid 0 and 1 80GB intel SSD for code. A couple of important things to keep in mind. Intel is still the way to go. The other SSDs can actually be slower in write performance than a normal hard drive if the usage pattern is small files. That is pretty much what you get when you compile. One other point is that crappy anti-virus software can affect IO performance anywhere from 12X trend micro, 24X Norton. Yes that is Norton will reduce your IO by 2400% in my tests. AVG Free is 19%. So he could have negated the performance gain from the SSD by running bad anti-virus software. I can say my productivity feels like it nearly tripled. The important measurement from a programming perspective is not necessarily just the second or two here or there. My work flow changed. I used to hit build and walk out and get drink from the fridge in the garage and it would still be compiling. Now I can't even do anything but glance at my email. These interruptions sometimes take you out of the zone and it can take a long time to get back in the zone. Getting up could easily cost me fifteen minutes or even a half an hour. Just because I lost my train of thought. Update: I'm running windows 7 and visual studio 8 on this machine.
- MichaelApproved 18y agoSo he didn't fix the compile speed but he fixed the speed of everything else. I know every second I have to sit and wait for my hard drive to catch up is a second I'm spacing out instead of working. As a developer I just want to get my idea out and code. Anything that can be done to make that faster makes me a happier coder.
- richcollins 18y agoI guess you shouldn't always trust your assumptions, but did he really think compilation might be I/O bound?
- markessien 18y agoThis could have quickly been tested by using a memory disk and trying if the compile would go faster when the files are located there.
- jwr 18y agoTo summarize the article: yeah, SSDs are faster, but not quite for everything, because if you're CPU bound, SSD won't make your code run faster. So, why does this get upvoted, again?
- rozim 18y agoThey should try distcc