10 ms·

John Carmack on Static Code Analysis

286 points by niyazpk 14y ago

seanalltogether 14y ago
It's too bad he didn't post any samples, I'd love to see what kind of common mistakes he caught so I can avoid them myself.
- glaze 14y ago
  PVS-Studio's author has a blog where he lists examples from open source projects: http://www.viva64.com/en/b/ http://www.viva64.com/en/b/
hsmyers 14y ago
Having used PC-Lint almost all the way back to it's origins, I can testify to just how scary it is to run this on your code. Code you wrote as well as code written by teammates. In self defense, you HAVE to spend time tuning the system in terms of warnings and errors---otherwise you drown in a sea of depressing information. I liked John's comment about attempting 'green field' coding. It is a tremendously valuable process given the time. Great article, definite thumbs up.
speg 14y ago
Any recommendations for tools in web development? I use JS Lint, but what about server side languages?
- buster 14y ago
  https://github.com/SublimeLinter/SublimeLinter https://github.com/SublimeLinter/SublimeLinter
- sergimansilla 14y ago
  Cloud9 IDE has static JavaScript Analysis integrated in the editor. https://c9.io/ https://c9.io/
- OriginalSyn 14y ago
  Which ones? PHP has a bunch of them, http://stackoverflow.com/questions/378959/is-there-a-static-code-analyzer-like-lint-for-php-files http://stackoverflow.com/questions/378959/is-there-a-static-... Here's a some for asp.net, http://stackoverflow.com/questions/4419253/seeking-a-code-analysis-tool-for-asp-net http://stackoverflow.com/questions/4419253/seeking-a-code-an...
- karmi 14y ago
  Try http://codeclimate.com http://codeclimate.com
- Too 14y ago
  For PHP: phpLint[1], almost makes php into a type safe language. For .NET you have Code Contracts[2], this requires you to annotate pre- and postconditions for your functions but after that it's for example impossible to pass a possibly null into a function without compile time error. It can even do range-check of numerics after calculations etc. [1]http://www.icosaedro.it/phplint/ http://www.icosaedro.it/phplint/ [2]http://research.microsoft.com/en-us/projects/contracts/ http://research.microsoft.com/en-us/projects/contracts/
  - nahname 14y ago
    ReSharper is the best lint tool you can get for dotnet development. It literally makes you a better programmer that writes cleaner code. It really sounds like JetBrains needs to make a similar tool for C++
    - adamzochowski 14y ago
      How does it compare to fxCop?
      - nahname 14y ago
        Night and day. FxCop requires days of customization to include something into your build step. Even then, the recommendations are weak at best. Furthermore, think back to Carmack's comment on how static analysis has to at least be in your build step. ReSharper has much better recommendations that require almost no tweaking (variable names you might disagree with). The best part, it tells you these recommendations at the time of writing. This is one of ReSharpers best features (IMO). Not only do programmers write better code with, they often write code that works the first time and compiling becomes just a technicality. Code improvement is noticeably better. So much so, that when my company is hiring and I am reviewing the candidates submission; I can immediately tell whether or not they were using ReSharper. Cannot recommend this tool enough. I feel crippled without it in VS.
        rbellio 14y ago
- MBlume 14y ago
  Pylint is fantastic Googling "$LANGUAGE lint" will probably find what you need.
  - pnathan 14y ago
    I've personally found pychecker and pyflakes to be much more useful out of the gate than pylint.
- viscanti 14y ago
  Flake8Lint (available for Vim and Sublime Text 2) works really well for me, and my co-workers. The optional complexity checking is a nice bonus. You also get the PEP 8 linting and some logic checks. It hurts at first, to see all the errors pop up, but once you tweak the settings (maybe you don't care so much about PEP 8 spacing and you'd like your equals signs to all match up, in which case it's simple to ignore those issues), it can help keep code consistent. I think the big win is on large projects with a number of people writing code. Having an enforceable "style" really helps cut down on bugs and makes everything more readable.
- farnulfo 14y ago
  Sonar from SonarSource can natively analyzed java code (open source). Additional plugins are available for other languages (http://docs.codehaus.org/display/SONAR/Plugin+Library http://docs.codehaus.org/display/SONAR/Plugin+Library). You can see a live demo for a lot of open source project at http://nemo.sonarsource.org http://nemo.sonarsource.org .
- Joeri 14y ago
  I can strongly recommend phpstorm for javascript and php inspections. Errors are highlighted as you type (including but not limited to what jslint catches) and quite a few can be automatically fixed. They also have a good spellcheck for misspelled variable names. No more poeple arrays in my code :)
- viggity 14y ago
  I wrote a static code analysis tool for .net (c#/vb) www.nitriq.com
- jaimefjorge 14y ago
  I would love for you to try qamine (www.qamine.com). I'm giving a trial for people to test the tool before I launch it. Feel free to join.
quaunaut 14y ago
I'm a web developer interested in diving into graphics programming sometime in the next year, but this made me stop and wonder: > If you aren’t deeply frightened about all the additional issues raised by concurrency, you aren’t thinking about it hard enough. Why exactly is that?
- amackera 14y ago
  The error conditions that cause bugs in concurrent code are difficult to replicate since concurrent code can have so many additional states of operation. It's also harder to program, especially in languages / environments that don't explicitly encourage it right from the beginning.
  - prodigal_erik 14y ago
    It's important to remember that some of the possible states arise from the memory model rather than what your code intentionally does. Say a producer thread fills a buffer and queues work, and a consumer thread receives the queue entry and reads the buffer. Does the consumer read any or all of what the producer intended? It depends on whether they were scheduled in different cores, and how full their caches were, and whether there was a memory barrier that's global (which is slow and not always needed) or only applies to the queue itself. Good platforms trade away just enough performance to guarantee correctness here.
- pfraze 14y ago
  Concurrency is complex. You have to sync between threads/processes/actors, and the sequence of a function doesn't include everything happening to the data in use. That makes it harder to predict and debug the program.
dexen 14y ago
Dear Lazyweb, the `Controler' part of my main codebase consists of interwoven PHP and MySQL. Is there static analysis tool that understands both, one in relation to the other?
- Too 14y ago
  Good question, there aren't exactly any good tools that interweave several languages into the same environment. Not just for php-mysql but also for php-html or html-javascript or php-javascript. Phpstorm gives you intellisense for sql-strings embedded in your php code but that's the most advanced cross-language-feature i've seen, except for LINQ in C#. Imagine a tool that could analyze this: //sql table T having columns int x, string y //php $data = sql_fetch_from_column(T) //javascript var d = <?=json_encode($data)?> for(a in d) var b = a.x / a.y; //Type error, int division by string not defined
- Darmani 14y ago
  The closest thing I'm aware of is a tool by the University of Tartu which does this for Java, described in the paper "An Interactive Tool for Analyzing Embedded SQL Quries." It can find the SQL errors (misspelled "name" and missing space before "AND") in the following code: public Statement ageFilter(int minimalAge) throws SQLException { Connection conn = getConnection(); String sql = "SELECT name FROM people WHERE nmae IS NOT NULL"; if (minimalAge > 0) sql += "AND (age >= " + minimalAge + ")"; return conn.prepareStatement(sql); } There are a lot of really interesting challenges in doing this. Their tool uses a powerful technique called "abstract parsing" which can analyze all possible parse trees created by running the SQL parser on a string produced by Java string operations. It's pretty impressive what modern static analysis can do, and just how weak the linters we've gotten used to are in comparison.
- mark_story 14y ago
  Probably not, but you might want to give phpcpd, pdepend and pmd a try. They're all open source static analysis tools for PHP, and will find a number of potential issues. I also find getting a good set of rules for phpcs is extremely helpful in PHP. You can use its sniffs to catch past mistakes and enforce consistent coding standards.
estebank 14y ago
It's a great article by an insightful individual. If you haven't read it, do so. You can read further discussion on this 270 days old article at http://news.ycombinator.com/item?id=3388290 http://news.ycombinator.com/item?id=3388290
pnathan 14y ago
A couple things: - Competitors with Coverity are CodeSonar[1] and Klocwork[2]. I've not seen Klocwork output, but CodeSonar and Coverity are in the same area of quality, with differing strengths. I can not recommend static analysis highly enough if you have a C/C++/Java/C# database. It's very expensive (well into five figures according to Carmack), but how expensive is a bug? What if you have your entire codebase checked daily for bugs? Consider the effect on your quality culture. :-) - The fact that you are paying "well into five figures" for a tool that essentially covers up design deficiencies in your language should start sounding alarm bells in your head. The proposition more or less goes, "To have reliable C++ code in certain areas, you need a static analyzer; to gain that same advantage in Haskell costs you nothing more than GHC". Of course Haskell doesn't have certain C/C++ capabilities; but it's worth meditating on for your next application, particularly if bugs are more important than performance. N.b- I don't know the ML family enough to say one way or the other in this regard. :-) [1] http://www.grammatech.com http://www.grammatech.com [2] http://www.klocwork.com http://www.klocwork.com
- stonemetal 14y ago
  If you don't mind MS, their code analysis tools come in the express(free) version of VS 2012.
  - pnathan 14y ago
    Personally, I mind MS. But that's neither here nor there. My understanding is that their tools don't do "whole program analysis". Is that incorrect (I'd love to hear that I am!)?
    - stonemetal 14y ago
      I can't say I know enough about it to say either way, in the article Carmack says: Technically, the Microsoft tool only performs local analysis, so it should be inferior to Coverity’s global analysis, but enabling it poured out mountains of errors, far more than Coverity reported. True, there were lots of false positives, but there was also a lot of scary, scary stuff.
santaragolabs 14y ago
So I've dealt with dozens of Fortune-100 companies implementing and using static code analysis tools. They can and will help but in general I feel that these tools are not much more than the code-equivalent of the syntax- and grammar- checker in your word processing software. I've been doing manual code reviews for a living now (mostly security related) for roughly 3 years now and while I get assisted from time to time by code analysis tools I still find heaps of bugs not caught by any of the tools mentioned by Carmack. The biggest issue for a development shop is to properly integrate these tools and to not overwhelm developers with too much false positives. I've had cases where a developer got a 1500 page PDF spit out by one of these static analysis tools. After spending two weeks going through everything the developer ended up with 50 pages of actual bugs; the rest were describing false positives. Then I got on-site and I still logged dozens and dozens of security-related bugs that the static analysis tools failed to find. Edit: also consider that one even needs a SAT solver to even do proper C-style preprocessor dependency checking. A lot of these code analysis tools are being run on debug builds only and then there when the release build is being made these tools are not being run meaning they fail to catch a lot of issues. It's insanely hard to write proper code analysis tools and static source code analysis tools which do not integrate with the compilation process I wouldn't trust at all. Nowadays with clang there are very nice possibilities for someone to write your own simple checks and integrate them into the build process. But even clang doesn't expose everything about the preprocessor that you might want to have from a static code analysis perspective.
- btilly 14y ago
  Beware the false false positive! A couple of years ago I ran across an article from Coverity about the challenges that they had getting people to use their tool. One of the most interesting was that the tool would find real bugs, but the developer wouldn't understand that the bug actually was real, and would say, "This tool is crap." Then they would not get a sale. They had this problem particularly strongly with race conditions. There are a number of checks that they took out because, even though they were finding real bugs, developers were convincing themselves that the bugs were not real even though they really were. It really does not help that the developers who are asked to evaluate are likely to be the same people who made the mistakes in the first place, so all kinds of defensive behavior are to be expected.
gaius 14y ago
Microsoft feels that game quality on the 360 impacts them more than application quality on Windows does. :-) Many a true word spoken in jest!
- cbs 14y ago
  Carmack is no doubt familiar with Microsoft's game cert process, it probably wasn't in jest at all. It does impact Microsoft more, because Microsoft has taken on responsibility for games to be good (well, meet certain baselines for various definitions of good) through their TCRs.
  - gaius 14y ago
    'Twas his smiley on the end, I didn't add it.
- georgemcbay 14y ago
  This actually makes a lot of sense. I mean, Microsoft is a gatekeeper on the Xbox 360 and enforces quite a bit of forced QA on every game officially released for the platform. If an Xbox 360 game crashes, I view it as a Microsoft failing, even if the game isn't developed by them. OTOH, if a non-Microsoft Windows app crashes, I don't blame Microsoft at all but rather the app developer.
zwieback 14y ago
The article mirrors my recent experience 100%. We've got a Coverity license and I've started using it recently. Luckily, our code base is relatively small, it's straight C and embedded (no mallocs, no OS). Even in this extremely simple environment it's shocking how many errors Coverity can ferret out. The false-positives are a problem and the general advice to get started is to initially ignore all existing bugs and focus on avoiding adding new bugs. Then, when you get the hang of writing code that passes the checks you go back and look for the worst of the older bugs, etc.
cpeterso 14y ago
FindBugs [1] is a great code analysis tool for Java. It's free, open source, and supports plugins for writing your own checks. The FindBugs site reports an interesting story from a Google test day: "Google held a global "fixit" day using UMD's FindBugs static analysis tool for finding coding mistakes in Java software. More than 700 engineers ran FindBugs from dozens of offices. Engineers have already submitted changes that made more than 1,100 of the 3,800 issues go away. Engineers filed more than 1,700 bug reports, of which 600 have already been marked as fixed. Work continues on addressing the issues raised by the fixit, and on supporting the integration of FindBugs into the software development process at Google." [1] http://findbugs.sourceforge.net/ http://findbugs.sourceforge.net/
- jadc 14y ago
  For Java code, I am a big fan of Sonar Source [1]. It is open source, is able to leverage checkers such as FindBugs, and has integration with code coverage tools such as Cobertura. I have found the clean dashboard to be a great boon, and have never felt intimated by the reported warnings like I've been with Coverity et al. [1] http://www.sonarsource.org http://www.sonarsource.org
  - npsimons 14y ago
    As long as we're on an open source kick here (since I prefer open source over proprietary): http://checkstyle.sourceforge.net/ http://checkstyle.sourceforge.net/ http://cppcheck.sourceforge.net/ http://cppcheck.sourceforge.net/ http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py http://google-styleguide.googlecode.com/svn/trunk/cpplint/cp... http://www.dwheeler.com/flawfinder/ http://www.dwheeler.com/flawfinder/ http://www.dsm.fordham.edu/~ftnchek/ http://www.dsm.fordham.edu/~ftnchek/ http://community.haskell.org/~ndm/hlint/ http://community.haskell.org/~ndm/hlint/ http://jlint.sourceforge.net/ http://jlint.sourceforge.net/ http://search.cpan.org/dist/Perl-Critic/bin/perlcritic http://search.cpan.org/dist/Perl-Critic/bin/perlcritic http://deployingradius.com/pscan/ http://deployingradius.com/pscan/ http://pychecker.sourceforge.net/ http://pychecker.sourceforge.net/ http://pypi.python.org/pypi/pyflakes http://pypi.python.org/pypi/pyflakes http://www.logilab.org/857 http://www.logilab.org/857 http://packages.debian.org/squeeze/rats http://packages.debian.org/squeeze/rats http://www.splint.org/ http://www.splint.org/ http://www.incava.org/projects/303902593 http://www.incava.org/projects/303902593 http://packages.debian.org/stable/devel/icheck http://packages.debian.org/stable/devel/icheck Please note: I have not used all of these; one day I was bored and decided to see how many code checking tools I could find packaged for Debian and put them all in a script with the warnings cranked to the max and make that part of my default nightly build and test script. Of course, you can also crank up warnings on your compilers and interpreters as well (-Wall -Wextra, python -tt3 with warnings.simplefilter("always"), perl -tTW with use strict and use warnings, etc)
rabidsnail 14y ago
Has anyone tried running their commits through CRM114, marking removals in bugfix commits as "bad"?
- eru 14y ago
  You could also try to train some statistical model and see whether you can find some common ground in commits that introduce bugs. And then fire that person. (Or rather, see if big commits introduce more bugs than short ones, or whatever else you might find.)
mrich 14y ago
For C/C++, also try just compiling with clang. It has great diagnostics. Also it has the static analyer whose C++ support just improved greatly in trunk.
Flow 14y ago
How good is the code analysis in IntelliJ compared to these tools?
- calpaterson 14y ago
  Code analysis is nice but stand alone static analysis tools are a lot more advanced. Even the free ones, like FindBugs catch a lot of issues.
Havoc 14y ago
Pretty sure I've read this exact thing ~1 year ago.
- samspot 14y ago
  Yeah it is old, you can check the article date.
js2 14y ago
(2011) Original HN discussion http://news.ycombinator.com/item?id=3388290 http://news.ycombinator.com/item?id=3388290
egonschiele 14y ago
I've been wanting something like this for Ruby for some time now. Since it's dynamically typed and ridiculously easy to monkey-patch, Ruby is a much harder challenge than C++. The two best efforts I have found are Diamondback Ruby (http://www.cs.umd.edu/projects/PL/druby/ http://www.cs.umd.edu/projects/PL/druby/) and Laser (http://carboni.ca/projects/p/laser http://carboni.ca/projects/p/laser)...but they mostly try to add static type-checking to Ruby code. After looking at these I implemented a contracts library for Ruby (https://github.com/egonSchiele/contracts.ruby https://github.com/egonSchiele/contracts.ruby) to get myself some better dynamic checking. The next step is to use the annotations for the contracts library to do better static code analysis. One thing I'm working on is generating tests automatically based on the contract annotations. But I've got a long way to go : ( If anyone knows about other projects that are working on static analysis for Ruby I'd be very interested in hearing about them!
- pjungwir 14y ago
  Looked at your Contracts library for Ruby. I was just thinking yesterday about starting a project like this! Some thoughts: - I like that the contracts are right there in the code and not in a separate file. - The actual Contract methods only run once upon loading each class, so they don't cause a performance hit. I'd like a way to make them all noops (so they don't add any instrumentation) when running in "production". - My thought is that if you can define pre- and post-conditions as you're doing, then you can cut out a lot of the checks in your tests, so that as long as the tests exercise the code, you can rely on the post-conditions to do quite a bit of the validation. Plus you're validating outputs all over the place constantly, so you'd probably catch edge cases you never even expected. - I'm disappointed that your system seems to offer little more than type checking (unless I'm mistaken). I'd like something that gives me true pre- and post-conditions: basically passing a block of Ruby code for each one, with common idioms abbreviated so I don't need to write a whole block if the conditions are simple. Good luck on the project!
  - egonschiele 14y ago
    Great thoughts! > I'd like a way to make them all noops Sure! There's a default failure callback that gets called when a contract fails. You can just clear this callback in prod: if ENV == PRODUCTION class Contract def self.failure_callback(data) # blank end end end > I'm disappointed that your system seems to offer little more than type checking Nay! You can write your own contracts to do anything you want. So yes, lambdas are allowed in contracts: Contract lambda { |x| x.is_valid_email? } => Bool def save_email(email) # save an email end > with common idioms abbreviated I provide some common idioms already, but you can make your own abbreviations: ValidEmail = lambda { |x| x.is_valid_email? } Contract ValidEmail => Bool def save_email(email) Let me know if you have any other questions :)
stcredzero 14y ago
> We had a period where one of the projects accidentally got the static analysis option turned off for a few months, and when I noticed and re-enabled it, there were piles of new errors that had been introduced in the interim. Similarly, programmers working just on the PC or PS3 would check in faulty code and not realize it until they got a “broken 360 build” email report. These were demonstrations that the normal development operations were continuously producing these classes of errors, and /analyze was effectively shielding us from a lot of them. Something which corroborates this: When penetration testers break into systems, they're often using new 0-day exploits. Think about that. Most of today's software development practice produces such a steady stream of low-level bugs, that penetration testers can assume that they're there! > Trying to retrofit a substantial codebase to be clean at maximum levels in PC-Lint is probably futile. I did some “green field” programming where I slavishly made every picky lint comment go away, but it is more of an adjustment than most experienced C/C++ programmers are going to want to make. I still need to spend some time trying to determine the right set of warnings to enable to let us get the most benefit from PC-Lint. This could be encouraged using game dynamics. Have a mechanism where a programmer can mark parts of the codebase "green-field." A programmer's "green-field score" consists of the number of lines of green-field code (or statements, whichever lousy metric you want) that he's successfully compiled with no warnings whatsoever. Combine this with random sampling code walkthroughs, which has many benefits but will also catch boilerplate, auto-generated, or copy-paste programming by a "Wally" who's trying to "write himself a new minivan."
deleted 14y ago
[deleted]
jaimefjorge 14y ago
This article is one of the reason I created my tool: http://www.qamine.com http://www.qamine.com Qamine integrates directly with github and is designed to be used by small and medium companies that cannot afford those expensive tools.
- Darmani 14y ago
  Your tool looks pretty cool and useful, but it seems quite a bit different than tools like Coverity and Astree which reason about the behavior of programs under all possible inputs. Coverity can detect race conditions. Your tool looks like it's using technology closer to that of Refactoring Crawler, which was a quite impressive achievement, but not a static analysis tool.
FredBrach 14y ago
There have been plenty of hugely successful and highly regarded titles that were filled with bugs and crashed a lot I think it's false, and a huge mistake. There is rare cases, but not plenty. Video games (I mean video games for core gamers) are products that demand first and above all quality to be successful. There is rather plenty of common games with a huge quality which become hits (Kingdom Rush for example or Starcraft which was finally kind of common in its time). One of the rules in the delovepment process at Blizzard is that they ship a game when it has less than 100 known bugs. Also, I would add that, it seems that, ID software did not make a successful games since Doom. Quake wasn't a commercial success, Quake2, Quake3, Doom3 and Rage neither (that's why ID software has been bought for only $100M). After all, ID Software lost one of its core value co-founder a long time ago (John Romero) who was responsible of the gameplay of ID's games... Quality in video games are everything, that's really my opinion. It's also really an edge for every indy developper which want to start a company in this sector, cases are countless.
- bgaluszka 14y ago
  > Anyway, nobody here, is still playing one of these games - starcraft for example, is a game still played. Well, I wouldn't say nobody. I, for one, play/watch Quakeworld and Quakelive on a regular basis.
  - FredBrach 14y ago
    Ok you're right. I've deleted that. I say right because we could discuss hours about what nobody means. Oh just a question, so did you bought quake?
    - bgaluszka 14y ago
      I bought them and won them on tournaments so I own few of those :)
      - deleted 14y ago
        [deleted]
nfriedly 14y ago
Funny timing, I just got jslint turned back on in our build today! (well, jsHint now due to the 'for(var i=0...' failing even with -vars enabled, but I digress...). Another dev and I spent literally the entire day fixing issues - and we had jslint running on every checkin until a few months ago! But, it was worth it. It feels great to know that those bugs won't happen again without a failing build :)
- ahoge 14y ago
  >due to the 'for(var i=0...' failing even with -vars enabled That's because `for(var i=0...` is completely bogus. What if you have two of those loops? Are you going to put `var` in the first one, but not in the second one? Are you going to use 'i' in the first one and 'k' in the second one? What if you need some refactoring and those two loops switch places? It's fine if you like Java, but JavaScript is different. When in Rome, do as the Romans do. I also wish there would be block scope, but merely pretending there is doesn't make it so. It's better if you declare all of your variables at the very top of the innermost function, because that's where they effectively are (thanks to hoisting). It's better if your code reflects how it's actually working, because that will always make it easier to comprehend. Once there is `let`, you should switch to Java's "declare on first use" rule and never use `var` again.
  - nfriedly 14y ago
    Right, I understand why. And, if I have two in the same scope, jshint will rightly alert me that I have an error. But I don't have this issue once in ~20k lines of js. So in my case, jslint not honoring its own -vars option was just a nuisance. And while I agree that declaring vars at the top is closer to whats actually happening, I disagree that it makes it easier to comprehend.
    - ahoge 14y ago
      function foo () { x = 5; // totally fine ... var x; // because we declare it here }
  - Joeri 14y ago

286 points by niyazpk 14y ago

It's too bad he didn't post any samples, I'd love to see what kind of common mistakes he caught so I can avoid them myself.

glaze 14y ago
PVS-Studio's author has a blog where he lists examples from open source projects: http://www.viva64.com/en/b/ http://www.viva64.com/en/b/

hsmyers 14y ago

Having used PC-Lint almost all the way back to it's origins, I can testify to just how scary it is to run this on your code. Code you wrote as well as code written by teammates. In self defense, you HAVE to spend time tuning the system in terms of warnings and errors---otherwise you drown in a sea of depressing information. I liked John's comment about attempting 'green field' coding. It is a tremendously valuable process given the time. Great article, definite thumbs up.

speg 14y ago

Any recommendations for tools in web development? I use JS Lint, but what about server side languages?

buster 14y ago
https://github.com/SublimeLinter/SublimeLinter https://github.com/SublimeLinter/SublimeLinter
sergimansilla 14y ago
Cloud9 IDE has static JavaScript Analysis integrated in the editor. https://c9.io/ https://c9.io/
OriginalSyn 14y ago
Which ones? PHP has a bunch of them, http://stackoverflow.com/questions/378959/is-there-a-static-code-analyzer-like-lint-for-php-files http://stackoverflow.com/questions/378959/is-there-a-static-... Here's a some for asp.net, http://stackoverflow.com/questions/4419253/seeking-a-code-analysis-tool-for-asp-net http://stackoverflow.com/questions/4419253/seeking-a-code-an...
karmi 14y ago
Try http://codeclimate.com http://codeclimate.com
Too 14y ago
For PHP: phpLint[1], almost makes php into a type safe language. For .NET you have Code Contracts[2], this requires you to annotate pre- and postconditions for your functions but after that it's for example impossible to pass a possibly null into a function without compile time error. It can even do range-check of numerics after calculations etc. [1]http://www.icosaedro.it/phplint/ http://www.icosaedro.it/phplint/ [2]http://research.microsoft.com/en-us/projects/contracts/ http://research.microsoft.com/en-us/projects/contracts/
- nahname 14y ago
  ReSharper is the best lint tool you can get for dotnet development. It literally makes you a better programmer that writes cleaner code. It really sounds like JetBrains needs to make a similar tool for C++
  - adamzochowski 14y ago
    How does it compare to fxCop?
    - nahname 14y ago
      Night and day. FxCop requires days of customization to include something into your build step. Even then, the recommendations are weak at best. Furthermore, think back to Carmack's comment on how static analysis has to at least be in your build step. ReSharper has much better recommendations that require almost no tweaking (variable names you might disagree with). The best part, it tells you these recommendations at the time of writing. This is one of ReSharpers best features (IMO). Not only do programmers write better code with, they often write code that works the first time and compiling becomes just a technicality. Code improvement is noticeably better. So much so, that when my company is hiring and I am reviewing the candidates submission; I can immediately tell whether or not they were using ReSharper. Cannot recommend this tool enough. I feel crippled without it in VS.
      - rbellio 14y ago
MBlume 14y ago
Pylint is fantastic Googling "$LANGUAGE lint" will probably find what you need.
- pnathan 14y ago
  I've personally found pychecker and pyflakes to be much more useful out of the gate than pylint.
viscanti 14y ago
Flake8Lint (available for Vim and Sublime Text 2) works really well for me, and my co-workers. The optional complexity checking is a nice bonus. You also get the PEP 8 linting and some logic checks. It hurts at first, to see all the errors pop up, but once you tweak the settings (maybe you don't care so much about PEP 8 spacing and you'd like your equals signs to all match up, in which case it's simple to ignore those issues), it can help keep code consistent. I think the big win is on large projects with a number of people writing code. Having an enforceable "style" really helps cut down on bugs and makes everything more readable.
farnulfo 14y ago
Sonar from SonarSource can natively analyzed java code (open source). Additional plugins are available for other languages (http://docs.codehaus.org/display/SONAR/Plugin+Library http://docs.codehaus.org/display/SONAR/Plugin+Library). You can see a live demo for a lot of open source project at http://nemo.sonarsource.org http://nemo.sonarsource.org .
Joeri 14y ago
I can strongly recommend phpstorm for javascript and php inspections. Errors are highlighted as you type (including but not limited to what jslint catches) and quite a few can be automatically fixed. They also have a good spellcheck for misspelled variable names. No more poeple arrays in my code :)
viggity 14y ago
I wrote a static code analysis tool for .net (c#/vb) www.nitriq.com
jaimefjorge 14y ago
I would love for you to try qamine (www.qamine.com). I'm giving a trial for people to test the tool before I launch it. Feel free to join.

quaunaut 14y ago

I'm a web developer interested in diving into graphics programming sometime in the next year, but this made me stop and wonder: > If you aren’t deeply frightened about all the additional issues raised by concurrency, you aren’t thinking about it hard enough. Why exactly is that?

amackera 14y ago
The error conditions that cause bugs in concurrent code are difficult to replicate since concurrent code can have so many additional states of operation. It's also harder to program, especially in languages / environments that don't explicitly encourage it right from the beginning.
- prodigal_erik 14y ago
  It's important to remember that some of the possible states arise from the memory model rather than what your code intentionally does. Say a producer thread fills a buffer and queues work, and a consumer thread receives the queue entry and reads the buffer. Does the consumer read any or all of what the producer intended? It depends on whether they were scheduled in different cores, and how full their caches were, and whether there was a memory barrier that's global (which is slow and not always needed) or only applies to the queue itself. Good platforms trade away just enough performance to guarantee correctness here.
pfraze 14y ago
Concurrency is complex. You have to sync between threads/processes/actors, and the sequence of a function doesn't include everything happening to the data in use. That makes it harder to predict and debug the program.

dexen 14y ago

Dear Lazyweb, the `Controler' part of my main codebase consists of interwoven PHP and MySQL. Is there static analysis tool that understands both, one in relation to the other?

Too 14y ago
Good question, there aren't exactly any good tools that interweave several languages into the same environment. Not just for php-mysql but also for php-html or html-javascript or php-javascript. Phpstorm gives you intellisense for sql-strings embedded in your php code but that's the most advanced cross-language-feature i've seen, except for LINQ in C#. Imagine a tool that could analyze this: //sql table T having columns int x, string y //php $data = sql_fetch_from_column(T) //javascript var d = <?=json_encode($data)?> for(a in d) var b = a.x / a.y; //Type error, int division by string not defined
Darmani 14y ago
The closest thing I'm aware of is a tool by the University of Tartu which does this for Java, described in the paper "An Interactive Tool for Analyzing Embedded SQL Quries." It can find the SQL errors (misspelled "name" and missing space before "AND") in the following code: public Statement ageFilter(int minimalAge) throws SQLException { Connection conn = getConnection(); String sql = "SELECT name FROM people WHERE nmae IS NOT NULL"; if (minimalAge > 0) sql += "AND (age >= " + minimalAge + ")"; return conn.prepareStatement(sql); } There are a lot of really interesting challenges in doing this. Their tool uses a powerful technique called "abstract parsing" which can analyze all possible parse trees created by running the SQL parser on a string produced by Java string operations. It's pretty impressive what modern static analysis can do, and just how weak the linters we've gotten used to are in comparison.
mark_story 14y ago
Probably not, but you might want to give phpcpd, pdepend and pmd a try. They're all open source static analysis tools for PHP, and will find a number of potential issues. I also find getting a good set of rules for phpcs is extremely helpful in PHP. You can use its sniffs to catch past mistakes and enforce consistent coding standards.

estebank 14y ago

It's a great article by an insightful individual. If you haven't read it, do so. You can read further discussion on this 270 days old article at http://news.ycombinator.com/item?id=3388290 http://news.ycombinator.com/item?id=3388290

pnathan 14y ago

A couple things: - Competitors with Coverity are CodeSonar[1] and Klocwork[2]. I've not seen Klocwork output, but CodeSonar and Coverity are in the same area of quality, with differing strengths. I can not recommend static analysis highly enough if you have a C/C++/Java/C# database. It's very expensive (well into five figures according to Carmack), but how expensive is a bug? What if you have your entire codebase checked daily for bugs? Consider the effect on your quality culture. :-) - The fact that you are paying "well into five figures" for a tool that essentially covers up design deficiencies in your language should start sounding alarm bells in your head. The proposition more or less goes, "To have reliable C++ code in certain areas, you need a static analyzer; to gain that same advantage in Haskell costs you nothing more than GHC". Of course Haskell doesn't have certain C/C++ capabilities; but it's worth meditating on for your next application, particularly if bugs are more important than performance. N.b- I don't know the ML family enough to say one way or the other in this regard. :-) [1] http://www.grammatech.com http://www.grammatech.com [2] http://www.klocwork.com http://www.klocwork.com

stonemetal 14y ago
If you don't mind MS, their code analysis tools come in the express(free) version of VS 2012.
- pnathan 14y ago
  Personally, I mind MS. But that's neither here nor there. My understanding is that their tools don't do "whole program analysis". Is that incorrect (I'd love to hear that I am!)?
  - stonemetal 14y ago
    I can't say I know enough about it to say either way, in the article Carmack says: Technically, the Microsoft tool only performs local analysis, so it should be inferior to Coverity’s global analysis, but enabling it poured out mountains of errors, far more than Coverity reported. True, there were lots of false positives, but there was also a lot of scary, scary stuff.

santaragolabs 14y ago

So I've dealt with dozens of Fortune-100 companies implementing and using static code analysis tools. They can and will help but in general I feel that these tools are not much more than the code-equivalent of the syntax- and grammar- checker in your word processing software. I've been doing manual code reviews for a living now (mostly security related) for roughly 3 years now and while I get assisted from time to time by code analysis tools I still find heaps of bugs not caught by any of the tools mentioned by Carmack. The biggest issue for a development shop is to properly integrate these tools and to not overwhelm developers with too much false positives. I've had cases where a developer got a 1500 page PDF spit out by one of these static analysis tools. After spending two weeks going through everything the developer ended up with 50 pages of actual bugs; the rest were describing false positives. Then I got on-site and I still logged dozens and dozens of security-related bugs that the static analysis tools failed to find. Edit: also consider that one even needs a SAT solver to even do proper C-style preprocessor dependency checking. A lot of these code analysis tools are being run on debug builds only and then there when the release build is being made these tools are not being run meaning they fail to catch a lot of issues. It's insanely hard to write proper code analysis tools and static source code analysis tools which do not integrate with the compilation process I wouldn't trust at all. Nowadays with clang there are very nice possibilities for someone to write your own simple checks and integrate them into the build process. But even clang doesn't expose everything about the preprocessor that you might want to have from a static code analysis perspective.

btilly 14y ago
Beware the false false positive! A couple of years ago I ran across an article from Coverity about the challenges that they had getting people to use their tool. One of the most interesting was that the tool would find real bugs, but the developer wouldn't understand that the bug actually was real, and would say, "This tool is crap." Then they would not get a sale. They had this problem particularly strongly with race conditions. There are a number of checks that they took out because, even though they were finding real bugs, developers were convincing themselves that the bugs were not real even though they really were. It really does not help that the developers who are asked to evaluate are likely to be the same people who made the mistakes in the first place, so all kinds of defensive behavior are to be expected.

gaius 14y ago

Microsoft feels that game quality on the 360 impacts them more than application quality on Windows does. :-) Many a true word spoken in jest!

cbs 14y ago
Carmack is no doubt familiar with Microsoft's game cert process, it probably wasn't in jest at all. It does impact Microsoft more, because Microsoft has taken on responsibility for games to be good (well, meet certain baselines for various definitions of good) through their TCRs.
- gaius 14y ago
  'Twas his smiley on the end, I didn't add it.
georgemcbay 14y ago
This actually makes a lot of sense. I mean, Microsoft is a gatekeeper on the Xbox 360 and enforces quite a bit of forced QA on every game officially released for the platform. If an Xbox 360 game crashes, I view it as a Microsoft failing, even if the game isn't developed by them. OTOH, if a non-Microsoft Windows app crashes, I don't blame Microsoft at all but rather the app developer.

zwieback 14y ago

The article mirrors my recent experience 100%. We've got a Coverity license and I've started using it recently. Luckily, our code base is relatively small, it's straight C and embedded (no mallocs, no OS). Even in this extremely simple environment it's shocking how many errors Coverity can ferret out. The false-positives are a problem and the general advice to get started is to initially ignore all existing bugs and focus on avoiding adding new bugs. Then, when you get the hang of writing code that passes the checks you go back and look for the worst of the older bugs, etc.

cpeterso 14y ago

FindBugs [1] is a great code analysis tool for Java. It's free, open source, and supports plugins for writing your own checks. The FindBugs site reports an interesting story from a Google test day: "Google held a global "fixit" day using UMD's FindBugs static analysis tool for finding coding mistakes in Java software. More than 700 engineers ran FindBugs from dozens of offices. Engineers have already submitted changes that made more than 1,100 of the 3,800 issues go away. Engineers filed more than 1,700 bug reports, of which 600 have already been marked as fixed. Work continues on addressing the issues raised by the fixit, and on supporting the integration of FindBugs into the software development process at Google." [1] http://findbugs.sourceforge.net/ http://findbugs.sourceforge.net/

jadc 14y ago
For Java code, I am a big fan of Sonar Source [1]. It is open source, is able to leverage checkers such as FindBugs, and has integration with code coverage tools such as Cobertura. I have found the clean dashboard to be a great boon, and have never felt intimated by the reported warnings like I've been with Coverity et al. [1] http://www.sonarsource.org http://www.sonarsource.org
- npsimons 14y ago
  As long as we're on an open source kick here (since I prefer open source over proprietary): http://checkstyle.sourceforge.net/ http://checkstyle.sourceforge.net/ http://cppcheck.sourceforge.net/ http://cppcheck.sourceforge.net/ http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py http://google-styleguide.googlecode.com/svn/trunk/cpplint/cp... http://www.dwheeler.com/flawfinder/ http://www.dwheeler.com/flawfinder/ http://www.dsm.fordham.edu/~ftnchek/ http://www.dsm.fordham.edu/~ftnchek/ http://community.haskell.org/~ndm/hlint/ http://community.haskell.org/~ndm/hlint/ http://jlint.sourceforge.net/ http://jlint.sourceforge.net/ http://search.cpan.org/dist/Perl-Critic/bin/perlcritic http://search.cpan.org/dist/Perl-Critic/bin/perlcritic http://deployingradius.com/pscan/ http://deployingradius.com/pscan/ http://pychecker.sourceforge.net/ http://pychecker.sourceforge.net/ http://pypi.python.org/pypi/pyflakes http://pypi.python.org/pypi/pyflakes http://www.logilab.org/857 http://www.logilab.org/857 http://packages.debian.org/squeeze/rats http://packages.debian.org/squeeze/rats http://www.splint.org/ http://www.splint.org/ http://www.incava.org/projects/303902593 http://www.incava.org/projects/303902593 http://packages.debian.org/stable/devel/icheck http://packages.debian.org/stable/devel/icheck Please note: I have not used all of these; one day I was bored and decided to see how many code checking tools I could find packaged for Debian and put them all in a script with the warnings cranked to the max and make that part of my default nightly build and test script. Of course, you can also crank up warnings on your compilers and interpreters as well (-Wall -Wextra, python -tt3 with warnings.simplefilter("always"), perl -tTW with use strict and use warnings, etc)

rabidsnail 14y ago

Has anyone tried running their commits through CRM114, marking removals in bugfix commits as "bad"?

eru 14y ago
You could also try to train some statistical model and see whether you can find some common ground in commits that introduce bugs. And then fire that person. (Or rather, see if big commits introduce more bugs than short ones, or whatever else you might find.)

mrich 14y ago

For C/C++, also try just compiling with clang. It has great diagnostics. Also it has the static analyer whose C++ support just improved greatly in trunk.

Flow 14y ago

How good is the code analysis in IntelliJ compared to these tools?

calpaterson 14y ago
Code analysis is nice but stand alone static analysis tools are a lot more advanced. Even the free ones, like FindBugs catch a lot of issues.

Havoc 14y ago

Pretty sure I've read this exact thing ~1 year ago.

samspot 14y ago
Yeah it is old, you can check the article date.

js2 14y ago

(2011) Original HN discussion http://news.ycombinator.com/item?id=3388290 http://news.ycombinator.com/item?id=3388290

egonschiele 14y ago

I've been wanting something like this for Ruby for some time now. Since it's dynamically typed and ridiculously easy to monkey-patch, Ruby is a much harder challenge than C++. The two best efforts I have found are Diamondback Ruby (http://www.cs.umd.edu/projects/PL/druby/ http://www.cs.umd.edu/projects/PL/druby/) and Laser (http://carboni.ca/projects/p/laser http://carboni.ca/projects/p/laser)...but they mostly try to add static type-checking to Ruby code. After looking at these I implemented a contracts library for Ruby (https://github.com/egonSchiele/contracts.ruby https://github.com/egonSchiele/contracts.ruby) to get myself some better dynamic checking. The next step is to use the annotations for the contracts library to do better static code analysis. One thing I'm working on is generating tests automatically based on the contract annotations. But I've got a long way to go : ( If anyone knows about other projects that are working on static analysis for Ruby I'd be very interested in hearing about them!

pjungwir 14y ago
Looked at your Contracts library for Ruby. I was just thinking yesterday about starting a project like this! Some thoughts: - I like that the contracts are right there in the code and not in a separate file. - The actual Contract methods only run once upon loading each class, so they don't cause a performance hit. I'd like a way to make them all noops (so they don't add any instrumentation) when running in "production". - My thought is that if you can define pre- and post-conditions as you're doing, then you can cut out a lot of the checks in your tests, so that as long as the tests exercise the code, you can rely on the post-conditions to do quite a bit of the validation. Plus you're validating outputs all over the place constantly, so you'd probably catch edge cases you never even expected. - I'm disappointed that your system seems to offer little more than type checking (unless I'm mistaken). I'd like something that gives me true pre- and post-conditions: basically passing a block of Ruby code for each one, with common idioms abbreviated so I don't need to write a whole block if the conditions are simple. Good luck on the project!
- egonschiele 14y ago
  Great thoughts! > I'd like a way to make them all noops Sure! There's a default failure callback that gets called when a contract fails. You can just clear this callback in prod: if ENV == PRODUCTION class Contract def self.failure_callback(data) # blank end end end > I'm disappointed that your system seems to offer little more than type checking Nay! You can write your own contracts to do anything you want. So yes, lambdas are allowed in contracts: Contract lambda { |x| x.is_valid_email? } => Bool def save_email(email) # save an email end > with common idioms abbreviated I provide some common idioms already, but you can make your own abbreviations: ValidEmail = lambda { |x| x.is_valid_email? } Contract ValidEmail => Bool def save_email(email) Let me know if you have any other questions :)

stcredzero 14y ago

> We had a period where one of the projects accidentally got the static analysis option turned off for a few months, and when I noticed and re-enabled it, there were piles of new errors that had been introduced in the interim. Similarly, programmers working just on the PC or PS3 would check in faulty code and not realize it until they got a “broken 360 build” email report. These were demonstrations that the normal development operations were continuously producing these classes of errors, and /analyze was effectively shielding us from a lot of them. Something which corroborates this: When penetration testers break into systems, they're often using new 0-day exploits. Think about that. Most of today's software development practice produces such a steady stream of low-level bugs, that penetration testers can assume that they're there! > Trying to retrofit a substantial codebase to be clean at maximum levels in PC-Lint is probably futile. I did some “green field” programming where I slavishly made every picky lint comment go away, but it is more of an adjustment than most experienced C/C++ programmers are going to want to make. I still need to spend some time trying to determine the right set of warnings to enable to let us get the most benefit from PC-Lint. This could be encouraged using game dynamics. Have a mechanism where a programmer can mark parts of the codebase "green-field." A programmer's "green-field score" consists of the number of lines of green-field code (or statements, whichever lousy metric you want) that he's successfully compiled with no warnings whatsoever. Combine this with random sampling code walkthroughs, which has many benefits but will also catch boilerplate, auto-generated, or copy-paste programming by a "Wally" who's trying to "write himself a new minivan."

deleted 14y ago

[deleted]

jaimefjorge 14y ago

This article is one of the reason I created my tool: http://www.qamine.com http://www.qamine.com Qamine integrates directly with github and is designed to be used by small and medium companies that cannot afford those expensive tools.

Darmani 14y ago
Your tool looks pretty cool and useful, but it seems quite a bit different than tools like Coverity and Astree which reason about the behavior of programs under all possible inputs. Coverity can detect race conditions. Your tool looks like it's using technology closer to that of Refactoring Crawler, which was a quite impressive achievement, but not a static analysis tool.

FredBrach 14y ago

There have been plenty of hugely successful and highly regarded titles that were filled with bugs and crashed a lot I think it's false, and a huge mistake. There is rare cases, but not plenty. Video games (I mean video games for core gamers) are products that demand first and above all quality to be successful. There is rather plenty of common games with a huge quality which become hits (Kingdom Rush for example or Starcraft which was finally kind of common in its time). One of the rules in the delovepment process at Blizzard is that they ship a game when it has less than 100 known bugs. Also, I would add that, it seems that, ID software did not make a successful games since Doom. Quake wasn't a commercial success, Quake2, Quake3, Doom3 and Rage neither (that's why ID software has been bought for only $100M). After all, ID Software lost one of its core value co-founder a long time ago (John Romero) who was responsible of the gameplay of ID's games... Quality in video games are everything, that's really my opinion. It's also really an edge for every indy developper which want to start a company in this sector, cases are countless.

bgaluszka 14y ago
> Anyway, nobody here, is still playing one of these games - starcraft for example, is a game still played. Well, I wouldn't say nobody. I, for one, play/watch Quakeworld and Quakelive on a regular basis.
- FredBrach 14y ago
  Ok you're right. I've deleted that. I say right because we could discuss hours about what nobody means. Oh just a question, so did you bought quake?
  - bgaluszka 14y ago
    I bought them and won them on tournaments so I own few of those :)
    - deleted 14y ago
      [deleted]

nfriedly 14y ago

Funny timing, I just got jslint turned back on in our build today! (well, jsHint now due to the 'for(var i=0...' failing even with -vars enabled, but I digress...). Another dev and I spent literally the entire day fixing issues - and we had jslint running on every checkin until a few months ago! But, it was worth it. It feels great to know that those bugs won't happen again without a failing build :)

ahoge 14y ago
>due to the 'for(var i=0...' failing even with -vars enabled That's because `for(var i=0...` is completely bogus. What if you have two of those loops? Are you going to put `var` in the first one, but not in the second one? Are you going to use 'i' in the first one and 'k' in the second one? What if you need some refactoring and those two loops switch places? It's fine if you like Java, but JavaScript is different. When in Rome, do as the Romans do. I also wish there would be block scope, but merely pretending there is doesn't make it so. It's better if you declare all of your variables at the very top of the innermost function, because that's where they effectively are (thanks to hoisting). It's better if your code reflects how it's actually working, because that will always make it easier to comprehend. Once there is `let`, you should switch to Java's "declare on first use" rule and never use `var` again.
- nfriedly 14y ago
  Right, I understand why. And, if I have two in the same scope, jshint will rightly alert me that I have an error. But I don't have this issue once in ~20k lines of js. So in my case, jslint not honoring its own -vars option was just a nuisance. And while I agree that declaring vars at the top is closer to whats actually happening, I disagree that it makes it easier to comprehend.
  - ahoge 14y ago
    function foo () { x = 5; // totally fine ... var x; // because we declare it here }
- Joeri 14y ago