10 ms·
Static Analysis in GCC 10
- 3fe9a03ccd14ca5 7y agoThese types of tools go a really long way in improving the reliability and safety of C code. Hats off to the redhat team for putting in the effort on this. Their blog posts have been really interesting. It’s definitely changing my perception of what redhat really does.
- jabedude 7y agoI've always had a positive impression of redhat, but I was recently blown away with their dedication to upstreaming contributions across different open source projects. I was investigating a new Linux kernel feature that redhat contributed and saw that the same developer opened pull requests that added support for the new kernel feature in three major open source projects. And one of the projects took over a year to accept the changes, but he was persistent in reaching out, making requested changes, etc. It really shows the passion at the company to share their contributions.
- Vogtinator 7y agoIt doesn't have to be passion - having something upstreamed has a lot of other benefits as well.
- mynegation 7y agoI, at one time, worked on a tool, commercial and external to the compiler, that did this (among other things). Probably the most intellectually challenging job I have ever had. I am happy static analysis makes inroads into mainstream! Few takeaways from that time: inter procedural matters: if your function reallocated a pointer passed as an argument, you want to treat it as ‘free’ regarding this argument, and conversely, if your function returns a newly allocated memory, you want to mark it as such, and so on. There is also a trade off between the breadth of the analysis and the human ability to comprehend it, author mentions 110 node path in the article. The subject of my unfinished PhD thesis and something I hope also picks up is the combination of static and dynamic analysis, used iteratively. If your static analysis flags a suspicious path but does not have the means to figure out if it is true or not, instrument it and leave it to the dynamic analysis to run through it (the idea here that total instrumentation a la valgrind is detrimental to performance so you will get some gain from selective instrumentation). Conversely, dynamic analysis may provide some hints as to where static analysis should be applied at a greater depth and provide automatic annotation of functions with regard to their behaviour and - possibly - invariants, that help with the state explosion.
- DyslexicAtheist 7y agoca 2000 - 2004 I had the luck to work on a massive C/C++code base building base station / telecoms infrastructure. We had several hundred engineers contributing code with all kind of different philosophies (many grey beards who did C/C++ for decades). Running Flexelint was part of our CI chain and also part of the internal coding standard (e.g. definition of done). There was no other time in my life where I learned so much about secure coding as I did back then. Biggest challenge was agreeing about false positives and we had 1 guy in the team who maintained the official wiki document on when a lint warning needed to go on a whitelist with an agreed description of why. The initial overhead to become lint-clean got a lot of push-back but thanks to management support we got there and you could really see how things stayed at that level even after years. It felt at times bureaucratic or like yak shaving, but in retrospect linting was what kept the code base at a quality I haven't seen ever again since. It also ensured everyone was on the same page. Taking linting seriously required a small learning curve though and lead to some discussions here or there. These discussions were really valuable since we got to really learn from another too. When I left and went to another project it felt like a step back where we were chasing the same old bugs due to bad coding practices and it was a major step back in my career as a dev. I miss those days. Really love that this becomes part of gcc.
- neilv 7y agoKudos for lint-clean, and for high-quality C/C++ programming. One of my own stories of that (coincidentally, from when I worked on dev tools for aerospace/datacomm): https://news.ycombinator.com/item?id=21158546 https://news.ycombinator.com/item?id=21158546
- ape4 7y agoYou don't want that option every time since its slower. But I wonder if there would be a smart way to run it occasionally, like an option to -fanalyzer every 10th time or when the size of a source file changes a lot, etc.
- saagarjha 7y agoNice, this looks pretty cool! It seems a bit like Clang’s static analyzer: https://clang-analyzer.llvm.org/ https://clang-analyzer.llvm.org/
- jchw 7y ago> As of GCC 10, that option text is now a clickable hyperlink (again, assuming a sufficiently capable terminal) Seems like mostly only GNOME Terminal and iTerm2. Here’s some that apparently won’t work: - Konsole - Kitty - LXDE Terminal - MATE Terminal - hyper - Windows Terminal - ConEmu - PuTTY ... so it’s kind of weird to suggest this is an accepted standard. Especially since some of the discussions in feature requests suggest they will likely not implement it due to security concerns or otherwise.
- cowsandmilk 7y agoI don’t see anything suggesting it is a standard. It seems like it is an enhancement that some terminals support. It isn’t like this feature degrades support for any other terminal, just improves life for some users. Why complain?
- jchw 7y agoBecause it can break terminals that don’t. Nothing forces you to put enough context in your link to be able to recover the URL. Unsuspecting developers who only target iTerm2 will put a link in for “Click here to complete OAuth2 flow” or something and it will be broken... In this case it isn’t the worst problem ever. but i think support for this should probably be removed from terminal emulators unless a quorum of popular terminal emulators support it. (Addendum: do developers actually do things that only work in virtually a single terminal emulator? Yeah. A lot of Node.JS software printed status using emoji that broke horridly on everything but macOS. It mostly works elsewhere now, and it was purely a visual problem, but I would assume many didn’t even realize what their app would’ve done in most terminal emulators. The same with hyperlinks would be a disaster.)
- pwdisswordfish2 7y agoSomeone should create a database of various terminal emulators and the features they support, so that applications would be able to look up what features are available. We shall call it the ‘terminal capabilities library’.
- archi42 7y agoOh, that's really nice. Though, as a user one should remember: The approach described here gives up at some point. So it doesn't prove the absence of a bug class (e.g. double free), but it finds some instances. Which is already a very good thing, and hugely non-trivial. The problem with "not giving up at some point" is the computational complexity: Analyzing big code bases takes half an eternity (days), while using huge amounts of memory (>128GB). And once you enter the "least-defined state", you either throw lots of false positives (which gives the users a hard time) or you need to "give up" (and hence potentially miss bugs). Disclaimer: I work for a company that builds static analysis tools. I don't see this as competition, though. Our tools are used in industries where "safety-critical" is really important - so the "giving up"-part of the analysis is no option for us, and solely relying only on GCC isn't an option for our customers either ;-)
- s3cur3 7y agoI’d be interested to know which company (if you’re free to say). IIUC, this is very different from the way Coverity (the biggest commercial player I’m aware of in this space) operates.
- yitchelle 7y agoInterestingly, different industry seems to have its own leading tool even though they all check for the same type of risks or errors. Q-AC is extremely prevalent in the Automotive Industry, Polyspace's Code Prover is also quite popular as well.
- archi42 7y agoI've to admit Q-AC doesn't ring a bell? A quick duck duck go doesn't bring up anything meaningful.
- yitchelle 7y agoThe company got taken over by Perforce. QAC was previously from Programming Research. https://www.perforce.com/products/helix-qac https://www.perforce.com/products/helix-qac
- ufo 7y agoI wonder... Has anyone here tried using gcc with the -fanalizer option? Did it find any bugs that you did not know about?
- deleted 7y ago[deleted]
- _bxg1 7y agoI have to wonder if Rust is putting pressure on C/++ to have more static analysis (while at the same time blazing trails in what's possible in that space, and what's possible in terms of error message helpfulness). I think it's a great idea to start baking these things into the compiler, even if it will never be 100% free of false-negatives because of the limitations of what the language can express and guarantee. Still seems like a great way to eliminate a lot of common problems, as a default across the ecosystem instead of as an extra step.
- thekhatribharat 7y agoI think you mean "false-positives". Most static analysis tools are "conservative": will throw more warnings than they can prove. But assuming they're "sound" (and don't "give up" because of time/space constraints), there'd be no "false-negatives".
- UncleMeat 7y agoVanishingly few (maybe even zero) industrial static analysis tools are sound. FPs are too common, especially for something like C++ where common code patterns can only be soundly handled by just forcing everything to Top. Good luck with your sound interprocedural pointer analysis as soon as somebody makes an indirect call.
- thekhatribharat 7y agoI think you mean industrial static analysis tools aren't complete. Sound means no FNs, Complete means no FPs.
- UncleMeat 7y agoNo I meant what I said. Virtually no tool is sound (zero false negatives, lots of false positives) because in practice false positives just make people stop using tools when they are too common.
- 6gvONxR4sf7o 7y agoI'm super happy to see more static tools to prevent or at least find buggy code.
- olivierduval 7y ago"using the compiler the code is written in as part of the compile-edit-debug cycle, rather than having static analysis as an extra tool “on the side” (perhaps proprietary)" Mmmm... and why not have an external tool, part of the GCC family but with a proper API, to allow to use ANY TOOL instead of bloating GCC with one more tool that won't be usable on other compilers and will need specific maintainers, althought this field is already really complex and need a lot of different knowledges ? For example, it could be based on intermediate code so better than just source-code or machine-code analysis...
- olivierduval 7y agoJust to be more specific: why not use the "UNIX philosophy" with a compiler to compile (translate to Intermediate Representation), an optimizer to optimize, an assembler to produce machine code from IR (with allocation registry) and so on...?
- aw1621107 7y agoIIRC in the past GCC (or maybe just some of the maintainers?) has been resistant to that kind of architecture because it makes it easier to write proprietary extensions/plugins/backends/frontends/etc. to GCC. I'm not sure whether that has changed. LLVM was designed to be a lot more modular, so it fits these use cases a lot better. In your example, you have the front ends (C, C++, Objective-C, Fortran frontends in LLVM proper, Rust, Zig, etc. as third-party front-ends) compiling to LLVM IR, opt for optimization, and different backends to generate machine code from IR.
- UncleMeat 7y agoBecause GCC is explicitly designed to be a tangled mess (as opposed to Clang/LLVM), in part because it makes it harder for groups with different beliefs about FLOSS code to repurpose it. Its a choice that has caused them to cede a lot of territory to Clang/LLVM.
- ndesaulniers 7y agoI think the major mistake was FSF refusing the objective c front end from Apple.
- deleted 7y ago[deleted]
- mshockwave 7y agoJust a side note that the equivalent solution in LLVM/Clang is Clang Static Analyzer: https://clang-analyzer.llvm.org https://clang-analyzer.llvm.org . But it's an external tool rather than integrating into clang
- marta_morena_23 7y ago"My thinking here is that it’s best to catch problems as early as possible as the code is written, using the compiler the code is written in as part of the compile-edit-debug cycle, rather than having static analysis as an extra tool “on the side” (perhaps proprietary)." Really? Your thinking? I mean seriously, why do you have to start off a good article and feature with a sentence like this? This is not your thinking, and if it was, you have been living under a rock for the past decades... Why the need to make it sound as if this decade long idea was yours?
- raegis 7y agoLet's all try to be nice to each other. Everybody's stuck at home and frustrated right now.
- glouwbug 7y agoNew users typically think this is Reddit. Best we can do is encourage civil discourse
- yjftsjthsd-h 7y agoAre they only allowed to think thoughts that nobody else has had?
- glouwbug 7y agoIt probably was their idea, but you are right, it was the idea others before their idea. Difference is, their idea came to fruition, and their idea has contributed to what is arguably the world's most important piece of software
- dang 7y agoCould you please not post in the flamewar style to HN? It's not what this site is for, and if you'll read https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html, you'll see that we're actively trying for a different sort of discussion.
- thekhatribharat 7y agoI believe type sytems/type theory is likely going to be the most popular method for formal verification of programs (aka static analysis). And of course, there's a limit to what you can prove about programs (ref: Rice's theorem).
- jfkebwjsbx 7y agoStatic analysis is not formal verification.
- thekhatribharat 7y agoSure, things like enforcing style guides, etc. can be seen as lightweight formal verification.
- irundebian 7y agoYes, but you wrote "formal verification of programs (aka static analysis)". Formal verification is not also known as static analysis.
- exdsq 7y agoIt's part of the field of formal methods.
- wyldfire 7y agoPreviously submitted as https://news.ycombinator.com/item?id=22708586 https://news.ycombinator.com/item?id=22708586
- leni536 7y agoIt looks great and useful. I suspect that this only works within a single translation unit and can't work between separate translation units. But maybe it could work together with lto, that would be pretty awesome. Some of the worst lifetime issues are 3rd party library calls with unclear ownership semantics and static analyzers are just as clueless as you are. The function signature doesn't help you out in this regard (in C). My recent "favorite" is libzip's zip_open_from_source that conditionally takes ownership of its zip_source_t* argument. https://libzip.org/documentation/zip_open_from_source.html https://libzip.org/documentation/zip_open_from_source.html https://libzip.org/documentation/zip_source_free.html https://libzip.org/documentation/zip_source_free.html
- WalterBright 7y agoDouble-free's can be tracked by doing data flow analysis on the function. This is how D does it in its nascent implementation of an Ownership/Borrowing system. It can be done without DFA if getting it only 90% right and having numerous false positives is acceptable. I've used many static checkers in the past, and the rate of false positives was high enough to dissuade me from using them. This is why D uses DFA to catch 100% of the positives with 0% negatives. I knew this could be done because the compilers were using DFA in the optimization pass. In order to get the tracking to work, one cannot just track things for a function named "free". After all, a common thing to do is write one's own custom storage allocators, and the compiler won't know what they are. Hence, there has to be some mechanism to tell the compiler when a pointer parameter to a function is getting "consumed" by the caller, and when it is just "loaned" to the caller (hence the nomenclature of an Ownership/Borrowing system). One of the difficulties to overcome with D in doing this is there are several complex semantic constructs that needed to be deconstructed into their component pointer operations. I noticed that Rust simplified this problem by simplifying the language :-) But once done, it works, and works satisfyingly well. Note that none of this is a criticism of what GCC 10 does, because the article gives insufficient detail to draw any informed conclusions. But I do see this as part of a general trend that people are sick and tired of memory safety bugs in programming languages, and it's good to see progress on all fronts here.
- anchpop 7y ago> This is why, for D, I was determined to use DFA to catch 100% of the positives with 0% negatives. I knew this could be done because my compilers were using DFA in the optimization pass. Is this really true? I thought this was impossible due to Rice's theorem
- WalterBright 7y agoFortunately, I am unaware that it was impossible and did it anyway :-) But it is possible I made a mistake. It is also true that for it to work, one has to change the way one writes code, like Rust does. This is why D requires and @live attribute be added to functions to enable the checking for just those functions, so it doesn't break every program out there. It will enable incremental use of the checking at the user's option.
- google234123 7y agoHow far behind LLVM/Clang is this?
- anarazel 7y agoLast time I checked - I'm not sure how long ago that is - llvm didn't detect double frees etc statically. There's an annotation framework for locking though, which I hope to play with more.
- google234123 7y agoI think you are being a bit disingenuous here since it's been in clang since 2013 as far as I'm aware.
- anarazel 7y agoHm. It was more recent than that. Perhaps I am just remembering that it was not possible to get it to trace user defined allocators / wrappers around malloc?
- deleted 7y ago[deleted]