6 ms·
Show HN: Compare Analysis Tools For Python, Ruby, C, PHP, Go
- raxxorrax 6y agoThat is really comprehensive and useful, thank you.
- omn1 6y agoThanks for the feedback. Glad you liked it. If you (or anyone else) find the time, please vote on a few tools and perhaps also add a comment for a tool? That would help us a lot as we basically have zero comments on the page right now and it's hard to crowdsource them.
- esnard 6y agoIt looks like those tools are sorted by votes, but some of them can analyze different languages, and votes are shared between their languages. For example, CodeScene, which supports 12 languages, is the currently most voted tool for PHP, and I've never heard of it. Not saying it's bad or anything, but I highly doubt it's popular in the PHP community, compared to other products.
- omn1 6y agoOh yeah, we realized that as well. What do you think about putting services for multiple languages into a separate group and exclude them from the votings for the individual languages?
- smichel17 6y agoI think it would make more sense to vote on tags. Then you could compare multi-language formatters with single language by how many votes they had for that tag. Iirc imgur had (has?) a system like this in addition to the main up/downvote (but I don't think you need a main up/downvote, since you're creating a list, not a feed).
- omn1 6y agoThat... makes a lot of sense to me. Probably even going away from a simple upvote/downvote towards a 1-5 star rating system would help as well. In the end we're interested in the quality of the tool (and the number of ratings).
- captn3m0 6y agoyou'll end up with a skewed rating anyway (either 1 or 5 stars for most ratings).
- omn1 6y agoSo it's not worth the trouble then?
- setr 6y agoWhen I was looking into writing recommendation systems, one paper made the interesting observation: in a 1-10 rating system, the only places where people will naturally agree upon is 1,5 and 10. Anything else can only be evaluated relative to the same user's other scores. (Comparing my 9 and 10 scores has meaning, but not comparing your 8 and my 9) The polarization is a natural result of the lack of definition
- rckoepke 6y agoEvan Miller argues that it's probably simplest to stick with thumbsup/thumbsdown, but not to fall into the trap of using dead-simple analysis methods like "Score = (Positive ratings) − (Negative ratings)". Instead he argues you should use the "Lower bound of Wilson score confidence interval for a Bernoulli parameter". He provides that equation and example code to calculate it (Ruby, SQL, and/or Excel). https://www.evanmiller.org/how-not-to-sort-by-average-rating.html https://www.evanmiller.org/how-not-to-sort-by-average-rating... https://news.ycombinator.com/item?id=15709405 https://news.ycombinator.com/item?id=15709405 https://www.evanmiller.org/bayesian-average-ratings.html https://www.evanmiller.org/bayesian-average-ratings.html https://www.evanmiller.org/ranking-items-with-star-ratings.html https://www.evanmiller.org/ranking-items-with-star-ratings.h... https://redditblog.com/2009/10/15/reddits-new-comment-sorting-system/ https://redditblog.com/2009/10/15/reddits-new-comment-sortin... (Found these links shared by another user a couple of weeks ago[0]) 0: https://news.ycombinator.com/item?id=24089960 https://news.ycombinator.com/item?id=24089960
- esnard 6y agoPlease do not exclude them! I bet some of them are very good, especially since you support "similar" languages (JavaScript / TypeScript f.e.). smichel17's solution makes the most sense to me.
- DJBunnies 6y agoFor real. Phpstan and phpcs for life.
- stabbles 6y agoNow we need a meta static analysis tool that reports from all static analysis tools
- omn1 6y agoHeh, already done. https://endler.dev/2017/obsolete/ https://endler.dev/2017/obsolete/ :)
- dhekir 6y agoIt would be useful to list which tools support the SARIF standardized format (https://sarifweb.azurewebsites.net/ https://sarifweb.azurewebsites.net/).
- omn1 6y agoInteresting. Never heard of it, but I created a ticket on Github to investigate. You can watch it to track updates: https://github.com/analysis-tools-dev/static-analysis/issues/495 https://github.com/analysis-tools-dev/static-analysis/issues... Thanks for the feedback!
- Strum355 6y agoHow does Sarif compare to LSIF?
- dhekir 6y agoI had never heard of LSIF. Its webpage says it intends to "define a standard format for language servers or other programming tools to dump their knowledge about a workspace". SARIF being targeted towards static analysis tools results, it seems more straightforward to support it both for producing results and for reading them. To be honest, since both formats are at least partially developed by Microsoft, they might have some common people behind them, but as a user and producer of static analysis tools, I'd prefer to have a single standard. SARIF is already standardized and gaining traction, so it seems the better choice for now.
- omn1 6y agoHey, this post got more attention than I thought. Happy to answer your questions and get some feedback on what to improve. Maybe people are interested in some tech: My colleague Jakub and me built this site with GatsbyJS and Cloudflare Edge Workers. The 99th percentile of response times from the workers is currently 9.7ms, which is impressive. The code is fully open source on Github [1]. It is based on submissions by 190 individual contributors so far [2] We went for an open model and completely depend on Github sponsors for the funding. We are not trying to rapidly grow here, rather build a steady business. You can read more about the buisness model in our first blog post [3]. If your company might be interested in sponsoring, let us know or check the offerings here: https://github.com/sponsors/analysis-tools-dev/ https://github.com/sponsors/analysis-tools-dev/ <3 [1]: https://github.com/analysis-tools-dev/website/ https://github.com/analysis-tools-dev/website/ [2]: https://github.com/analysis-tools-dev/static-analysis https://github.com/analysis-tools-dev/static-analysis [3]: https://analysis-tools.dev/blog/static-analysis-is-broken-lets-fix-it https://analysis-tools.dev/blog/static-analysis-is-broken-le...
- pqdbr 6y agoI'm not familiar with Cloudflare Edge Workers, so sorry if this is an obvious question, but what kind of data store (database) can you use in a setup like this?
- omn1 6y agoCloudflare provides a simple key-value store for values up to 2MB (https://www.cloudflare.com/products/workers-kv/ https://www.cloudflare.com/products/workers-kv/). We don't use that, though and instead use firebase on GCP plus a daily data backup at the moment.
- oneweekwonder 6y agoA while ago this[0] hosting solution of blackblaze b2 and cloudflare workers looked interesting. I believe it uses a aws s3 compatible api[1]. [0]: https://jross.me/free-personal-image-hosting-with-backblaze-b2-and-cloudflare-workers/ https://jross.me/free-personal-image-hosting-with-backblaze-... [1]: https://blog.cloudflare.com/backblaze-b2-and-the-s3-compatible-api-on-cloudflare/ https://blog.cloudflare.com/backblaze-b2-and-the-s3-compatib...
- scott31 6y agoHow can I easily integrate all of them in my CI?
- omn1 6y agoIt sounds a bit ironic, but I'll bite: you shouldn't. Rather carefully pick a few tools based on peer-review, license, and your goals and start slowly. Too many tools at once will cause analysis fatigue. That said, in the future we hope to provide a paid video course about static analysis for e.g. Python and Go if there is market interest.
- tpool 6y agoNearly all of the major security-focus static tools have CI integration, and guidance on best practices for integration.
- muglug 6y agoI'm not sure voting works?
- omn1 6y agoIt does work, there is just no instant feedback. Voting happens through firebase and when you reload the page it's not reflected immediately. Anyone knows how other platforms like Reddit solve that?
- muglug 6y agoIt seems not to? I voted on something last night (selfishly, my own tool), but it's still not reflected.
- omn1 6y agoStatic pages with the votes get rendered once per day. It might be that you just missed that deadline yesterday. We'll look into it now. Please ping us if you can't see the votes by tomorrow.
- giovannibonetti 6y agoAm I the only one worried about unauthenticated voting? I guess it'll become a major headache pretty quickly. Imagine people using bots to vote "up" on their favorite tools and "down" on the others.
- jakubsacha 6y agoYeah this is concerning us as well ;-) So we have two measures in place right now: - We rate limit voting requests - You can vote just once per tool
- yboris 6y agoShameless self-promotion time: TypeScript Call Graph A CLI to generate an interactive graph of functions and calls from your TypeScript files. https://github.com/whyboris/TypeScript-Call-Graph https://github.com/whyboris/TypeScript-Call-Graph
- omn1 6y agoLooks great! The threshold for addition to analysis-tools.dev is (an arbitrary) 20 stars. If you like, you can already create a pull request here: https://github.com/analysis-tools-dev/static-analysis/blob/master/CONTRIBUTING.md https://github.com/analysis-tools-dev/static-analysis/blob/m... Our list is open source, we just render a website version (also open source) from it for some more functionality.
- videlov 6y agoLooks nice! I built a couple of tools similar to that for Java that helped me in refactoring large codebases: https://github.com/krlvi/jar-graph https://github.com/krlvi/jar-graph for visualising interclass dependencies https://github.com/krlvi/java-rdeps https://github.com/krlvi/java-rdeps for recursively discovering all transitive call sites of a given Java method https://github.com/krlvi/cyclist https://github.com/krlvi/cyclist for detecting cyclical dependencies between packages in the same Jar Sharing just in case it is useful to somebody else
- Benjamin_Dobell 6y agoDoesn't look like my project meets the eligibility requirements yet, too new. So I'll just share here for anyone interested. Luanalysis - An IDE for statically typed Lua development. https://github.com/Benjamin-Dobell/IntelliJ-Luanalysis https://github.com/Benjamin-Dobell/IntelliJ-Luanalysis
- omn1 6y agoYou can already create a pull request and then we'll merge it as soon as the acceptance criteria is met. In the meantime you'll get some exposure through the PR on https://github.com/analysis-tools-dev/static-analysis https://github.com/analysis-tools-dev/static-analysis as well, which is where we get some traffic as well (~500 unique visitors per week).
- rgovostes 6y agoSadly, many commercial source auditing tools like Coverity expressly forbid you from publishing any comparison or benchmark of their products, which is why you won't find great information out there.
- LittlePeter 6y agoWhat would be the repercussions if you still do? Assume the website is hosted in some banana republic...
- omn1 6y agoI wonder what might be the motivation behind that. Sounds like a strange policy if they have confidence in their product. Maybe it's for legal reasons.
- rgovostes 6y agoIt doesn't allow them to control the narrative about their product. Suppose the reviewer doesn't know what they're doing, and has the wrong settings selected. Or uses code samples that are devilishly difficult (e.g., heavy use of function pointers) so that the results show the product in a very negative light.
- omn1 6y agoIf I build a car and someone shifts the gears without using the clutch, breaks the car and then whines about it, that might be irritating but totally legal. So I wonder if that sort of business practice would hold in front of court.
- tpool 6y agoGreat work! I think it would be helpful to tag static analyzer tools that are dedicated to security with a security tag (SAST tools like, Brakeman, Fortify SCA, Checkmarx CxSAST, Coverity, etc.) OWASP lists a bunch here: https://owasp.org/www-community/Source_Code_Analysis_Tools https://owasp.org/www-community/Source_Code_Analysis_Tools
- omn1 6y agoGood idea. Will do!
- KoenDG 6y agoNever heard of Teamscale before, yet it has a lot of votes in all its categories. Also one of those expensive ones.
- omn1 6y agoWe'll split up votes per category in the future. This way tools that support multiple categories won't have a competitive advantage in the future.
- KoenDG 6y agoThis site could really use a filter option for cost of the product being mentioned.
- omn1 6y agoYes, that's the next feature we're working on. You'll be able to filter by license, maintenance level, and type (commandline, service,...). Jakub is currently restructuring the YAML source to allow for that (https://github.com/analysis-tools-dev/static-analysis/pull/484 https://github.com/analysis-tools-dev/static-analysis/pull/4...). We wanted to launch early to gauge if we're heading in the right direction. Thanks for the feedback.