3 ms·

The README is quite ahead of reality: it never mention that Mago is still beta software. A roadmap to a first release was created 5 days ago, https://github.com

The README is quite ahead of reality: it never mention that Mago is still beta software. A roadmap to a first release was created 5 days ago, https://github.com/carthage-software/mago/issues/405 https://github.com/carthage-software/mago/issues/405. I've just tried to apply it to a medium-sized project, and it spitted tens of thousands of errors where phpstan and psalm don't see any. At first glance, it's because Mago does not parse phpdoc. In its current beta state, Mago is meaningless for all the big PHP projects, which are its main target. Mago might succeed, but I wouldn't bet on it. Its main selling point is that it promises to be faster than the usual static analysers-linters (phpstan and psalm). But if it does not reach feature parity with them, the speed gain probably won't convince PHP projects to drop the standard tools. Since phpstan and Co keep evolving, keeping feature parity will require constant work. And PHP is more niche than Python or JS, so contributors mastering Rust and PHP will be fewer, compared to phpstan/psalm which are written in PHP.

eider 1y ago
I pulled one of classes from my project that does not depend on anything external and fed it to their demo site. It threw up bunch of errors complaining that RuntimeException is undefined. It seems it doesn't understand built-ins unless you prefix them with \, even though they are imported properly with use statement. This is pretty core issue to lack support for. Calling it "beta" is actually giving it a lock of slack, I'd say it's closer to being proof of concept.
- azjezz2 1y ago
  Mago author here. The online playground is running a very old version (~0.20.0 from months ago, which did not even have a static analyzer) and gives a poor impression of the current tool. That's on me to fix or take down. The issue you saw with built-in classes like `RuntimeException` was absolutely a bug in those early alpha versions, but it has been fixed for a long time now. The analyzer has matured a lot since then. The current beta is stable enough to be the sole static analysis tool for a couple of extremely well-typed projects: - https://github.com/azjezz/psl/ https://github.com/azjezz/psl/ - https://github.com/carthage-software/cel-php https://github.com/carthage-software/cel-php I'd definitely encourage trying the latest release locally to get a real feel for it. Thanks again for the write-up!
azjezz2 1y ago
Mago author here. Thanks for the feedback. You're right that the README should be clearer about the beta status and the current feature set. That's a great suggestion, and I'll get that updated. Regarding the errors you saw, I suspect the main culprit isn't a general lack of PHPDoc parsing, but rather the two biggest remaining features we're actively working on: support for magic _@method_ and _@property_ tags. Mago has full support for generics (@template), assertions (@psalm-assert*), conditional types, etc., but the absence of those two is definitely a major source of noise on established projects right now. They are our top priority and should land in the next beta release. On the topic of feature parity, you're right that it's a moving target. Our goal isn't to be a 1-to-1 clone of Psalm or PHPStan, but a different tool with its own strengths (see: https://github.com/carthage-software/mago/discussions/379 https://github.com/carthage-software/mago/discussions/379). For example, Mago will flag code like `[0 => $a, $b] = ["a", "b"]` as an error, which other tools currently do not. We're very aware of the current noise level. We test Mago daily against massive, multi-million-line codebases. On one such project, the first beta reported ~250,000 errors; we're now down to ~30,000. While still a lot, it shows how quickly we're closing the gap on false positives. Thanks again for the valuable feedback. It's a long road, but we're confident we can reach and surpass the current standards in a very short time.
- idoubtit 1y ago
  Thanks for explaining your goal and some of the context. I suggest that stating them prominently would help the project. When I read Mago's home page, I downloaded the latest release, followed the "Getting started" process on a local repository, then nearly lost interest when I saw the amount of false errors. If I had first read "here is our goal, with this roadmap, this kind of validation on real projects, and this position toward existing well-known tools", I would have been much more willing to follow the project and accept its false positives. As a side note, for my first try with Mago, I think its lack of parsing `@property` was a major source of false errors, because the source code it analyzed had a few omnipresent classes that used it. BTW, Mago panicked when I tried to lint another repository... I'll open a issue.
  - azjezz2 1y ago

eider 1y ago

I pulled one of classes from my project that does not depend on anything external and fed it to their demo site. It threw up bunch of errors complaining that RuntimeException is undefined. It seems it doesn't understand built-ins unless you prefix them with \, even though they are imported properly with use statement. This is pretty core issue to lack support for. Calling it "beta" is actually giving it a lock of slack, I'd say it's closer to being proof of concept.

azjezz2 1y ago
Mago author here. The online playground is running a very old version (~0.20.0 from months ago, which did not even have a static analyzer) and gives a poor impression of the current tool. That's on me to fix or take down. The issue you saw with built-in classes like `RuntimeException` was absolutely a bug in those early alpha versions, but it has been fixed for a long time now. The analyzer has matured a lot since then. The current beta is stable enough to be the sole static analysis tool for a couple of extremely well-typed projects: - https://github.com/azjezz/psl/ https://github.com/azjezz/psl/ - https://github.com/carthage-software/cel-php https://github.com/carthage-software/cel-php I'd definitely encourage trying the latest release locally to get a real feel for it. Thanks again for the write-up!

azjezz2 1y ago

Mago author here. Thanks for the feedback. You're right that the README should be clearer about the beta status and the current feature set. That's a great suggestion, and I'll get that updated. Regarding the errors you saw, I suspect the main culprit isn't a general lack of PHPDoc parsing, but rather the two biggest remaining features we're actively working on: support for magic _@method_ and _@property_ tags. Mago has full support for generics (@template), assertions (@psalm-assert*), conditional types, etc., but the absence of those two is definitely a major source of noise on established projects right now. They are our top priority and should land in the next beta release. On the topic of feature parity, you're right that it's a moving target. Our goal isn't to be a 1-to-1 clone of Psalm or PHPStan, but a different tool with its own strengths (see: https://github.com/carthage-software/mago/discussions/379 https://github.com/carthage-software/mago/discussions/379). For example, Mago will flag code like `[0 => $a, $b] = ["a", "b"]` as an error, which other tools currently do not. We're very aware of the current noise level. We test Mago daily against massive, multi-million-line codebases. On one such project, the first beta reported ~250,000 errors; we're now down to ~30,000. While still a lot, it shows how quickly we're closing the gap on false positives. Thanks again for the valuable feedback. It's a long road, but we're confident we can reach and surpass the current standards in a very short time.

idoubtit 1y ago
Thanks for explaining your goal and some of the context. I suggest that stating them prominently would help the project. When I read Mago's home page, I downloaded the latest release, followed the "Getting started" process on a local repository, then nearly lost interest when I saw the amount of false errors. If I had first read "here is our goal, with this roadmap, this kind of validation on real projects, and this position toward existing well-known tools", I would have been much more willing to follow the project and accept its false positives. As a side note, for my first try with Mago, I think its lack of parsing `@property` was a major source of false errors, because the source code it analyzed had a few omnipresent classes that used it. BTW, Mago panicked when I tried to lint another repository... I'll open a issue.
- azjezz2 1y ago