3 ms·

I'm pretty surprised that Google chose to fork SunSpider without even suggesting their change to the maintainer (me) first, let alone providing a patch. That's

I'm pretty surprised that Google chose to fork SunSpider without even suggesting their change to the maintainer (me) first, let alone providing a patch. That's not to say I necessarily agree that running each test 50 times is a better test, but it seems somewhat outside open source norms of courtesy to fork and then blog about it without even proposing your change first.

scott_s 15y ago
I think that what they did is reasonable. I don't think even they think that running each test 50 times is best ("While repeating a trivial test many times isn’t a great solution, it does provide an opportunity for some optimization"). I think this is more of an experiment, less of a formal change request. I think it's appropriate to publicly post the results and code behind experiments.
pbiggar 15y ago
I would generally agree, but to any observer it looks like SunSpider isn't being actively developed: http://bit.ly/lScLRi http://bit.ly/lScLRi - my correctness bugs have gone unanswered since August, - other mozillians bugs have not been fixed either (at least one since 2009), - the SunSpider svn logs show that there is no work being done on the sunspider tests themselves, since the release of 0.9.1 in January 2010, - the last fix to the harness was 5 months ago (which broke shell tests for everyone except JSC). If you [1] want to remain the "maintainer" you have to show that's you're maintaining it. While this isn't how I would have gone about it (and I was thinking about it), I can't really fault Google for this. [1] And by "you", I mean "Apple", not you personally.
- sayrer 15y ago
  They should still file bugs and ping the maintainers, at least.
  - pbiggar 15y ago
    I agree that would be the right way to do it, and I'm sure that we [1] would have done it that way. There is a certain shoot-first-ask-questions-later approach to forking that, while unpleasant, is certainly effective. There are two ways Google could have gone: 1) Talk to Apple, file bugs, wait around for a year to see if anything is being done. Then threaten to fork, but wait when Apple says they're doing it, but they're really not, they're just trying to avoid a fork. Or worse, Apple would simply refuse, saying that it wasn't a good direction for the SunSpider benchmarks to go, which would be legitimate. Then Google is stuck with worse V8 scores forever. 2) Just fork it. They didn't cite a reason, but when asked they no doubt have a few up their sleeves. Now Apple is on the back foot, doesn't have a plan, and perhaps has lost maintainership of SunSpider forever. From a public perception point of view, this is a real winner for Google. They have better open source cred than Apple [2], and they keep V8 up-to-date pretty well. I would be unsurprised if people started reporting the forked scores from now on. That will probably depend on how they steward the project, for example by doing some trivial maintenance like the bugs we've reported on SunSpider. While I don't agree with this approach, I can see how it achieves their aims better. It's reminiscent of the launch of V8 or Google code, where the surprise is used to spur competition from the incumbents. With luck, Apple will update SunSpider like they suggested they would in 2009, there'll be a merge, everything will be forgiven, and JS benchmarking will be better. [1] for those watching at home, sayrer and I both work on the JS team at Mozilla [3] (he's actually my boss). [2] Google may have a closed development model for the V8 engine, but that's par for the course with Apple. [3] Nothing I say represents Mozilla, etc, etc.
- podperson 15y ago
  Indeed, all seems terribly quiet on the Apple JavaScript Engine front lately. For a while they were leading the pack (and, in some ways, they probably still are) but aside from Mobile Safari, not much lately.
  - calloc 15y ago
    I'm hoping that is mainly because of the run-up to the new OS X Lion release.
- othermaciej 15y ago
  If they filed bugs and got impatient, I'd understand. This blog post is the first I've heard of this particular idea though. Your bugs are good ones and should be addressed. I will try to do an update soon. FYI, if you want a patch approved, you should flag it for review (and make sure it follows the style guide), your patch would have been noticed much sooner had you flagged it.
  - pbiggar 15y ago
    > If they filed bugs and got impatient, I'd understand. I think it's acceptable to look at bugs others have filed to determine whether to file bugs. > I will try to do an update soon. Great, thanks! (I know you won't like this, but it sort of validates Google's approach. By making a big splash about this, they accomplish far more than they would have by filing bugs and waiting, or contacting you directly.) > FYI, if you want a patch approved, you should flag it for review Funnily enough, we're dealing with problems like this at Mozilla too - a contribution comes in, but is missed because the review flag isn't set, or it needs work, or whatever. We're adding dashboards and metrics and things to avoid this happening at Mozilla - and it happens a lot. I'll mark them for review now.
kragen 15y ago
I don't think so, although I know you've been involved in the open-source community since before it was called that, and more deeply than I have been. But consider: GNU Emacs was started in exactly this way, as a fork of Gosmacs. There are dozens of patched versions of the Linux kernel, most of which are not worth Linus's attention. Most Linux distributions ship patched versions of most of their packages, and in many cases, their changes are not suitable for incorporation in the upstream. Making a modified version of a piece of software is the very core of free software. In cases where you want the maintainer to accept your change, it's often a good idea to discuss it with them first in order to see if they agree that it's useful. However, in other cases, it's more useful to try it out first so that you have better data to go into the discussion with.
mbelshe 15y ago
Sorry to surprise you, Maciej. As others on this thread have already have already surmised, we were not trying to change (or fork) SunSpider. We just wanted to demonstrate the impact of the smallest possible change when we make it a longer test.

scott_s 15y ago

I think that what they did is reasonable. I don't think even they think that running each test 50 times is best ("While repeating a trivial test many times isn’t a great solution, it does provide an opportunity for some optimization"). I think this is more of an experiment, less of a formal change request. I think it's appropriate to publicly post the results and code behind experiments.

pbiggar 15y ago

I would generally agree, but to any observer it looks like SunSpider isn't being actively developed: http://bit.ly/lScLRi http://bit.ly/lScLRi - my correctness bugs have gone unanswered since August, - other mozillians bugs have not been fixed either (at least one since 2009), - the SunSpider svn logs show that there is no work being done on the sunspider tests themselves, since the release of 0.9.1 in January 2010, - the last fix to the harness was 5 months ago (which broke shell tests for everyone except JSC). If you [1] want to remain the "maintainer" you have to show that's you're maintaining it. While this isn't how I would have gone about it (and I was thinking about it), I can't really fault Google for this. [1] And by "you", I mean "Apple", not you personally.

sayrer 15y ago
They should still file bugs and ping the maintainers, at least.
- pbiggar 15y ago
  I agree that would be the right way to do it, and I'm sure that we [1] would have done it that way. There is a certain shoot-first-ask-questions-later approach to forking that, while unpleasant, is certainly effective. There are two ways Google could have gone: 1) Talk to Apple, file bugs, wait around for a year to see if anything is being done. Then threaten to fork, but wait when Apple says they're doing it, but they're really not, they're just trying to avoid a fork. Or worse, Apple would simply refuse, saying that it wasn't a good direction for the SunSpider benchmarks to go, which would be legitimate. Then Google is stuck with worse V8 scores forever. 2) Just fork it. They didn't cite a reason, but when asked they no doubt have a few up their sleeves. Now Apple is on the back foot, doesn't have a plan, and perhaps has lost maintainership of SunSpider forever. From a public perception point of view, this is a real winner for Google. They have better open source cred than Apple [2], and they keep V8 up-to-date pretty well. I would be unsurprised if people started reporting the forked scores from now on. That will probably depend on how they steward the project, for example by doing some trivial maintenance like the bugs we've reported on SunSpider. While I don't agree with this approach, I can see how it achieves their aims better. It's reminiscent of the launch of V8 or Google code, where the surprise is used to spur competition from the incumbents. With luck, Apple will update SunSpider like they suggested they would in 2009, there'll be a merge, everything will be forgiven, and JS benchmarking will be better. [1] for those watching at home, sayrer and I both work on the JS team at Mozilla [3] (he's actually my boss). [2] Google may have a closed development model for the V8 engine, but that's par for the course with Apple. [3] Nothing I say represents Mozilla, etc, etc.
podperson 15y ago
Indeed, all seems terribly quiet on the Apple JavaScript Engine front lately. For a while they were leading the pack (and, in some ways, they probably still are) but aside from Mobile Safari, not much lately.
- calloc 15y ago
  I'm hoping that is mainly because of the run-up to the new OS X Lion release.
othermaciej 15y ago
If they filed bugs and got impatient, I'd understand. This blog post is the first I've heard of this particular idea though. Your bugs are good ones and should be addressed. I will try to do an update soon. FYI, if you want a patch approved, you should flag it for review (and make sure it follows the style guide), your patch would have been noticed much sooner had you flagged it.
- pbiggar 15y ago
  > If they filed bugs and got impatient, I'd understand. I think it's acceptable to look at bugs others have filed to determine whether to file bugs. > I will try to do an update soon. Great, thanks! (I know you won't like this, but it sort of validates Google's approach. By making a big splash about this, they accomplish far more than they would have by filing bugs and waiting, or contacting you directly.) > FYI, if you want a patch approved, you should flag it for review Funnily enough, we're dealing with problems like this at Mozilla too - a contribution comes in, but is missed because the review flag isn't set, or it needs work, or whatever. We're adding dashboards and metrics and things to avoid this happening at Mozilla - and it happens a lot. I'll mark them for review now.

kragen 15y ago

I don't think so, although I know you've been involved in the open-source community since before it was called that, and more deeply than I have been. But consider: GNU Emacs was started in exactly this way, as a fork of Gosmacs. There are dozens of patched versions of the Linux kernel, most of which are not worth Linus's attention. Most Linux distributions ship patched versions of most of their packages, and in many cases, their changes are not suitable for incorporation in the upstream. Making a modified version of a piece of software is the very core of free software. In cases where you want the maintainer to accept your change, it's often a good idea to discuss it with them first in order to see if they agree that it's useful. However, in other cases, it's more useful to try it out first so that you have better data to go into the discussion with.

mbelshe 15y ago

Sorry to surprise you, Maciej. As others on this thread have already have already surmised, we were not trying to change (or fork) SunSpider. We just wanted to demonstrate the impact of the smallest possible change when we make it a longer test.