3 ms·

> not obvious to me Program X measured less than Program Y. > … comparisons where only one variable changes… Seems like your idea of a comparison, between fo

by igouy 8y ago

> not obvious to me Program X measured less than Program Y. > … comparisons where only one variable changes… Seems like your idea of a comparison, between for-example Lisp and Java programs, would require writing all the Lisp programs as if they were being written in Java. (We used up our "one variable change" with SBCL instead-of OpenJDK). > I spoke too soon You chose to summarize data: "Java outperforms SBCL by a very wide margin -- 1.25-4x faster". Do you mean that you made mistakes when you made your summary.

pron 8y ago
> Program X measured less than Program Y. That's like asking what does a program do and being told, "it does what the code says." Technically true, but not what we mean when we ask. Also, I don't think that's really what the site's authors thought. After all, when I click on the Haskell page, I don't see a comparison of Haskell programs, but a comparison of the fastest Haskell program with the fastest Java program. It's not obvious to me what that means, or even that it means anything at all. > Seems like your idea of a comparison, between for-example Lisp and Java programs, would require writing all the Lisp programs as if they were being written in Java. (We used up our "one variable change" with SBCL instead-of OpenJDK). That's exactly right, but because that's rather tedious, I would present just meaningful comparisons rather than non-meaningful ones, so no inter-language comparisons. I'm not saying they're always meaningless, but they can't be made with the data currently available on that site. Unfortunately, we cannot manufacture (non-fictitious) meaning out of data as we like. > Do you mean that you made mistakes when you made your summary. Yes. By "better than nothing" I overestimated the significance of results, at least in this case. After a closer look I realized that different programs use different algorithms, and so the comparison tells me nothing about the relative speed of those languages.
- igouy 8y ago
  > That's like… No it is not. Program X may not have measured less than Program Y. > … meaningful comparisons… Earlier you complained there was "code that's very unidiomatic" but now you say "writing all the Lisp programs as if they were being written in Java" would be exactly right? 1) You seem to be contradicting your own argument. 2) Your desire for "rigorous" comparison is in-conflict with your desire for inter-language comparison. > … different programs use different algorithms… For some programs, for some definition of "algorithm". > … the comparison tells me nothing about the relative speed of those languages. What is the speed of a programming language?
  - pron 8y ago
    > Earlier you complained there was "code that's very unidiomatic" but now you say "writing all the Lisp programs as if they were being written in Java" would be exactly right? I may have misinterpreted your comment. By "as if they were written in Java" I meant using a similar algorithm. Alternatively, there could be a rule about idiomatic code (enforced by experts), which would compare very different things, but would also be at least somewhat meaningful. > For some programs I don't know. Whenever someone refers me to a result on that site that looks weird, I see very different algorithms. > for some definition of "algorithm". For a very well-accepted definition of an algorithm in the context of real performance comparisons. > What is the speed of a programming language? I'm not sure (although perhaps I could come up with some definition given enough thought), but I didn't make a website that compares languages based on performance. In fact, I said that the website is totally meaningless. But the people behind it may have something in mind, and if they do, they're not telling us. All I can do is think of ways to use the site's data in ways that can at least be meaningful.
    - igouy 8y ago
      > … a similar algorithm. Will someone else look at those similar algorithms and "see very different algorithms"? > … a result on that site that looks weird, I see very different algorithms. Do regex libraries for different languages in-fact always implement the same algorithms? > What is the speed of a programming language? I'm not sure… You did say "Java outperforms SBCL by a very wide margin -- 1.25-4x faster". Saying "… so the comparison tells me nothing about the relative speed of those languages" suggests you are looking for something that tells us about "the relative speed" of those languages?
      - pron 8y ago
        > Will someone else look at those similar algorithms and "see very different algorithms"? I would think so. > Do regex libraries for different languages in-fact always implement the same algorithms? I don't know. But I do know that when something is presented as an answer, it's good to know what the question is. > suggests you are looking for something that tells us about "the relative speed" of those languages? I can think of a couple of ways the two can compare. One is the relative speeds (and other metrics) of programs of a similar nature, written simple idiomatic code. This gives a general "feel" for what kind of workloads different languages may be suitable for. Another is comparing the performance of the compiler/interpreter on some microbenchmark of the same algorithms, preferably written by people who coordinate, and preferably without mixing too many variables (e.g. measuring parallel scheduling separately and regex separately). Either one of these is of some interest. The benchmark game does neither. If there is some idea behind the site's comparisons, it's certainly keeping it a mystery.

by igouy 8y ago

pron 8y ago

> Program X measured less than Program Y. That's like asking what does a program do and being told, "it does what the code says." Technically true, but not what we mean when we ask. Also, I don't think that's really what the site's authors thought. After all, when I click on the Haskell page, I don't see a comparison of Haskell programs, but a comparison of the fastest Haskell program with the fastest Java program. It's not obvious to me what that means, or even that it means anything at all. > Seems like your idea of a comparison, between for-example Lisp and Java programs, would require writing all the Lisp programs as if they were being written in Java. (We used up our "one variable change" with SBCL instead-of OpenJDK). That's exactly right, but because that's rather tedious, I would present just meaningful comparisons rather than non-meaningful ones, so no inter-language comparisons. I'm not saying they're always meaningless, but they can't be made with the data currently available on that site. Unfortunately, we cannot manufacture (non-fictitious) meaning out of data as we like. > Do you mean that you made mistakes when you made your summary. Yes. By "better than nothing" I overestimated the significance of results, at least in this case. After a closer look I realized that different programs use different algorithms, and so the comparison tells me nothing about the relative speed of those languages.

igouy 8y ago
> That's like… No it is not. Program X may not have measured less than Program Y. > … meaningful comparisons… Earlier you complained there was "code that's very unidiomatic" but now you say "writing all the Lisp programs as if they were being written in Java" would be exactly right? 1) You seem to be contradicting your own argument. 2) Your desire for "rigorous" comparison is in-conflict with your desire for inter-language comparison. > … different programs use different algorithms… For some programs, for some definition of "algorithm". > … the comparison tells me nothing about the relative speed of those languages. What is the speed of a programming language?
- pron 8y ago
  > Earlier you complained there was "code that's very unidiomatic" but now you say "writing all the Lisp programs as if they were being written in Java" would be exactly right? I may have misinterpreted your comment. By "as if they were written in Java" I meant using a similar algorithm. Alternatively, there could be a rule about idiomatic code (enforced by experts), which would compare very different things, but would also be at least somewhat meaningful. > For some programs I don't know. Whenever someone refers me to a result on that site that looks weird, I see very different algorithms. > for some definition of "algorithm". For a very well-accepted definition of an algorithm in the context of real performance comparisons. > What is the speed of a programming language? I'm not sure (although perhaps I could come up with some definition given enough thought), but I didn't make a website that compares languages based on performance. In fact, I said that the website is totally meaningless. But the people behind it may have something in mind, and if they do, they're not telling us. All I can do is think of ways to use the site's data in ways that can at least be meaningful.
  - igouy 8y ago
    > … a similar algorithm. Will someone else look at those similar algorithms and "see very different algorithms"? > … a result on that site that looks weird, I see very different algorithms. Do regex libraries for different languages in-fact always implement the same algorithms? > What is the speed of a programming language? I'm not sure… You did say "Java outperforms SBCL by a very wide margin -- 1.25-4x faster". Saying "… so the comparison tells me nothing about the relative speed of those languages" suggests you are looking for something that tells us about "the relative speed" of those languages?
    - pron 8y ago
      > Will someone else look at those similar algorithms and "see very different algorithms"? I would think so. > Do regex libraries for different languages in-fact always implement the same algorithms? I don't know. But I do know that when something is presented as an answer, it's good to know what the question is. > suggests you are looking for something that tells us about "the relative speed" of those languages? I can think of a couple of ways the two can compare. One is the relative speeds (and other metrics) of programs of a similar nature, written simple idiomatic code. This gives a general "feel" for what kind of workloads different languages may be suitable for. Another is comparing the performance of the compiler/interpreter on some microbenchmark of the same algorithms, preferably written by people who coordinate, and preferably without mixing too many variables (e.g. measuring parallel scheduling separately and regex separately). Either one of these is of some interest. The benchmark game does neither. If there is some idea behind the site's comparisons, it's certainly keeping it a mystery.