6 ms·

This doesn't seem to investigate tuning at all. I see the Tornado script forks processes, for example, but it doesn't experiment with different values (or PyPy

by mrj 14y ago

This doesn't seem to investigate tuning at all. I see the Tornado script forks processes, for example, but it doesn't experiment with different values (or PyPy?). The Java code appears to be run with default VM settings, which pretty much always deserves some tweaks to run in a server environment. Those are just the two I'm most familiar with. The rest seem to have similar faults. Plus, it would be a more interesting test if the server had to perform some kind of work. Simply echoing the request is not a typical usage and could really bias the results in favor of setups that would fail on a real project. I better title might be, "Erlang wins unrealistic test over other VMs in their default configuration."

dvirsky 14y ago
I contributed the tornado script to this project, but the results of it are yet to be published (or if they have I've missed them). I haven't tested it myself on AWS, just on a physical desktop machine, I got great results, but they are so good it of course seems to be an irrelevant comparison. I have benchmarked the ws4py code in the test on that machine, and it was much slower than the tornado code - quite obvious since it ran in a single process thus on a single CPU. BTW The default number of processes when you fork is the number of CPUs, which sounded like a fair estimate to me, not knowing the target machine, so I left it.
huggyface 14y ago
This doesn't seem to investigate tuning at all. This is always the response to any benchmark where one's pet technologies don't win. If you have a magic quadrant of tuning, put it forth. If not, you have said nothing that counters the results.
- hermanhermitage 14y ago
  Indeed. It seems pretty clear to me it is framed as an benchmark of non tuned idiomatic performance. Maybe it doesn't call it out explicitly - but in my view that is a reasonable enough test. I'm pretty sure all the platforms could support an FFI binding or equivalent to an optimized epoll C implementation - or hours of tuning.
- SkyMarshal 14y ago
  He doesn't have to, it's enough just to point out gaping flaws in the 'study' to call it into question. The onus is on the study's author to make it comprehensive and thorough, not the critics. This one in particular is clearly incomplete, especially when things like Haskell, Java, and Python are represented by a single framework, and not even the best or most optimized ones. For example, why not use Yesod/Warp, BlueEyes, and Tornado on PyPy, respectively, for those instead? Also, the submission title is editorializing flame bait. The study's author didn't make any claims whatsoever about 'Haskell', 'Java', and 'Python', etc. but rather about Snap, Webbit, and ws4py.
  - davidw 14y ago
    > The onus is on the study's author to make it comprehensive and thorough, not the critics. It's always possible for the critics to go out there and do something better, rather than limiting themselves to pointing out flaws.
    - politician 14y ago
      Doing both is also possible...
    - calibraxis 14y ago
      The problem is that it takes effort to make good comparative benchmarks — for a platform you're not totally immersed in, you'd ideally speak with the community about your methods. Furthermore, good benchmarks require real investigation explaining the differences; otherwise you could be measuring something silly. And you should graph some of the stats by time. Maybe in an ideal world, quality benchmarks would be commonplace. But in this world, doing good benchmarks in response to everyone's bad benchmarks is a heavy burden.
    - pyre 14y ago
      So what you're really saying is that no one is 'allowed' to make a critical remark unless they are willing to stand up and do better themselves?
    - cantankerous 14y ago
- jfoutz 14y ago
  It seems more about running on an amazon instance and getting throttled. Haskell got every successful result back in less than a second, erlang took 7 seconds and go took 49(!) seconds to get an answer back. I'm really impressed that erlang didn't drop a single connection. I suspect if you ran the same test with just erlang 5 times, you'd see it start to behave like go.
  - nosequel 14y ago
    > I suspect if you ran the same test with just erlang 5 times, you'd see it start to behave like go. I don't think it would. Erlang was designed for things like this exactly and has many many many years up on the competition (Go in this case). I'm sure Go will get there, but by then, with Erlang's further threading improvements coming in R16 it will also get better.
- jbooth 14y ago
  Sure, but language shootouts are more commonly about the code run than the language itself. I really like Go and can believe that Erlang performs well. For the java example.. wtf is "webbit"? I've programmed Java for 10 years and never heard of that. Why doesn't he just find a way to run on Jetty or Tomcat like everyone else? I'd believe for an example like this that native languages can outperform java, because it's basically a ton of no-op web requests. But Java underperforming Node and frickin Python? Python's a great language but famously slow. No way is this study legit.
  - akkartik 14y ago
    But java did better than node and python.
  - fizx 14y ago
    Webbit seems like a thin wrapper around Netty to make Netty do websockets. Unless Webbit grossly screws things up, it seems like a reasonable choice, because all the work falls to Netty, a well-known and much-used library.
- mibbitier 14y ago
  It's not even a benchmark of languages. So I have no idea why it's the languages listed in the title. It's a benchmark of specific websocket implementations, run with default configurations, which happen to be in different languages. Drawing any conclusions about the merits of each language based on the results would be very foolish.
- _delirium 14y ago
  It's a common response to off-the-cuff benchmarks, yes, but imo a deserved one. Most peer-reviewed, published benchmarks go to considerably more effort to make sure they're comparing like to like, using settings that would be considered sensible by users of each of the technologies under consideration, etc., etc. In part, that's because you wouldn't be able to get your benchmark paper accepted to a reputable journal if you didn't do that. Basically, benchmarking is hard, and if you threw together a quick benchmark over a weekend, there is a good chance it might not be representative.
  - scoot 14y ago
    If there are sensible defaults and a technology doesn't use them "by default", then that is a failing of the technology, not the benchmark.
    - mistercow 14y ago
      Arguably, though, what makes a default "sensible" depends on how the technology is being used. I'm speculating here, but it could be that there are settings that greatly increase performance with < 1000 connections, but greatly degrade performance with > 3000 connections. In that case, would it not be a sensible default to start out with the fast, lower scale option?
    - fijal 14y ago
      Is using pypy a reasonable default? Who's there to make that choice? I suppose the answer is "no", while it makes sense to use it while benchmarking.
- deleted 14y ago
  [deleted]
regularfry 14y ago
It's perfectly valid to say "this is what you get out of the box" which is how a large proportion of people are going to be running their servers. Admittedly, not a lot of people are going to be both running a server in a default configuration and having to deal with C10K, but slashdotting can happen to anyone.

by mrj 14y ago

dvirsky 14y ago

I contributed the tornado script to this project, but the results of it are yet to be published (or if they have I've missed them). I haven't tested it myself on AWS, just on a physical desktop machine, I got great results, but they are so good it of course seems to be an irrelevant comparison. I have benchmarked the ws4py code in the test on that machine, and it was much slower than the tornado code - quite obvious since it ran in a single process thus on a single CPU. BTW The default number of processes when you fork is the number of CPUs, which sounded like a fair estimate to me, not knowing the target machine, so I left it.

huggyface 14y ago

This doesn't seem to investigate tuning at all. This is always the response to any benchmark where one's pet technologies don't win. If you have a magic quadrant of tuning, put it forth. If not, you have said nothing that counters the results.

hermanhermitage 14y ago
Indeed. It seems pretty clear to me it is framed as an benchmark of non tuned idiomatic performance. Maybe it doesn't call it out explicitly - but in my view that is a reasonable enough test. I'm pretty sure all the platforms could support an FFI binding or equivalent to an optimized epoll C implementation - or hours of tuning.
SkyMarshal 14y ago
He doesn't have to, it's enough just to point out gaping flaws in the 'study' to call it into question. The onus is on the study's author to make it comprehensive and thorough, not the critics. This one in particular is clearly incomplete, especially when things like Haskell, Java, and Python are represented by a single framework, and not even the best or most optimized ones. For example, why not use Yesod/Warp, BlueEyes, and Tornado on PyPy, respectively, for those instead? Also, the submission title is editorializing flame bait. The study's author didn't make any claims whatsoever about 'Haskell', 'Java', and 'Python', etc. but rather about Snap, Webbit, and ws4py.
- davidw 14y ago
  > The onus is on the study's author to make it comprehensive and thorough, not the critics. It's always possible for the critics to go out there and do something better, rather than limiting themselves to pointing out flaws.
  - politician 14y ago
    Doing both is also possible...
  - calibraxis 14y ago
    The problem is that it takes effort to make good comparative benchmarks — for a platform you're not totally immersed in, you'd ideally speak with the community about your methods. Furthermore, good benchmarks require real investigation explaining the differences; otherwise you could be measuring something silly. And you should graph some of the stats by time. Maybe in an ideal world, quality benchmarks would be commonplace. But in this world, doing good benchmarks in response to everyone's bad benchmarks is a heavy burden.
  - pyre 14y ago
    So what you're really saying is that no one is 'allowed' to make a critical remark unless they are willing to stand up and do better themselves?
  - cantankerous 14y ago
jfoutz 14y ago
It seems more about running on an amazon instance and getting throttled. Haskell got every successful result back in less than a second, erlang took 7 seconds and go took 49(!) seconds to get an answer back. I'm really impressed that erlang didn't drop a single connection. I suspect if you ran the same test with just erlang 5 times, you'd see it start to behave like go.
- nosequel 14y ago
  > I suspect if you ran the same test with just erlang 5 times, you'd see it start to behave like go. I don't think it would. Erlang was designed for things like this exactly and has many many many years up on the competition (Go in this case). I'm sure Go will get there, but by then, with Erlang's further threading improvements coming in R16 it will also get better.
jbooth 14y ago
Sure, but language shootouts are more commonly about the code run than the language itself. I really like Go and can believe that Erlang performs well. For the java example.. wtf is "webbit"? I've programmed Java for 10 years and never heard of that. Why doesn't he just find a way to run on Jetty or Tomcat like everyone else? I'd believe for an example like this that native languages can outperform java, because it's basically a ton of no-op web requests. But Java underperforming Node and frickin Python? Python's a great language but famously slow. No way is this study legit.
- akkartik 14y ago
  But java did better than node and python.
- fizx 14y ago
  Webbit seems like a thin wrapper around Netty to make Netty do websockets. Unless Webbit grossly screws things up, it seems like a reasonable choice, because all the work falls to Netty, a well-known and much-used library.
mibbitier 14y ago
It's not even a benchmark of languages. So I have no idea why it's the languages listed in the title. It's a benchmark of specific websocket implementations, run with default configurations, which happen to be in different languages. Drawing any conclusions about the merits of each language based on the results would be very foolish.
_delirium 14y ago
It's a common response to off-the-cuff benchmarks, yes, but imo a deserved one. Most peer-reviewed, published benchmarks go to considerably more effort to make sure they're comparing like to like, using settings that would be considered sensible by users of each of the technologies under consideration, etc., etc. In part, that's because you wouldn't be able to get your benchmark paper accepted to a reputable journal if you didn't do that. Basically, benchmarking is hard, and if you threw together a quick benchmark over a weekend, there is a good chance it might not be representative.
- scoot 14y ago
  If there are sensible defaults and a technology doesn't use them "by default", then that is a failing of the technology, not the benchmark.
  - mistercow 14y ago
    Arguably, though, what makes a default "sensible" depends on how the technology is being used. I'm speculating here, but it could be that there are settings that greatly increase performance with < 1000 connections, but greatly degrade performance with > 3000 connections. In that case, would it not be a sensible default to start out with the fast, lower scale option?
  - fijal 14y ago
    Is using pypy a reasonable default? Who's there to make that choice? I suppose the answer is "no", while it makes sense to use it while benchmarking.
deleted 14y ago
[deleted]

regularfry 14y ago

It's perfectly valid to say "this is what you get out of the box" which is how a large proportion of people are going to be running their servers. Admittedly, not a lot of people are going to be both running a server in a default configuration and having to deal with C10K, but slashdotting can happen to anyone.