10 ms·

Scala vs. Go TCP Benchmark

139 points by rck 13y ago

shin_lao 13y ago
I think the JVM TCP stack is extremely mature, but I'm really surprised to see that Go is ten times slower. It would have been interesting to have a C/C++ benchmark for reference.
- skrebbel 13y ago
  Your comment makes me wonder whether it matters on which JVM the benchmark is run. Are Oracle, GNU Classpath and Dalvik equally fast?
- wyuenho 13y ago
  With Go using 20 times less memory, I'd take Go any day since I can just throw in 20 more processes and up my throughput 5-10x. But of course, scaling like this takes a bit of design and the processes have to share nothing.
  - shin_lao 13y ago
    I would take the memory measurement with a grain of salt. A 200 MiB large JVM doesn't mean all the 200 MiB are used. They could have been reserved (preemptive memory allocation).
    - levosmetalo 13y ago
      I agree. I've seen so many times clueless people complain about JVM memory footprint for tiny "benchmarks". In practice, it's never an issue for long running web applications. That, and people that don't understand how JIT works and why it has start-up penalty.
      - chetanahuja 13y ago
        In practice, it's never an issue for long running web applications. You might want to clarify that statement a bit. It almost sounds like you are implying that memory pressure is never an issue for long running web applications in java. Did you mean to say something else?
        levosmetalo 13y ago
        The initial amount of memory required by VM, while being significant for small command line utilities is only a fraction of total memory required by an application. In a web application tuned for performance a lot of the memory will be used for caching anyways. Also, I'd be glad to pay a small memory hit upfront if that means that I will get a top quality GC and very low probability of memory leaks in the long run. As for the startup time, MojoJolo summed it up.
        chetanahuja 13y ago
        I note that the memory pressure question got swept under the rug there :-) That's ok. It's not a revelation to anybody here (I hope) the enormous cost in memory overhead you have to pay for acceptable performance from the JVM. That "top quality GC" basically requires a 2X overhead (on top of your actual cache payload) to perform with reliable low latency and high throughput.
        fauigerzigerk 13y ago
        I agree completely. And in spite of that "top quality" GC and all the tuning in the world you're still running the risk of having the world stop on you for tens of seconds on larger systems. The JVM (at least OpenJDK, probably not Azul) is quickly becoming untenable as server memory goes into the hundereds of GBs. I'm reluctantly moving back to C++ for that reason alone.
        virmundi 13y ago
        How do you get around heap fragmentation? I know that the JVM (Oracle I believe) is really limited to about 32 GB of RAM before it has real issues. But the nice thing is that the GC will compact the heap for better future performance. As a possible work around to the JVM limit, a distributed architecture with N JVMs running a portion of the task could solve the small memory space with minimal system overhead. What I mean by this let's say you need to have 64 GB of memory for your app. Given the comment above, Java would not do well with this. But you could have 4 16 GB VMs each handling 1/4 of the work. The GC would prevent fragmentation that you'd see in long running C++ apps and still provide you with operational capacity.
      - MojoJolo 13y ago
        @chetanahuja, I think he means that long running web apps was just initialized once and will continue running until they are stopped or restarted. So even though starting up JVM may takes time, that overhead was only done once.
    - wyuenho 13y ago
      It doesn't matter whether they are used or not by the JVM, to the OS, they are allocated and gone. Startup time is a moot point in long running apps.
      - jongraehl 13y ago
        Allocated memory that's paged out and not used is nearly irrelevant (I believe this is true even though a 200mb JVM heap will likely see some unnecessary rotation in/out due to GC - ideally you'd manually configure the heap appropriately (smaller) if you intended to challenge total system memory w/ many processes).
  - trailfox 13y ago
    With Go using 20 times less memory, I'd take Go any day since I can just throw in 20 more processes and up my throughput 5-10x. But of course, scaling like this takes a bit of design and the processes have to share nothing. That doesn't even make sense. The Scala version can simply spin up more threads. It's not an issue of parallelism, the Scala version just happens to be faster, throwing more Go processes at the problem won't help. Go is already multithreaded. You don't need multiple processes to use the cores on the box. The JVM scales memory usage by default (heap size etc.) based on the amount of memory on the host. If you're going to worry about 100 MB RAM it's easy to constrain the JVM with -Xmx
    - deleted 13y ago
      [deleted]
- vidarh 13y ago
  If the JVM and/or the Scala runtime does user-space buffering and Go forwards straight to the read/write syscalls, that alone would probably sufficient to explain the difference when reading/writing buffers this small. If you don't do buffering in user space, the context switches in/out of the kernel will kill you when you do many small read or writes. No idea if that's it, but that's the first place I tend to look when troubleshooting networking app performance as so many people just blindly use syscalls as if they're free.
  - jongraehl 13y ago
    The better performance of the Scala client+server if anything suggests less buffering, not more, since the next ping can't be written until the previous pong is received. p.s. to parent: "the JVM TCP stack?" really?
    - vidarh 13y ago
      I admit I haven't checked the example thoroughly - if it goes in lockstep then buffering won't be the culprit. But you're wrong that better performance implies less buffering. A typical way to write such applications to do less buffering is to do select() or poll() or equivalent followed by a large non-blocking read, and then picking your data out of that buffer. As pointed out above, if this "benchmark" does ping/pong's in lockstep across a single connection, then buffering vs no-buffering will make exactly no difference, as there's no additional data available to read. But in scenarios where the amount of data is larger, the time saved from fewer context switches quickly adds up and gives you far more time to actually process the data. Usually your throughput will increase, but your latency will also tend to drop despite the buffering as long as the buffers are of a reasonable size. Buffering is a problem when the buffers grows to large multiples of the typical transaction size, but for typical application level protocols that takes really huge buffers.
      - jongraehl 13y ago
        My comment was specific to this benchmark, which has 4 byte messages that cannot be buffered due to the ping (wait for pong) ping ... repetition in each client. Of course buffering matters for full-pipeline throughput.
damian2000 13y ago
Isn't this more like a comparison of the JVM's TCP library vs Go's TCP library... not so much Scala vs Go?
- est 13y ago
  They are all wrappers around the OS TCP/IP stack.
willvarfar 13y ago
The benefit of go is the cheap threads and CSP which make it scale well for complex servers. I think we'll see the Go runtime gaining Single System Image distribution, performance improvements and libraries not services (e.g. groupcache) making it a very different world to develop in than Scala.
- octo_t 13y ago
  Scala has Actors, which are as pure CSP as you can get (Go's CSP has shared mutable state, Actors do not).
dschiptsov 13y ago
... that the Go server had a memory footprint of only about 10 MB vs. Scala’s nearly 200 MB. Priceless! 200Mb for such a crappy server with only hundred connections. So, it is not about whose wrapper around epoll is thinner, but about how the data are represented and copied.
- ratherbefuddled 13y ago
  You can't really come to a conclusion since no information is given about the configuration of the JVM. Most likely the minimum heap size was unset which often means it will default to 1/64 of the total system memory.
  - dschiptsov 13y ago
    I have my conclusions for quite long time.)
- coldtea 13y ago
  >200Mb for such a crappy server with only hundred connections. Yes, and almost a megabyte of disk space for a "hello world" program. C++ is so inefficient, right? That's not the way you judge the performance of a program and/or it's memory use -- and neither is comparing the 200MB of the JVM with the 10MB of Go here...
  - dschiptsov 13y ago
    No, it is almost 2Mb per connection. Why so-called performance of a wrapper around OS syscalls is more important than resource costs of having a pending connection?)
est 13y ago
> The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally, so as to better measure pure processing overhead. The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time. Another let's rape localhost:8080 on a MacBook Pro™ benchmark
- aa0 13y ago
  Benchmarked? More like benchmucked.
  - rkangel 13y ago
    The way I see it, instead of basing a decision on what's currently fashionable, they've gone for some empirical data and have actually conducted an experiment. Sure this is a fairly simplistic first step, but so should all first steps be as you gain understanding, and it's proved its worth by producing a surprising result in a very simple case, that's worth understanding before moving on to more complicated tests. What you've got here is engineering, rather than just making it up as you go along.
    - dmit 13y ago
      The only conclusion that can be reached from this is that a certain implementation of a TCP client/server for sending 4-byte payloads over the OS X loopback interface written in Scala is faster than another implementation written in Go. That conclusion in itself is absolutely useless, because nobody would ever use that particular code to perform that particular task. But it's also dangerous since it generalizes the result of this particular test to imply that one language is better than another at all tasks related to pumping bytes over the wire. And that aside, it still fails as an experiment because the results are neither provable nor repeatable. There is not enough information about the environment used to run the test and there is no detailed test output. We get two numbers ("average round trip" for each implementation), but we can't check that they are correct, and we can't look at the distribution or calculate other, potentially more useful metrics. Even the code provided is different from the code used in the actual test.
madisp 13y ago
"The actual test code contained some functionality to deal with connection errors, omitted here for brevity." Any way to see the actual test code?
anuraj 13y ago
It is the JVM rather than Scala - one of the fastest VMs implemented till date.
aaron42net 13y ago
If the eventual production app would run on Linux (which I'm only guessing based on the context), this benchmark should probably be run there. Darwin's surprisingly higher system call and context switch overhead can be deceptive for apps that are OS-bound.
cmccabe 13y ago
EDIT. OK, so. I ran this benchmark myself on an 8-core Xeon running Linux. 2.13 GHz, CentOS 6.2. Kernel was 2.6.32-220.el6.x86_64. 50 gigs of RAM. I got somewhere between 2.0 and 2.2 "milliseconds per ping" for Scala 2.9, and somewhere between 3.5 and 3.7 for Go 1.1. This is not the 10x difference that the authors reported, but it is something. The difference may be due in part to the different platform and hardware I am using. Contrary to what I wrote earlier, I noticed that GOMAXPROCS=8 did seem to be slower than GOMAXPROCS=4 here. I got around 4 "milliseconds per ping" with GOMAXPROCS=8. Using a mutex and explicit condition variable shaved off maybe 0.2 milliseconds on average (very rough estimation). Again contrary to what I wrote earlier, Nagle on versus off didn't seem to matter in the Go code. I still think you should always have it off for a test like this, but on my setup I did not see a difference. I still don't think this benchmark is showing what they think it is. I have a hunch that this is more of a scheduler benchmark than a TCP benchmark at all. I think I'd have to haul out vtune to get any further, and I'm getting kind of tired (after midnight here).
- BarkMore 13y ago
  Nagle's algorithm is disabled by default in Go (http://golang.org/pkg/net/#TCPConn.SetNoDelay http://golang.org/pkg/net/#TCPConn.SetNoDelay).
- jongraehl 13y ago
  > First of all, they're testing on MacOS, which is not going to be the platform they're actually using for the server code. The backends can be very different. Unless you have evidence that Go has MacOS-only bugs, this is meaningless speculation (though I agree that it's possible that there's no problem on Linux and that people don't generally run mac servers, I'm not sure why we should privilege your hypothesis). Agreed about GOMAXPROCS=4 - that seemed questionable to me (I don't know what it does, precisely, but I don't see a 4-anything limit in the Scala code).
  - jongraehl 13y ago
    Thanks to parent (after edit) for really testing. It turns out that he was right that Go on Mac is substantially worse than Linux - my bad. Maybe the Go lib authors didn't put much effort into reading the subtly different BSD/Darwin vs Linux syscall semantics. To explain the Nagle algorithm's irrelevance to this case, we have to understand how it works. It doesn't delay any writes once you read. It only affects two consecutive small writes (my memory was fuzzy so I checked http://en.wikipedia.org/wiki/Nagle's_algorithm http://en.wikipedia.org/wiki/Nagle's_algorithm ). Odds of preemption between the client's write and its read seem small, so it shouldn't matter whether you Nagle or not.
dlsspy 13y ago
Did you consider running the go client against the scala server and vice versa? Also, that's kind of a lot of code. Here's my rewrite of the server: http://play.golang.org/p/hKztKKQf7v http://play.golang.org/p/hKztKKQf7v It doesn't return the exact same result, but since you're not verifying the results, it is effectively the same (4 bytes in, 4 bytes back out). I did slightly better with a hand-crafted one. A little cleanup on the client here: http://play.golang.org/p/vRNMzBFOs5 http://play.golang.org/p/vRNMzBFOs5 I'm guessing scala's hiding some magic, though. I made a small change to the way the client is working, buffering reads and writes independently (can be observed later) and I get similar numbers (dropped my local runs from ~12 to .038). This is that version: http://play.golang.org/p/8fR6-y6EBy http://play.golang.org/p/8fR6-y6EBy Now, I don't know scala, but based on the constraints of the program, these actually all do the same thing. They time how long it takes to write 4 bytes * N and read 4 bytes * N. (my version adds error checking). The go version is reporting a bit more latency going in and out of the stack for individual syscalls. I suspect the scala version isn't even making those, as it likely doesn't need to observe the answers. You just get more options in a lower level language.
- jongraehl 13y ago
  I think you're on the right track in supposing that there can't be a huge performance difference in such a simple task, given that both languages are compiled and reasonably low-level. The most plausible explanation would amount essentially to a misconfigured library, not a fundamental advantage due to say, advanced JVM JIT. Your suggestion to try server-{a,b} x client-{a,b} is also a good one. Your modified Go server doesn't return "Pong" for "Ping". It returns "Ping". And the "a small change" version is nonsense. It's fundamentally different. - you're firing off all your requests before waiting for any replies, and so hiding the latency in the more common RPC style request-response chain, which is a real problem. You speculate a lot ("hiding some magic" "likely doesn't need to observe the answers") when you haven't offered any insight. EDIT: Nagle doesn't matter here - it doesn't delay any writes once you read (waiting server response). It only affects 2+ consecutive small writes (here I'm trusting http://en.wikipedia.org/wiki/Nagle's_algorithm http://en.wikipedia.org/wiki/Nagle's_algorithm - my own recollection was fuzzy). If Go sleeps client threads between the ping and the read-response call then I suppose it would matter (but only a little? and other comments say that Go defaults to no Nagle alg. anyway).
deleted 13y ago
[deleted]
luikore 13y ago
So this Go server is slower than a single threaded, blocking TCP server in Ruby. And the memory? Almost the same: require "socket" s = TCPServer.new 'localhost', 1201 loop do c = s.accept if c.read(4) c << 'Pong' end c.close end
- dlsspy 13y ago
  It's not obvious to me what that does. Does that only service one of the 100 clients or one of their 10,000 pings each? Or both?
  - luikore 13y ago
    It's one at a time. But, since the server only performs a one-shot task without the "hang on and wait 30 seconds"-like long connections, and, the default socket backlog of TcpServer is > 100, so every client gets served within the delay of (0~99)*6ms. In short, every round is fully served, and the concurrency level is 100.
- jaekwon 13y ago
  hmm, i tried running that against the go client but the client wouldn't connect. dial tcp 127.0.0.1:1201: connection refused i changed 'localhost' to '127.0.0.1' and now I see a whole lot of dial tcp 127.0.0.1:1201: connection reset by peer some connections were successful though.
  - luikore 13y ago
    The first one may be your localhost not mapped to 127.0.0.1 ? The second one may be a problem of backlog size. On my system (ruby 2.1.0dev (2013-08-06) [x86_64-darwin12.4.0]) it's 128 and there's no reset. But you can manual change it with s.listen(1000)
smegel 13y ago
Quick look at the language benchmark games, and Go is not 10x slower than Java for most tests. Java is often 3-5x slower than C/C++/etc (although maybe it didn't get enough time to warm up).
- trailfox 13y ago
  A quick look at the language benchmark game, and Go varies between 7x slower than Java and approx. the same speed as Java (varies by test): http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=java http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... I fail to see how Go being slower than Java relates to C being faster than Java.
  - smegel 13y ago
    Well given both C and Go are native compiled languages, I would think C is a realistic (if distant) goal for Go performance, at least for algorithmic stuff where you're not bouncing in and out of the runtime. I was commenting on how far it has to go.
    - trailfox 13y ago
      Ok, I see. Technically a JIT compiles to native code too, so for a long running app there shouldn't be much difference. Both Go and Java are garbage collected, but the JVM has the more sophisticated GC.
bad_user 13y ago
So even if Scala is ahead in this (flawed) benchmark, that's not how you write a TCP server in Scala, because you want to do it non-blocking. Not doing it based on asynchronous I/O means that in a real-world scenario the server will choke under the weight of slow connections, not to mention be susceptible to really cheap DoS attacks like Slowloris [1]. Seriously, it goes beyond the underlying I/O API that you're using. If anywhere in the code you're reading from an InputStream or you're writing to an OutputStream that's connected to an open socket, then that's a blocking call that can crush your server. Right now, every Java Servlets container that's not compatible with the latest Servlets API 3.1 can be brought down with Slowloris, even if under the hood they are using NIO. Option A for writing a server in Scala is Netty [2]. Option B for writing a server in Scala is the new I/O layer in Akka [3]. [1] http://ha.ckers.org/slowloris/ http://ha.ckers.org/slowloris/ [2] http://netty.io/ http://netty.io/ [3] http://doc.akka.io/docs/akka/snapshot/scala/io.html http://doc.akka.io/docs/akka/snapshot/scala/io.html
- caniszczyk 13y ago
  Also see Finagle (uses Netty): http://twitter.github.io/finagle/ http://twitter.github.io/finagle/
geal 13y ago
Ok, so. When you need to write a load balancer and want to test different languages for the task, you don't do a benchmark like that one. Writing a "ping-pong"? And not using the same client to test both servers? It would not have been too hard to write a simple proxy in both languages. Not even worrying about parsing HTTP headers, just testing TCP, that is really easy. Now, if you really want to test the performance, you have to implement it differently. Just two small features you would need in both: * non blocking IO: right now, you're starting a new future in Scala, that's easy to write but not really efficient (it might work better with goroutines) * zero copy: if you're load balancing, you will spend your time moving bits, so you'd better make sure that you don't copy them too much. It is possible with Scala, but it looks like Go does not support it Now, when you have reasonable testing grounds (that woudn't be more that a hundred lines in both languages), better get your statistics right. "The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time" -> that is NOT how you should test. Here, you would want to know what is the nominal pressure the balancer could handle, so you must measure a lot of metrics: * RTT * bandwidth (per user and total) * time to warmup (the JVM optimizes a lot of things on the fly, you have to wait for it) * operational limits (what is the maximal bandwidth for which the performance crashes? same for number of users) And then, you don't measure only the average values. You must measure the standard deviation. Because you could have a good average, but wildly varying data, and that is not good. Last thing: the macbook may not be a good testing system.
jaekwon 13y ago
Results from my Mac: Go server vs Go client: 10ms Scala server vs Scala client: 3ms Go server vs Scala client: 4ms Scala server vs Go client: ???? Scala server against the Go client is really slow (?). I reduced the ping count by a factor of 100, and extrapolating I think it would have reported around 670ms. What gives? I don't know much about Scala Futures, but isn't Scala's client doing something completely different than the Go client? Scala's client with Future.sequence looks like it's calling each `ping` method sequentially. Printing the connection identifier {0,100} on open & close shows that while it isn't completely sequential, only about a handful of connections are open at a time. On the other hand, the Go client appears to switch amongst goroutines more frequently. All the connections open before any connection closes. In other words, the I think the difference in performance is due to the difference in how randomly the connections are shuffled. The terrible performance time in the last case I think shows a bottleneck in the Scala server rather than the Go client.
corresation 13y ago
As a word of advice -- you are almost certainly wasting your time writing a load balancer. There are close to zero cases where someone can legitimately justify such an exercise. In any case, your benchmark is flawed (as virtually all benchmarks are). The reason Go the client is slower is because it tries to pervasively "thread" via the M:N scheduler -- every wait check causes it to yield the actual thread and switch goroutine, creating a large amount of overhead. The Scala cases, on the other hand, is dramatically more limited and will not yield this overhead. The Go server does not have this fault, and is likely top performance. And aren't we talking about a server anyways? Now as to the client, while we could naively criticize M:N scheduling based upon this, try giving it a more realistic workload (unless you seriously plan on load balancing pongs): Instead of ping/pong, return larger lengths of data preferably over actual network connections (not localhost) - e.g. 32KB. The Go client will catch up if not shoot into the lead. M:N scheduling is optimal for most real-world workloads, though it is less optimal for spin-off-a-million-goroutines that do nothing type tests. This is not a test of TCP overhead, or a realistic test, but instead demonstrates the small overhead of goroutines when you give each a minuscule amount of wait work.
- diakritikal 13y ago
  You're right about the Go client, I'm also scratching my head as to why it's doing the extra lookup instead of just using a dial function directly: tcpAddr, _ := net.ResolveTCPAddr("tcp4", "localhost:1201") conn, _ := net.DialTCP("tcp", nil, tcpAddr) Vs. val socket = new Socket("localhost", 1201) Edit: Go has e.g. net.Dial("tcp", "localhost:1201") and as someone pointed out elsewhere for a more accurate bench in both clients why not use the numerical address instead?
Oculus 13y ago
What would be the advantages of using a custom built load balancer vs. something off the shelf like Nginx?
voidlogic 13y ago
>The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally No no no. Assuming your production code runs on Linux THAT is where you need to do this test. It is extremely naive to assume that either the JVM or the Go runtime will perform system interfacing tasks even remotely similar between OSX and Linux. Linux is what you will use in production. Linux is almost always faster (more effort from devs both on the kernel TCP/IP side and the runtime/userspace side). Write your Scala, Java, Go wherever you want, but please, benchmark it in a clone of your production environment! P.S. In production I assume your client and server will not be local... don't do this, kernels do awesome/dirty optimizations over loop-back interfaces, sometimes even bypassing large parts of the TCP/IP stack, parts you want included in any meaningful benchmark.
msie 13y ago
Ugh, after reading all the comments here I wonder how a mere-mortal programmer gets multi-threaded, network programming done right. It's not clear to me if there is a clear winner between Go and Scala/JVM. Are the majority of programs out there crappy, memory-hogging and non-performant? EDIT: Any good references out there? Thanks!

139 points by rck 13y ago

I think the JVM TCP stack is extremely mature, but I'm really surprised to see that Go is ten times slower. It would have been interesting to have a C/C++ benchmark for reference.

skrebbel 13y ago
Your comment makes me wonder whether it matters on which JVM the benchmark is run. Are Oracle, GNU Classpath and Dalvik equally fast?
wyuenho 13y ago
With Go using 20 times less memory, I'd take Go any day since I can just throw in 20 more processes and up my throughput 5-10x. But of course, scaling like this takes a bit of design and the processes have to share nothing.
- shin_lao 13y ago
  I would take the memory measurement with a grain of salt. A 200 MiB large JVM doesn't mean all the 200 MiB are used. They could have been reserved (preemptive memory allocation).
  - levosmetalo 13y ago
    I agree. I've seen so many times clueless people complain about JVM memory footprint for tiny "benchmarks". In practice, it's never an issue for long running web applications. That, and people that don't understand how JIT works and why it has start-up penalty.
    - chetanahuja 13y ago
      In practice, it's never an issue for long running web applications. You might want to clarify that statement a bit. It almost sounds like you are implying that memory pressure is never an issue for long running web applications in java. Did you mean to say something else?
      - levosmetalo 13y ago
        The initial amount of memory required by VM, while being significant for small command line utilities is only a fraction of total memory required by an application. In a web application tuned for performance a lot of the memory will be used for caching anyways. Also, I'd be glad to pay a small memory hit upfront if that means that I will get a top quality GC and very low probability of memory leaks in the long run. As for the startup time, MojoJolo summed it up.
        chetanahuja 13y ago
        I note that the memory pressure question got swept under the rug there :-) That's ok. It's not a revelation to anybody here (I hope) the enormous cost in memory overhead you have to pay for acceptable performance from the JVM. That "top quality GC" basically requires a 2X overhead (on top of your actual cache payload) to perform with reliable low latency and high throughput.
        fauigerzigerk 13y ago
        I agree completely. And in spite of that "top quality" GC and all the tuning in the world you're still running the risk of having the world stop on you for tens of seconds on larger systems. The JVM (at least OpenJDK, probably not Azul) is quickly becoming untenable as server memory goes into the hundereds of GBs. I'm reluctantly moving back to C++ for that reason alone.
        virmundi 13y ago
        How do you get around heap fragmentation? I know that the JVM (Oracle I believe) is really limited to about 32 GB of RAM before it has real issues. But the nice thing is that the GC will compact the heap for better future performance. As a possible work around to the JVM limit, a distributed architecture with N JVMs running a portion of the task could solve the small memory space with minimal system overhead. What I mean by this let's say you need to have 64 GB of memory for your app. Given the comment above, Java would not do well with this. But you could have 4 16 GB VMs each handling 1/4 of the work. The GC would prevent fragmentation that you'd see in long running C++ apps and still provide you with operational capacity.
    - MojoJolo 13y ago
      @chetanahuja, I think he means that long running web apps was just initialized once and will continue running until they are stopped or restarted. So even though starting up JVM may takes time, that overhead was only done once.
  - wyuenho 13y ago
    It doesn't matter whether they are used or not by the JVM, to the OS, they are allocated and gone. Startup time is a moot point in long running apps.
    - jongraehl 13y ago
      Allocated memory that's paged out and not used is nearly irrelevant (I believe this is true even though a 200mb JVM heap will likely see some unnecessary rotation in/out due to GC - ideally you'd manually configure the heap appropriately (smaller) if you intended to challenge total system memory w/ many processes).
- trailfox 13y ago
  With Go using 20 times less memory, I'd take Go any day since I can just throw in 20 more processes and up my throughput 5-10x. But of course, scaling like this takes a bit of design and the processes have to share nothing. That doesn't even make sense. The Scala version can simply spin up more threads. It's not an issue of parallelism, the Scala version just happens to be faster, throwing more Go processes at the problem won't help. Go is already multithreaded. You don't need multiple processes to use the cores on the box. The JVM scales memory usage by default (heap size etc.) based on the amount of memory on the host. If you're going to worry about 100 MB RAM it's easy to constrain the JVM with -Xmx
  - deleted 13y ago
    [deleted]
vidarh 13y ago
If the JVM and/or the Scala runtime does user-space buffering and Go forwards straight to the read/write syscalls, that alone would probably sufficient to explain the difference when reading/writing buffers this small. If you don't do buffering in user space, the context switches in/out of the kernel will kill you when you do many small read or writes. No idea if that's it, but that's the first place I tend to look when troubleshooting networking app performance as so many people just blindly use syscalls as if they're free.
- jongraehl 13y ago
  The better performance of the Scala client+server if anything suggests less buffering, not more, since the next ping can't be written until the previous pong is received. p.s. to parent: "the JVM TCP stack?" really?
  - vidarh 13y ago
    I admit I haven't checked the example thoroughly - if it goes in lockstep then buffering won't be the culprit. But you're wrong that better performance implies less buffering. A typical way to write such applications to do less buffering is to do select() or poll() or equivalent followed by a large non-blocking read, and then picking your data out of that buffer. As pointed out above, if this "benchmark" does ping/pong's in lockstep across a single connection, then buffering vs no-buffering will make exactly no difference, as there's no additional data available to read. But in scenarios where the amount of data is larger, the time saved from fewer context switches quickly adds up and gives you far more time to actually process the data. Usually your throughput will increase, but your latency will also tend to drop despite the buffering as long as the buffers are of a reasonable size. Buffering is a problem when the buffers grows to large multiples of the typical transaction size, but for typical application level protocols that takes really huge buffers.
    - jongraehl 13y ago
      My comment was specific to this benchmark, which has 4 byte messages that cannot be buffered due to the ping (wait for pong) ping ... repetition in each client. Of course buffering matters for full-pipeline throughput.

damian2000 13y ago

Isn't this more like a comparison of the JVM's TCP library vs Go's TCP library... not so much Scala vs Go?

est 13y ago
They are all wrappers around the OS TCP/IP stack.

willvarfar 13y ago

The benefit of go is the cheap threads and CSP which make it scale well for complex servers. I think we'll see the Go runtime gaining Single System Image distribution, performance improvements and libraries not services (e.g. groupcache) making it a very different world to develop in than Scala.

octo_t 13y ago
Scala has Actors, which are as pure CSP as you can get (Go's CSP has shared mutable state, Actors do not).

dschiptsov 13y ago

... that the Go server had a memory footprint of only about 10 MB vs. Scala’s nearly 200 MB. Priceless! 200Mb for such a crappy server with only hundred connections. So, it is not about whose wrapper around epoll is thinner, but about how the data are represented and copied.

ratherbefuddled 13y ago
You can't really come to a conclusion since no information is given about the configuration of the JVM. Most likely the minimum heap size was unset which often means it will default to 1/64 of the total system memory.
- dschiptsov 13y ago
  I have my conclusions for quite long time.)
coldtea 13y ago
>200Mb for such a crappy server with only hundred connections. Yes, and almost a megabyte of disk space for a "hello world" program. C++ is so inefficient, right? That's not the way you judge the performance of a program and/or it's memory use -- and neither is comparing the 200MB of the JVM with the 10MB of Go here...
- dschiptsov 13y ago
  No, it is almost 2Mb per connection. Why so-called performance of a wrapper around OS syscalls is more important than resource costs of having a pending connection?)

est 13y ago

> The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally, so as to better measure pure processing overhead. The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time. Another let's rape localhost:8080 on a MacBook Pro™ benchmark

aa0 13y ago
Benchmarked? More like benchmucked.
- rkangel 13y ago
  The way I see it, instead of basing a decision on what's currently fashionable, they've gone for some empirical data and have actually conducted an experiment. Sure this is a fairly simplistic first step, but so should all first steps be as you gain understanding, and it's proved its worth by producing a surprising result in a very simple case, that's worth understanding before moving on to more complicated tests. What you've got here is engineering, rather than just making it up as you go along.
  - dmit 13y ago
    The only conclusion that can be reached from this is that a certain implementation of a TCP client/server for sending 4-byte payloads over the OS X loopback interface written in Scala is faster than another implementation written in Go. That conclusion in itself is absolutely useless, because nobody would ever use that particular code to perform that particular task. But it's also dangerous since it generalizes the result of this particular test to imply that one language is better than another at all tasks related to pumping bytes over the wire. And that aside, it still fails as an experiment because the results are neither provable nor repeatable. There is not enough information about the environment used to run the test and there is no detailed test output. We get two numbers ("average round trip" for each implementation), but we can't check that they are correct, and we can't look at the distribution or calculate other, potentially more useful metrics. Even the code provided is different from the code used in the actual test.

madisp 13y ago

"The actual test code contained some functionality to deal with connection errors, omitted here for brevity." Any way to see the actual test code?

anuraj 13y ago

It is the JVM rather than Scala - one of the fastest VMs implemented till date.

aaron42net 13y ago

If the eventual production app would run on Linux (which I'm only guessing based on the context), this benchmark should probably be run there. Darwin's surprisingly higher system call and context switch overhead can be deceptive for apps that are OS-bound.

cmccabe 13y ago

EDIT. OK, so. I ran this benchmark myself on an 8-core Xeon running Linux. 2.13 GHz, CentOS 6.2. Kernel was 2.6.32-220.el6.x86_64. 50 gigs of RAM. I got somewhere between 2.0 and 2.2 "milliseconds per ping" for Scala 2.9, and somewhere between 3.5 and 3.7 for Go 1.1. This is not the 10x difference that the authors reported, but it is something. The difference may be due in part to the different platform and hardware I am using. Contrary to what I wrote earlier, I noticed that GOMAXPROCS=8 did seem to be slower than GOMAXPROCS=4 here. I got around 4 "milliseconds per ping" with GOMAXPROCS=8. Using a mutex and explicit condition variable shaved off maybe 0.2 milliseconds on average (very rough estimation). Again contrary to what I wrote earlier, Nagle on versus off didn't seem to matter in the Go code. I still think you should always have it off for a test like this, but on my setup I did not see a difference. I still don't think this benchmark is showing what they think it is. I have a hunch that this is more of a scheduler benchmark than a TCP benchmark at all. I think I'd have to haul out vtune to get any further, and I'm getting kind of tired (after midnight here).

BarkMore 13y ago
Nagle's algorithm is disabled by default in Go (http://golang.org/pkg/net/#TCPConn.SetNoDelay http://golang.org/pkg/net/#TCPConn.SetNoDelay).
jongraehl 13y ago
> First of all, they're testing on MacOS, which is not going to be the platform they're actually using for the server code. The backends can be very different. Unless you have evidence that Go has MacOS-only bugs, this is meaningless speculation (though I agree that it's possible that there's no problem on Linux and that people don't generally run mac servers, I'm not sure why we should privilege your hypothesis). Agreed about GOMAXPROCS=4 - that seemed questionable to me (I don't know what it does, precisely, but I don't see a 4-anything limit in the Scala code).
- jongraehl 13y ago
  Thanks to parent (after edit) for really testing. It turns out that he was right that Go on Mac is substantially worse than Linux - my bad. Maybe the Go lib authors didn't put much effort into reading the subtly different BSD/Darwin vs Linux syscall semantics. To explain the Nagle algorithm's irrelevance to this case, we have to understand how it works. It doesn't delay any writes once you read. It only affects two consecutive small writes (my memory was fuzzy so I checked http://en.wikipedia.org/wiki/Nagle's_algorithm http://en.wikipedia.org/wiki/Nagle's_algorithm ). Odds of preemption between the client's write and its read seem small, so it shouldn't matter whether you Nagle or not.

dlsspy 13y ago

Did you consider running the go client against the scala server and vice versa? Also, that's kind of a lot of code. Here's my rewrite of the server: http://play.golang.org/p/hKztKKQf7v http://play.golang.org/p/hKztKKQf7v It doesn't return the exact same result, but since you're not verifying the results, it is effectively the same (4 bytes in, 4 bytes back out). I did slightly better with a hand-crafted one. A little cleanup on the client here: http://play.golang.org/p/vRNMzBFOs5 http://play.golang.org/p/vRNMzBFOs5 I'm guessing scala's hiding some magic, though. I made a small change to the way the client is working, buffering reads and writes independently (can be observed later) and I get similar numbers (dropped my local runs from ~12 to .038). This is that version: http://play.golang.org/p/8fR6-y6EBy http://play.golang.org/p/8fR6-y6EBy Now, I don't know scala, but based on the constraints of the program, these actually all do the same thing. They time how long it takes to write 4 bytes * N and read 4 bytes * N. (my version adds error checking). The go version is reporting a bit more latency going in and out of the stack for individual syscalls. I suspect the scala version isn't even making those, as it likely doesn't need to observe the answers. You just get more options in a lower level language.

jongraehl 13y ago
I think you're on the right track in supposing that there can't be a huge performance difference in such a simple task, given that both languages are compiled and reasonably low-level. The most plausible explanation would amount essentially to a misconfigured library, not a fundamental advantage due to say, advanced JVM JIT. Your suggestion to try server-{a,b} x client-{a,b} is also a good one. Your modified Go server doesn't return "Pong" for "Ping". It returns "Ping". And the "a small change" version is nonsense. It's fundamentally different. - you're firing off all your requests before waiting for any replies, and so hiding the latency in the more common RPC style request-response chain, which is a real problem. You speculate a lot ("hiding some magic" "likely doesn't need to observe the answers") when you haven't offered any insight. EDIT: Nagle doesn't matter here - it doesn't delay any writes once you read (waiting server response). It only affects 2+ consecutive small writes (here I'm trusting http://en.wikipedia.org/wiki/Nagle's_algorithm http://en.wikipedia.org/wiki/Nagle's_algorithm - my own recollection was fuzzy). If Go sleeps client threads between the ping and the read-response call then I suppose it would matter (but only a little? and other comments say that Go defaults to no Nagle alg. anyway).

deleted 13y ago

[deleted]

luikore 13y ago

So this Go server is slower than a single threaded, blocking TCP server in Ruby. And the memory? Almost the same: require "socket" s = TCPServer.new 'localhost', 1201 loop do c = s.accept if c.read(4) c << 'Pong' end c.close end

dlsspy 13y ago
It's not obvious to me what that does. Does that only service one of the 100 clients or one of their 10,000 pings each? Or both?
- luikore 13y ago
  It's one at a time. But, since the server only performs a one-shot task without the "hang on and wait 30 seconds"-like long connections, and, the default socket backlog of TcpServer is > 100, so every client gets served within the delay of (0~99)*6ms. In short, every round is fully served, and the concurrency level is 100.
jaekwon 13y ago
hmm, i tried running that against the go client but the client wouldn't connect. dial tcp 127.0.0.1:1201: connection refused i changed 'localhost' to '127.0.0.1' and now I see a whole lot of dial tcp 127.0.0.1:1201: connection reset by peer some connections were successful though.
- luikore 13y ago
  The first one may be your localhost not mapped to 127.0.0.1 ? The second one may be a problem of backlog size. On my system (ruby 2.1.0dev (2013-08-06) [x86_64-darwin12.4.0]) it's 128 and there's no reset. But you can manual change it with s.listen(1000)

smegel 13y ago

Quick look at the language benchmark games, and Go is not 10x slower than Java for most tests. Java is often 3-5x slower than C/C++/etc (although maybe it didn't get enough time to warm up).

trailfox 13y ago
A quick look at the language benchmark game, and Go varies between 7x slower than Java and approx. the same speed as Java (varies by test): http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=java http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t... I fail to see how Go being slower than Java relates to C being faster than Java.
- smegel 13y ago
  Well given both C and Go are native compiled languages, I would think C is a realistic (if distant) goal for Go performance, at least for algorithmic stuff where you're not bouncing in and out of the runtime. I was commenting on how far it has to go.
  - trailfox 13y ago
    Ok, I see. Technically a JIT compiles to native code too, so for a long running app there shouldn't be much difference. Both Go and Java are garbage collected, but the JVM has the more sophisticated GC.

bad_user 13y ago

So even if Scala is ahead in this (flawed) benchmark, that's not how you write a TCP server in Scala, because you want to do it non-blocking. Not doing it based on asynchronous I/O means that in a real-world scenario the server will choke under the weight of slow connections, not to mention be susceptible to really cheap DoS attacks like Slowloris [1]. Seriously, it goes beyond the underlying I/O API that you're using. If anywhere in the code you're reading from an InputStream or you're writing to an OutputStream that's connected to an open socket, then that's a blocking call that can crush your server. Right now, every Java Servlets container that's not compatible with the latest Servlets API 3.1 can be brought down with Slowloris, even if under the hood they are using NIO. Option A for writing a server in Scala is Netty [2]. Option B for writing a server in Scala is the new I/O layer in Akka [3]. [1] http://ha.ckers.org/slowloris/ http://ha.ckers.org/slowloris/ [2] http://netty.io/ http://netty.io/ [3] http://doc.akka.io/docs/akka/snapshot/scala/io.html http://doc.akka.io/docs/akka/snapshot/scala/io.html

caniszczyk 13y ago
Also see Finagle (uses Netty): http://twitter.github.io/finagle/ http://twitter.github.io/finagle/

geal 13y ago

Ok, so. When you need to write a load balancer and want to test different languages for the task, you don't do a benchmark like that one. Writing a "ping-pong"? And not using the same client to test both servers? It would not have been too hard to write a simple proxy in both languages. Not even worrying about parsing HTTP headers, just testing TCP, that is really easy. Now, if you really want to test the performance, you have to implement it differently. Just two small features you would need in both: * non blocking IO: right now, you're starting a new future in Scala, that's easy to write but not really efficient (it might work better with goroutines) * zero copy: if you're load balancing, you will spend your time moving bits, so you'd better make sure that you don't copy them too much. It is possible with Scala, but it looks like Go does not support it Now, when you have reasonable testing grounds (that woudn't be more that a hundred lines in both languages), better get your statistics right. "The client would make 100 concurrent connections and send a total of 1 million pings to the server, evenly distributed over the connections. We measured the average round trip time" -> that is NOT how you should test. Here, you would want to know what is the nominal pressure the balancer could handle, so you must measure a lot of metrics: * RTT * bandwidth (per user and total) * time to warmup (the JVM optimizes a lot of things on the fly, you have to wait for it) * operational limits (what is the maximal bandwidth for which the performance crashes? same for number of users) And then, you don't measure only the average values. You must measure the standard deviation. Because you could have a good average, but wildly varying data, and that is not good. Last thing: the macbook may not be a good testing system.

jaekwon 13y ago

Results from my Mac: Go server vs Go client: 10ms Scala server vs Scala client: 3ms Go server vs Scala client: 4ms Scala server vs Go client: ???? Scala server against the Go client is really slow (?). I reduced the ping count by a factor of 100, and extrapolating I think it would have reported around 670ms. What gives? I don't know much about Scala Futures, but isn't Scala's client doing something completely different than the Go client? Scala's client with Future.sequence looks like it's calling each `ping` method sequentially. Printing the connection identifier {0,100} on open & close shows that while it isn't completely sequential, only about a handful of connections are open at a time. On the other hand, the Go client appears to switch amongst goroutines more frequently. All the connections open before any connection closes. In other words, the I think the difference in performance is due to the difference in how randomly the connections are shuffled. The terrible performance time in the last case I think shows a bottleneck in the Scala server rather than the Go client.

corresation 13y ago

As a word of advice -- you are almost certainly wasting your time writing a load balancer. There are close to zero cases where someone can legitimately justify such an exercise. In any case, your benchmark is flawed (as virtually all benchmarks are). The reason Go the client is slower is because it tries to pervasively "thread" via the M:N scheduler -- every wait check causes it to yield the actual thread and switch goroutine, creating a large amount of overhead. The Scala cases, on the other hand, is dramatically more limited and will not yield this overhead. The Go server does not have this fault, and is likely top performance. And aren't we talking about a server anyways? Now as to the client, while we could naively criticize M:N scheduling based upon this, try giving it a more realistic workload (unless you seriously plan on load balancing pongs): Instead of ping/pong, return larger lengths of data preferably over actual network connections (not localhost) - e.g. 32KB. The Go client will catch up if not shoot into the lead. M:N scheduling is optimal for most real-world workloads, though it is less optimal for spin-off-a-million-goroutines that do nothing type tests. This is not a test of TCP overhead, or a realistic test, but instead demonstrates the small overhead of goroutines when you give each a minuscule amount of wait work.

diakritikal 13y ago
You're right about the Go client, I'm also scratching my head as to why it's doing the extra lookup instead of just using a dial function directly: tcpAddr, _ := net.ResolveTCPAddr("tcp4", "localhost:1201") conn, _ := net.DialTCP("tcp", nil, tcpAddr) Vs. val socket = new Socket("localhost", 1201) Edit: Go has e.g. net.Dial("tcp", "localhost:1201") and as someone pointed out elsewhere for a more accurate bench in both clients why not use the numerical address instead?

Oculus 13y ago

What would be the advantages of using a custom built load balancer vs. something off the shelf like Nginx?

voidlogic 13y ago

>The experiments where performed on a 2.7Ghz quad core MacBook Pro with both client and server running locally No no no. Assuming your production code runs on Linux THAT is where you need to do this test. It is extremely naive to assume that either the JVM or the Go runtime will perform system interfacing tasks even remotely similar between OSX and Linux. Linux is what you will use in production. Linux is almost always faster (more effort from devs both on the kernel TCP/IP side and the runtime/userspace side). Write your Scala, Java, Go wherever you want, but please, benchmark it in a clone of your production environment! P.S. In production I assume your client and server will not be local... don't do this, kernels do awesome/dirty optimizations over loop-back interfaces, sometimes even bypassing large parts of the TCP/IP stack, parts you want included in any meaningful benchmark.

msie 13y ago

Ugh, after reading all the comments here I wonder how a mere-mortal programmer gets multi-threaded, network programming done right. It's not clear to me if there is a clear winner between Go and Scala/JVM. Are the majority of programs out there crappy, memory-hogging and non-performant? EDIT: Any good references out there? Thanks!