4 ms·

> Programs are programs The programs in the shootout are nothing like the programs people typically run and write. For instance, most of the programs I write d

by enneff 14y ago

> Programs are programs The programs in the shootout are nothing like the programs people typically run and write. For instance, most of the programs I write do a lot of I/O. The shootout programs do none. > Startup time doesn't matter at all It certainly does in cluster environments where you regularly spin up new server instances. It's also important for command-line tools.

coldtea 14y ago
>The programs in the shootout are nothing like the programs people typically run and write. For instance, most of the programs I write do a lot of I/O. The shootout programs do none. Which is as well, because you only want to benchmark for CPU bound parts of the program. IO is the same speed for all languages, so it cancels out. >It certainly does in cluster environments where you regularly spin up new server instances. Not really a common use case. >It's also important for command-line tools. I don't think command-line-tools are Go's or Java's niche.
- enneff 14y ago
  > IO is the same speed for all languages, so it cancels out. That's not true, particularly in concurrent systems. Recent improvements to Go's scheduler (that make it more aware of various I/O concerns) have made it 2x faster under some workloads. You don't just optimize the compiler to produce the tightest code, you must also finesse its integration with the rest of the world. > Not really a common use case. If you say so. > I don't think command-line-tools are Go's or Java's niche. There are a lot of Go command-line tools out there. I think Go is a superb language for this area.
  - coldtea 14y ago
    >>Not really a common use case. >If you say so Why the irony? How many cluster deployments you know where you "spin up instances all the time"??? Compared to general server / network application programming done in both Java and Go it's LESS than a drop in the bucket. If you work somewhere that this does not hold, you could also be an outlier.
    - redbad 14y ago
      > Why the irony? How many cluster deployments you know > where you "spin up instances all the time"??? Any cluster environment whose applications abide 12-Factor semantics stands to benefit greatly from low startup times. This describes both SOA environments like Amazon and Netflix, and PaaS environments like Heroku and [what is enabled by] Docker. http://12factor.net http://12factor.net
bad_user 14y ago
> It certainly does in cluster environments where you regularly spin up new server instances. It's also important for command-line tools. This is certainly true, however not because of the startup time per se which is entirely negligible, especially compared to the time it takes to boot the whole freaking OS, but rather because of the warmup time of the JVM ... for high-volume apps, cold JVMs don't provide a good throughput and so you need a smart load-balancer that knows to not route as many requests as it can to cold machines. To alleviate that, when the auto-scaling feature kicks in to add new machines on Amazon's EC2 (using their load balancer called ELB), it ads 3 machines instead of one. Then the auto-scaling feature kicks in again when the latency levels are lower than a threshold for 30 mins, removing machines from the cluster and so it stabilizes itself to the optimal number of machines required. Compared to Go however, I would still go with Java, because you can do a lot more with a Java server. Our app is both CPU bound and I/O bound and is able to process about 4000 requests per second per server in under 30ms per served request on c1.medium instances. And this is for a real world app that needs to do logging and requests to Memcached/DynamoDB and make predictions in real time. In my experience, the only way we could have better bang for the buck would be to build the app in C/C++.
Roboprog 14y ago
It seems odd to me that the language shootout is so "algorithmic", and that there doesn't seem to be a test that works through a lot of string manipulation. Most web apps seem to simply generate a bunch of useless beans and womp through strings in prepared statements and page templates. Benchmark that???
- igouy 14y ago
  ? regex-dna reverse-complement k-nucleotide
igouy 14y ago
> the programs people typically run and write How do you know what programs people typically run and write? Assumption? Guess? Data?
- enneff 14y ago
  A large part of my job is meeting programmers around the world and talking to them about what they do. But you can draw the same conclusion by browsing GitHub.
  - igouy 14y ago
    Thank you for understanding that it's ordinary to be asked for clarification about something we've said. 1) I think the way in which the benchmarks game programs are most unlike a lot of other programs is that these are tiny tiny toy programs. "Toy programs" as-in -- http://books.google.com/books?id=gQ-fSqbLfFoC&lpg=PP1&dq=Computer%20Architecture&pg=PA37#v=onepage&q=toy%20benchmark&f=false http://books.google.com/books?id=gQ-fSqbLfFoC&lpg=PP1&#3... And that's intentional. 2) Why do you say "...I/O. The shootout programs do none."? regex-dna, k-nucleotide, and reverse-complement do read data in FASTA format before processing. (fasta and reverse-complement also write data in FASTA format -- but redirected to /dev/null) Not TBs of I/O but not none.

by enneff 14y ago

coldtea 14y ago

>The programs in the shootout are nothing like the programs people typically run and write. For instance, most of the programs I write do a lot of I/O. The shootout programs do none. Which is as well, because you only want to benchmark for CPU bound parts of the program. IO is the same speed for all languages, so it cancels out. >It certainly does in cluster environments where you regularly spin up new server instances. Not really a common use case. >It's also important for command-line tools. I don't think command-line-tools are Go's or Java's niche.

enneff 14y ago
> IO is the same speed for all languages, so it cancels out. That's not true, particularly in concurrent systems. Recent improvements to Go's scheduler (that make it more aware of various I/O concerns) have made it 2x faster under some workloads. You don't just optimize the compiler to produce the tightest code, you must also finesse its integration with the rest of the world. > Not really a common use case. If you say so. > I don't think command-line-tools are Go's or Java's niche. There are a lot of Go command-line tools out there. I think Go is a superb language for this area.
- coldtea 14y ago
  >>Not really a common use case. >If you say so Why the irony? How many cluster deployments you know where you "spin up instances all the time"??? Compared to general server / network application programming done in both Java and Go it's LESS than a drop in the bucket. If you work somewhere that this does not hold, you could also be an outlier.
  - redbad 14y ago
    > Why the irony? How many cluster deployments you know > where you "spin up instances all the time"??? Any cluster environment whose applications abide 12-Factor semantics stands to benefit greatly from low startup times. This describes both SOA environments like Amazon and Netflix, and PaaS environments like Heroku and [what is enabled by] Docker. http://12factor.net http://12factor.net

bad_user 14y ago

> It certainly does in cluster environments where you regularly spin up new server instances. It's also important for command-line tools. This is certainly true, however not because of the startup time per se which is entirely negligible, especially compared to the time it takes to boot the whole freaking OS, but rather because of the warmup time of the JVM ... for high-volume apps, cold JVMs don't provide a good throughput and so you need a smart load-balancer that knows to not route as many requests as it can to cold machines. To alleviate that, when the auto-scaling feature kicks in to add new machines on Amazon's EC2 (using their load balancer called ELB), it ads 3 machines instead of one. Then the auto-scaling feature kicks in again when the latency levels are lower than a threshold for 30 mins, removing machines from the cluster and so it stabilizes itself to the optimal number of machines required. Compared to Go however, I would still go with Java, because you can do a lot more with a Java server. Our app is both CPU bound and I/O bound and is able to process about 4000 requests per second per server in under 30ms per served request on c1.medium instances. And this is for a real world app that needs to do logging and requests to Memcached/DynamoDB and make predictions in real time. In my experience, the only way we could have better bang for the buck would be to build the app in C/C++.

Roboprog 14y ago

It seems odd to me that the language shootout is so "algorithmic", and that there doesn't seem to be a test that works through a lot of string manipulation. Most web apps seem to simply generate a bunch of useless beans and womp through strings in prepared statements and page templates. Benchmark that???

igouy 14y ago
? regex-dna reverse-complement k-nucleotide

igouy 14y ago

> the programs people typically run and write How do you know what programs people typically run and write? Assumption? Guess? Data?

enneff 14y ago
A large part of my job is meeting programmers around the world and talking to them about what they do. But you can draw the same conclusion by browsing GitHub.
- igouy 14y ago
  Thank you for understanding that it's ordinary to be asked for clarification about something we've said. 1) I think the way in which the benchmarks game programs are most unlike a lot of other programs is that these are tiny tiny toy programs. "Toy programs" as-in -- http://books.google.com/books?id=gQ-fSqbLfFoC&lpg=PP1&dq=Computer%20Architecture&pg=PA37#v=onepage&q=toy%20benchmark&f=false http://books.google.com/books?id=gQ-fSqbLfFoC&lpg=PP1&#3... And that's intentional. 2) Why do you say "...I/O. The shootout programs do none."? regex-dna, k-nucleotide, and reverse-complement do read data in FASTA format before processing. (fasta and reverse-complement also write data in FASTA format -- but redirected to /dev/null) Not TBs of I/O but not none.