7 ms·

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my e

> having half your product in Ruby and half in C is awful to have to deal with. Why? In many cases, only a tiny fraction of your code needs to be fast. In my experience, it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby.

danking00 14y ago
Seconded. I recently wrote a image searching program (find a set of images as sub-images in another set of images) and I wrote most of it in Racket (Scheme dialect). I wrote the inner loop to calculate the correlation matrix in C and made an FFI call. I'm not familiar with Ruby's FFI, but I found making FFI calls dead simple in Racket.
- pak 14y ago
  FFI is also dead simple in Ruby. https://github.com/ffi/ffi https://github.com/ffi/ffi It's pretty much `gem install ffi` and off you go. I've used a Ruby gem [1] that talks to a hashtable storage library via FFI and found it perfectly stable. [1] https://github.com/jmettraux/rufus-tokyo https://github.com/jmettraux/rufus-tokyo
pak 14y ago
> it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby. Agreed. Contrary to what lbrandy said, I've tried it, and it works very well. Writing Ruby extensions in C, or wrapping C libraries for Ruby, is really easy. Wrapping a PNG library for Ruby allowed us to draw about a trillion pixels' worth of images based on the human genome across a modest cluster of ~100 cores in a few days [1,2,3]. For reference, a terapixel image is about ~4TB of uncompressed data and ~800GB compressed, with Microsoft putting up comparable numbers for a project they ran on a fancier HPC cluster in 2010 [4]. We were heavily compressing it into PNG tiles on the fly so ours came down to ~60GB. Regarding pointers to pointers to ints--if you keep all the data within the heap of the Ruby extension, as we did with the wrapped PNG library, then it can be stored in whatever simple data structure you want. [1] http://chromozoom.org/?db=hg19 http://chromozoom.org/?db=hg19 [2] https://github.com/rothlab/chromozoom https://github.com/rothlab/chromozoom [3] http://bioinformatics.oxfordjournals.org/content/29/3/384.full#sec-3 http://bioinformatics.oxfordjournals.org/content/29/3/384.fu... [4] http://research.microsoft.com/en-us/projects/terapixel/ http://research.microsoft.com/en-us/projects/terapixel/
- Scaevolus 14y ago
  That's a neat project! When generating huge numbers of png images, it's worth it to tweak the compression settings. Run png optimizers like optipng/pngcrush on your output, and use the settings it finds. LodePNG is easy to use, but doesn't let you tweak as many options as libpng. The tiles I've inspected are 15-30% larger than necessary-- besides using sub-optimal compression heuristics, they include unnecessary metadata (creation/modification time, aspect ratio).
  - pak 14y ago
    Yeah, I did consider optipng/pngcrush. In the end it was a tradeoff between running time and space. I could afford the space but not the time. When I added in pngcrush, because it brute forces a whole bunch of compression routines on each tile, running times went up dramatically and I preferred having the job complete in days as opposed to weeks. It might be something to consider if we generate more tiles on a faster cluster.
jarrett 14y ago
Indeed. And there is more than one way to do it. The most obvious thing is to write a Ruby C extension. But you could also write a totally separate worker process in the language of your choice and glue it to your main Ruby app with a message queue. In most real-world cases, either solution will get you the performance you need to avoid 10xing your servers.
lbrandy 14y ago
Everyone always agrees with this and, in my experience, has never really tried it. It works better in theory than in practice (or, at a minimum, it works only on a small subset of the types of problems that a naive view might otherwise lead you to believe). The problem is the data isn't organized in a way to be computation efficient, and all the C code in the world isn't going to make chasing pointers to pointers to pointers to ints fast. A ton of work has to go into computation-friendly data organization (see all of numpy, scipy, etc) before you can ever actually optimize effectively. EDIT: since all of my responses are of the form: "oh no I did it once with great results", I'll retract the part where I said "have never really tried it". I'm not arguing it's never worked. I'm arguing that the set of likely performance problems and the set of easy-to-optimize-with-C problems don't overlap as much as people would like to believe.
- jeltz 14y ago
  I have tried it and it was both trivial to do and gave large performance gains. The ruby implementation does not add that many levels of indirection.
- jonhohle 14y ago
  I agree with it and have implemented it. After profiling a backfill script which was looking like it was going to take about two weeks to run, I found an area where certain bit-twiddling was being done which was extremely slow in Ruby. I rewrote this section in C, and was able to make this section 1000x faster then the ruby equivalent (this was MRI 1.8.5 several years ago). The total runtime for the script was brought down to about 8 hours (from about 165 TPS to around 7,000 TPS). Fortunately, I've found that Ruby makes it exceptionally easy to write C extensions, and I often think of reusing C libraries in cases where I'm worried about performance.
  - lbrandy 14y ago
    I didn't mean to imply it wasn't ever possible. Just that the subset of things that it works great for is smaller than people wish, or believed. That "I'll write in Ruby and optimize in C when I need to" is a myth on the order of the "sufficiently smart compiler". There are things for which it just plain doesn't work. Things that are quite common. Yes, you can make image transformations for your webapp fast by piping in a C-extension, because the C code can control the i/o and data organization as well as the algorithms. The integration surface is tiny. You can't, however, make the language's memory usage and garbage collection faster. At least not as easily. You won't be "dropping into C" to fix your garbage collection problems, at least not in the way people dream of when they think of "performance problems" at the outset.
frehpt 14y ago
Option A: -Write your application in Go Option B: -Write application in Ruby -Profile Ruby application to identify code to be rewritten in C -Learn C if you don't already know it -Rewrite various parts of your application in C -Run Valgrind to search for memleaks -Ensure that your target systems all have required libs installed Option A seems much simpler...
- threeseed 14y ago
  Go does't have 1/100000 the amount of libraries and support that Ruby or C does. Option A only seems simpler because you ignore the time/complexity in coding the task at hand.
  - jthol 14y ago
    ... and the Java programmer laughs in the corner (Let's not talk about generics though).
    - vidarh 14y ago
      ... of the asylum he's been committed to after having to deal with one too many AbstractFactoryFactory classes.
      - mitchi 14y ago
        That's Design Patterns code. I'm sure that if I start programming seriously in Java, I can bring all the readability of C with me!
  - kgabis 14y ago
    Why not 1/10e100? Or even less? Go's library support is quite nice already and it's growing really fast.
    - threeseed 14y ago
      Because I am being realistic. Ruby, C, Java etc have had decades to build a comprehensive set of developer libraries. Go hasn't. It's a "cute" platform that I am sure will beat Ruby in a few benchmarks. But I don't see any reason why anyone would seriously consider it for a real world app.
up_and_up 14y ago
A really simple solution in that case is to run JRuby and then write computationally intensive code in a JVM compatible lang like Scala or Clojure. This route seems popular since you can easily give JRuby access to your Scala classes.
- pkolaczk 14y ago
  Right, but if already writing in Scala, why ever write some parts of code in Ruby? Scala gets you expressivity of Ruby with additional benefit of static typesafety.
  - up_and_up 14y ago
    Well a pretty typical scenario is you are writing a ruby app and run into an area that needs additional horsepower (say a gaming API or something). Being on JRuby would give you more options at that point.
papsosouid 14y ago
>Why Because it is a huge pain in the ass, and very rarely does any good. As soon as you are writing C, you are dealing with all the headaches that go with it for both development and deployment. You've lost a big part of the benefit of using scripting language X. And this part: >In many cases, only a tiny fraction of your code needs to be fast Isn't true. People repeat it a lot, but I've seen no evidence to support it. Most people are using scripting languages for web apps which really are just generally slow all over. They would need to write 80% of it in C to get good performance. Seems like using a language that is as productive as scripting languages, but is also fast, makes a lot more sense.

danking00 14y ago

Seconded. I recently wrote a image searching program (find a set of images as sub-images in another set of images) and I wrote most of it in Racket (Scheme dialect). I wrote the inner loop to calculate the correlation matrix in C and made an FFI call. I'm not familiar with Ruby's FFI, but I found making FFI calls dead simple in Racket.

pak 14y ago
FFI is also dead simple in Ruby. https://github.com/ffi/ffi https://github.com/ffi/ffi It's pretty much `gem install ffi` and off you go. I've used a Ruby gem [1] that talks to a hashtable storage library via FFI and found it perfectly stable. [1] https://github.com/jmettraux/rufus-tokyo https://github.com/jmettraux/rufus-tokyo

pak 14y ago

> it is very reasonable to code that part in some fast language while delegating the bigger (and often more complex) part in a higher level language such as Ruby. Agreed. Contrary to what lbrandy said, I've tried it, and it works very well. Writing Ruby extensions in C, or wrapping C libraries for Ruby, is really easy. Wrapping a PNG library for Ruby allowed us to draw about a trillion pixels' worth of images based on the human genome across a modest cluster of ~100 cores in a few days [1,2,3]. For reference, a terapixel image is about ~4TB of uncompressed data and ~800GB compressed, with Microsoft putting up comparable numbers for a project they ran on a fancier HPC cluster in 2010 [4]. We were heavily compressing it into PNG tiles on the fly so ours came down to ~60GB. Regarding pointers to pointers to ints--if you keep all the data within the heap of the Ruby extension, as we did with the wrapped PNG library, then it can be stored in whatever simple data structure you want. [1] http://chromozoom.org/?db=hg19 http://chromozoom.org/?db=hg19 [2] https://github.com/rothlab/chromozoom https://github.com/rothlab/chromozoom [3] http://bioinformatics.oxfordjournals.org/content/29/3/384.full#sec-3 http://bioinformatics.oxfordjournals.org/content/29/3/384.fu... [4] http://research.microsoft.com/en-us/projects/terapixel/ http://research.microsoft.com/en-us/projects/terapixel/

Scaevolus 14y ago
That's a neat project! When generating huge numbers of png images, it's worth it to tweak the compression settings. Run png optimizers like optipng/pngcrush on your output, and use the settings it finds. LodePNG is easy to use, but doesn't let you tweak as many options as libpng. The tiles I've inspected are 15-30% larger than necessary-- besides using sub-optimal compression heuristics, they include unnecessary metadata (creation/modification time, aspect ratio).
- pak 14y ago
  Yeah, I did consider optipng/pngcrush. In the end it was a tradeoff between running time and space. I could afford the space but not the time. When I added in pngcrush, because it brute forces a whole bunch of compression routines on each tile, running times went up dramatically and I preferred having the job complete in days as opposed to weeks. It might be something to consider if we generate more tiles on a faster cluster.

jarrett 14y ago

Indeed. And there is more than one way to do it. The most obvious thing is to write a Ruby C extension. But you could also write a totally separate worker process in the language of your choice and glue it to your main Ruby app with a message queue. In most real-world cases, either solution will get you the performance you need to avoid 10xing your servers.

lbrandy 14y ago

Everyone always agrees with this and, in my experience, has never really tried it. It works better in theory than in practice (or, at a minimum, it works only on a small subset of the types of problems that a naive view might otherwise lead you to believe). The problem is the data isn't organized in a way to be computation efficient, and all the C code in the world isn't going to make chasing pointers to pointers to pointers to ints fast. A ton of work has to go into computation-friendly data organization (see all of numpy, scipy, etc) before you can ever actually optimize effectively. EDIT: since all of my responses are of the form: "oh no I did it once with great results", I'll retract the part where I said "have never really tried it". I'm not arguing it's never worked. I'm arguing that the set of likely performance problems and the set of easy-to-optimize-with-C problems don't overlap as much as people would like to believe.

jeltz 14y ago
I have tried it and it was both trivial to do and gave large performance gains. The ruby implementation does not add that many levels of indirection.
jonhohle 14y ago
I agree with it and have implemented it. After profiling a backfill script which was looking like it was going to take about two weeks to run, I found an area where certain bit-twiddling was being done which was extremely slow in Ruby. I rewrote this section in C, and was able to make this section 1000x faster then the ruby equivalent (this was MRI 1.8.5 several years ago). The total runtime for the script was brought down to about 8 hours (from about 165 TPS to around 7,000 TPS). Fortunately, I've found that Ruby makes it exceptionally easy to write C extensions, and I often think of reusing C libraries in cases where I'm worried about performance.
- lbrandy 14y ago
  I didn't mean to imply it wasn't ever possible. Just that the subset of things that it works great for is smaller than people wish, or believed. That "I'll write in Ruby and optimize in C when I need to" is a myth on the order of the "sufficiently smart compiler". There are things for which it just plain doesn't work. Things that are quite common. Yes, you can make image transformations for your webapp fast by piping in a C-extension, because the C code can control the i/o and data organization as well as the algorithms. The integration surface is tiny. You can't, however, make the language's memory usage and garbage collection faster. At least not as easily. You won't be "dropping into C" to fix your garbage collection problems, at least not in the way people dream of when they think of "performance problems" at the outset.

frehpt 14y ago

Option A: -Write your application in Go Option B: -Write application in Ruby -Profile Ruby application to identify code to be rewritten in C -Learn C if you don't already know it -Rewrite various parts of your application in C -Run Valgrind to search for memleaks -Ensure that your target systems all have required libs installed Option A seems much simpler...

threeseed 14y ago
Go does't have 1/100000 the amount of libraries and support that Ruby or C does. Option A only seems simpler because you ignore the time/complexity in coding the task at hand.
- jthol 14y ago
  ... and the Java programmer laughs in the corner (Let's not talk about generics though).
  - vidarh 14y ago
    ... of the asylum he's been committed to after having to deal with one too many AbstractFactoryFactory classes.
    - mitchi 14y ago
      That's Design Patterns code. I'm sure that if I start programming seriously in Java, I can bring all the readability of C with me!
- kgabis 14y ago
  Why not 1/10e100? Or even less? Go's library support is quite nice already and it's growing really fast.
  - threeseed 14y ago
    Because I am being realistic. Ruby, C, Java etc have had decades to build a comprehensive set of developer libraries. Go hasn't. It's a "cute" platform that I am sure will beat Ruby in a few benchmarks. But I don't see any reason why anyone would seriously consider it for a real world app.

up_and_up 14y ago

A really simple solution in that case is to run JRuby and then write computationally intensive code in a JVM compatible lang like Scala or Clojure. This route seems popular since you can easily give JRuby access to your Scala classes.

pkolaczk 14y ago
Right, but if already writing in Scala, why ever write some parts of code in Ruby? Scala gets you expressivity of Ruby with additional benefit of static typesafety.
- up_and_up 14y ago
  Well a pretty typical scenario is you are writing a ruby app and run into an area that needs additional horsepower (say a gaming API or something). Being on JRuby would give you more options at that point.

papsosouid 14y ago

>Why Because it is a huge pain in the ass, and very rarely does any good. As soon as you are writing C, you are dealing with all the headaches that go with it for both development and deployment. You've lost a big part of the benefit of using scripting language X. And this part: >In many cases, only a tiny fraction of your code needs to be fast Isn't true. People repeat it a lot, but I've seen no evidence to support it. Most people are using scripting languages for web apps which really are just generally slow all over. They would need to write 80% of it in C to get good performance. Seems like using a language that is as productive as scripting languages, but is also fast, makes a lot more sense.