5 ms·

> I wish all the "big data" tooling was written in Ruby. On a practical note, Ruby's heavy use of reflection, inheritance and method_missing style dispatch cur

> I wish all the "big data" tooling was written in Ruby. On a practical note, Ruby's heavy use of reflection, inheritance and method_missing style dispatch currently makes performance for big data tasks less than ideal. Python is less expressive but its "one true way" makes things like vectorization and type specialization easier for data tasks. Sometimes you really just need a better Fortran.

lloeki 6y ago
I believe you’re confusing Ruby with Rails. While these can eventually come in handy sometimes, they are by and large seldom used, but Rails heavily leverages such patterns for flashy yet ultimately questionable magic. As a long time Rubyist, I believe Rails, for all its innovation, has been warping the view of what Ruby is, and when and how to use its features responsibly.
- abernard1 6y ago
  I concur. The dynamic inheritance part of Ruby is still core even in a non-Rails ecosystem. That does make optimizing performance more difficult. But on a much simpler level, a lot of Pythonic data code is just functions + values, without them being complected in an object. That makes a lot of stuff like wrapping numerical libraries in C very easy in Python. I think you'd still have a bit of a culture shock writing procedural, return-by-value code in Ruby. That's pretty normal in Python and I think one of the (many) reasons it got picked up by STEM disciplines in academia. Looks over shoulder at awful code in Numerical Recipes in Fortran book. Different strokes for different folks and all that, I just think there are probably constraints (and cultural values) that Python has that make it more suited to data problems than Ruby.
  - mumblemumble 6y ago
    I'm inclined to agree. One of Python's strengths in this space is that it's a procedural language with just enough in the way of object-oriented facilities. That makes it possible to write reasonably object-oriented libraries with easy-to-learn interfaces, while still sticking to procedural idioms for the actual data hacking. Which, in turn, means makes it a comfortable environment and interfacing mechanism for both engineers and analysts. That said, sometimes I look at tools like https://moosetechnology.org https://moosetechnology.org and desperately wish I could be using something like that at work.
    - abernard1 6y ago
      Totally. Another thing is that outside of scientific data applications, ordinary data processing apps at scale have been map-reduce for a long time. That's an inherently functional paradigm that's closer to Python. Having parallelized heavily OO Ruby code across machines by unrolling it into functions that could be sent data payloads... that's not a fun thing to sell in an OO culture. In Python you're kind of already writing code that way, so increasing performance by splitting the data onto different machines is simple.
      - mumblemumble 6y ago
        I don't think that would really be too bad, TBH. Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. All you really have to do to translate map and reduce into an OO paradigm is have the map and reduce operations take objects that implement specific interfaces instead of functions that have a certain signature.
        abernard1 6y ago
        > Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. Yeah, but that required everybody learning a different style of programming that wasn't OO ;-). The problem is for "map", you have to get rid of shared state and the interleaving of state and time. Interleaving state and time is basically the only thing objects do beyond providing a namespace and polymorphism. There's also an inherent 1-ary behavior in most OO code. If you've got a complicated object graph, making something a bulk N-ary operation is non-trivial. A lot of ORMs have this problem when trying to optimize queries. When you're passing a vector of "state" to a function, it's much easier to make things bulk. (Anyone who's had to make legacy code provide a bulk API with an efficient DB insert operation knows what I'm talking about). What you usually get in Ruby is a bunch of code that's like Duck.quack() and a complex set of things below that interface. What you need for data processing is something like quack(ducks[]). The way of parallelizing real-deal objects keeping them objects is with actors, and then you need to worry about distributed state a lot more. Anyhoo, that's a long way of saying that the much-derided procedural or functional style of Python avoids this problem from the get-go. You can just slap the "map" step onto a stream processing system without cutting your graph into pieces that don't share state.
  - cutler 6y ago
    I don't think that's been true for quite a while. Even a function like len('abc') is implemented as 'abc'.__len()__ under the hood so no different from Ruby really.
- lukeschlather 6y ago
  Well, the problem is that Rails (and ActiveSupport) is borderline idiomatic Ruby. I imagine the vast majority of Ruby codebases use them. And honestly I don't even know how vanilla Ruby stacks up to Python, it's almost an inconceivable idea (though I left Ruby behind some years ago.)
  - vinceguidry 6y ago
    Rails is dominating Ruby less and less every year. There are plenty of alternative web frameworks to use, and Rails is a really heavy-weight thing to include if you're not building a website. I'll pull in an ActiveSupport module every once in awhile but so few of the things I write are websites that need Rails.
    - cutler 6y ago
      There's no evidence for this in the job market. Most Ruby jobs are Rails jobs.
- chrisseaton 6y ago
  > While these can eventually come in handy sometimes, they are by and large seldom used I don't think they are seldom used in practice - I think large parts of the Ruby ecosystem use them pervasively, even if you disregard Rails. But even if we don't agree on that - it's missing the point. The point is you pay for much of Ruby's metaprogramming, even if you aren't using it. So not using it doesn't help you. A concrete example is that you pay for `Kernel#binding` every single time you finish using a local variable but Ruby retains it for the binding, even if you never ever call `Kernel#binding`. Even JITs cannot remove this cost.
  - pizza234 6y ago
    > The point is you pay for much of Ruby's metaprogramming, even if you aren't using it. I don't think there has been any meaningful attempt at assessing the performance of Python vs Ruby. If we consider the benchmarks game, their performance is comparable. I assume that PyPy is faster, but whether to consider it representative of Python's general performance is arguable.
  - jashmatthews 6y ago
    Which cost do you mean? Needing to keep the value alive for deoptimization if Kernel#binding is actually called or?
rat87 6y ago
Python isn't less expressive for the most part except for lack of blocks(although can be emulated with decorators and def _), and ruby's strategic lack of parentheses and not being to add/change methods in c based classes. Python has just as much access to reflection, it has more powerful and arguably easier to shoot yourself in the foot inheritance (python has multiple inheritance while ruby only has mixins). Python also has method_missing in __getattr__. Arguably python tends to use these less often then ruby but I've definitely used both reflection and __getattr__. I've also once had to debug a really hairy multiple inheritance issue(in a test framework that used base classes to test different hardware features where you might want to have both you'd need to use multiple inheritance and then make sure they were initialized in the right order, multiple inheritance can get really annoying)
masklinn 6y ago
> On a practical note, Ruby's heavy use of reflection, inheritance and method_missing style dispatch currently makes performance for big data tasks less than ideal. Python is less expressive but its "one true way" makes things like vectorization and type specialization easier for data tasks. FWIW the things you mention are available in Python. There are things Python doesn't have like singleton objects but I'd say those are syntactic sugar & conveniences (not unlike classes existing from the start of the `class` statement) rather than a lower level of expressivity. Python absolutely does have extensive reflection, inheritance, and "method_missing-style dispatch" (cf `__getattribute__` and `__getattr__`), and so has to accommodate the possibility of them being used… unless it doesn't want to which specialised Ruby implementations (à la numba and friends) could certainly do as well.

lloeki 6y ago

I believe you’re confusing Ruby with Rails. While these can eventually come in handy sometimes, they are by and large seldom used, but Rails heavily leverages such patterns for flashy yet ultimately questionable magic. As a long time Rubyist, I believe Rails, for all its innovation, has been warping the view of what Ruby is, and when and how to use its features responsibly.

abernard1 6y ago
I concur. The dynamic inheritance part of Ruby is still core even in a non-Rails ecosystem. That does make optimizing performance more difficult. But on a much simpler level, a lot of Pythonic data code is just functions + values, without them being complected in an object. That makes a lot of stuff like wrapping numerical libraries in C very easy in Python. I think you'd still have a bit of a culture shock writing procedural, return-by-value code in Ruby. That's pretty normal in Python and I think one of the (many) reasons it got picked up by STEM disciplines in academia. Looks over shoulder at awful code in Numerical Recipes in Fortran book. Different strokes for different folks and all that, I just think there are probably constraints (and cultural values) that Python has that make it more suited to data problems than Ruby.
- mumblemumble 6y ago
  I'm inclined to agree. One of Python's strengths in this space is that it's a procedural language with just enough in the way of object-oriented facilities. That makes it possible to write reasonably object-oriented libraries with easy-to-learn interfaces, while still sticking to procedural idioms for the actual data hacking. Which, in turn, means makes it a comfortable environment and interfacing mechanism for both engineers and analysts. That said, sometimes I look at tools like https://moosetechnology.org https://moosetechnology.org and desperately wish I could be using something like that at work.
  - abernard1 6y ago
    Totally. Another thing is that outside of scientific data applications, ordinary data processing apps at scale have been map-reduce for a long time. That's an inherently functional paradigm that's closer to Python. Having parallelized heavily OO Ruby code across machines by unrolling it into functions that could be sent data payloads... that's not a fun thing to sell in an OO culture. In Python you're kind of already writing code that way, so increasing performance by splitting the data onto different machines is simple.
    - mumblemumble 6y ago
      I don't think that would really be too bad, TBH. Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. All you really have to do to translate map and reduce into an OO paradigm is have the map and reduce operations take objects that implement specific interfaces instead of functions that have a certain signature.
      - abernard1 6y ago
        > Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. Yeah, but that required everybody learning a different style of programming that wasn't OO ;-). The problem is for "map", you have to get rid of shared state and the interleaving of state and time. Interleaving state and time is basically the only thing objects do beyond providing a namespace and polymorphism. There's also an inherent 1-ary behavior in most OO code. If you've got a complicated object graph, making something a bulk N-ary operation is non-trivial. A lot of ORMs have this problem when trying to optimize queries. When you're passing a vector of "state" to a function, it's much easier to make things bulk. (Anyone who's had to make legacy code provide a bulk API with an efficient DB insert operation knows what I'm talking about). What you usually get in Ruby is a bunch of code that's like Duck.quack() and a complex set of things below that interface. What you need for data processing is something like quack(ducks[]). The way of parallelizing real-deal objects keeping them objects is with actors, and then you need to worry about distributed state a lot more. Anyhoo, that's a long way of saying that the much-derided procedural or functional style of Python avoids this problem from the get-go. You can just slap the "map" step onto a stream processing system without cutting your graph into pieces that don't share state.
- cutler 6y ago
  I don't think that's been true for quite a while. Even a function like len('abc') is implemented as 'abc'.__len()__ under the hood so no different from Ruby really.
lukeschlather 6y ago
Well, the problem is that Rails (and ActiveSupport) is borderline idiomatic Ruby. I imagine the vast majority of Ruby codebases use them. And honestly I don't even know how vanilla Ruby stacks up to Python, it's almost an inconceivable idea (though I left Ruby behind some years ago.)
- vinceguidry 6y ago
  Rails is dominating Ruby less and less every year. There are plenty of alternative web frameworks to use, and Rails is a really heavy-weight thing to include if you're not building a website. I'll pull in an ActiveSupport module every once in awhile but so few of the things I write are websites that need Rails.
  - cutler 6y ago
    There's no evidence for this in the job market. Most Ruby jobs are Rails jobs.
chrisseaton 6y ago
> While these can eventually come in handy sometimes, they are by and large seldom used I don't think they are seldom used in practice - I think large parts of the Ruby ecosystem use them pervasively, even if you disregard Rails. But even if we don't agree on that - it's missing the point. The point is you pay for much of Ruby's metaprogramming, even if you aren't using it. So not using it doesn't help you. A concrete example is that you pay for `Kernel#binding` every single time you finish using a local variable but Ruby retains it for the binding, even if you never ever call `Kernel#binding`. Even JITs cannot remove this cost.
- pizza234 6y ago
  > The point is you pay for much of Ruby's metaprogramming, even if you aren't using it. I don't think there has been any meaningful attempt at assessing the performance of Python vs Ruby. If we consider the benchmarks game, their performance is comparable. I assume that PyPy is faster, but whether to consider it representative of Python's general performance is arguable.
- jashmatthews 6y ago
  Which cost do you mean? Needing to keep the value alive for deoptimization if Kernel#binding is actually called or?

rat87 6y ago

Python isn't less expressive for the most part except for lack of blocks(although can be emulated with decorators and def _), and ruby's strategic lack of parentheses and not being to add/change methods in c based classes. Python has just as much access to reflection, it has more powerful and arguably easier to shoot yourself in the foot inheritance (python has multiple inheritance while ruby only has mixins). Python also has method_missing in __getattr__. Arguably python tends to use these less often then ruby but I've definitely used both reflection and __getattr__. I've also once had to debug a really hairy multiple inheritance issue(in a test framework that used base classes to test different hardware features where you might want to have both you'd need to use multiple inheritance and then make sure they were initialized in the right order, multiple inheritance can get really annoying)

masklinn 6y ago

> On a practical note, Ruby's heavy use of reflection, inheritance and method_missing style dispatch currently makes performance for big data tasks less than ideal. Python is less expressive but its "one true way" makes things like vectorization and type specialization easier for data tasks. FWIW the things you mention are available in Python. There are things Python doesn't have like singleton objects but I'd say those are syntactic sugar & conveniences (not unlike classes existing from the start of the `class` statement) rather than a lower level of expressivity. Python absolutely does have extensive reflection, inheritance, and "method_missing-style dispatch" (cf `__getattribute__` and `__getattr__`), and so has to accommodate the possibility of them being used… unless it doesn't want to which specialised Ruby implementations (à la numba and friends) could certainly do as well.