3 ms·

Totally. Another thing is that outside of scientific data applications, ordinary data processing apps at scale have been map-reduce for a long time. That's an

Totally. Another thing is that outside of scientific data applications, ordinary data processing apps at scale have been map-reduce for a long time. That's an inherently functional paradigm that's closer to Python. Having parallelized heavily OO Ruby code across machines by unrolling it into functions that could be sent data payloads... that's not a fun thing to sell in an OO culture. In Python you're kind of already writing code that way, so increasing performance by splitting the data onto different machines is simple.

mumblemumble 6y ago
I don't think that would really be too bad, TBH. Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. All you really have to do to translate map and reduce into an OO paradigm is have the map and reduce operations take objects that implement specific interfaces instead of functions that have a certain signature.
- abernard1 6y ago
  > Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. Yeah, but that required everybody learning a different style of programming that wasn't OO ;-). The problem is for "map", you have to get rid of shared state and the interleaving of state and time. Interleaving state and time is basically the only thing objects do beyond providing a namespace and polymorphism. There's also an inherent 1-ary behavior in most OO code. If you've got a complicated object graph, making something a bulk N-ary operation is non-trivial. A lot of ORMs have this problem when trying to optimize queries. When you're passing a vector of "state" to a function, it's much easier to make things bulk. (Anyone who's had to make legacy code provide a bulk API with an efficient DB insert operation knows what I'm talking about). What you usually get in Ruby is a bunch of code that's like Duck.quack() and a complex set of things below that interface. What you need for data processing is something like quack(ducks[]). The way of parallelizing real-deal objects keeping them objects is with actors, and then you need to worry about distributed state a lot more. Anyhoo, that's a long way of saying that the much-derided procedural or functional style of Python avoids this problem from the get-go. You can just slap the "map" step onto a stream processing system without cutting your graph into pieces that don't share state.
  - pjmlp 6y ago
    Smalltalk collections have map reduce, and it is as OOP as it gets.
    - abernard1 6y ago
      Parallelize smalltalk collections across multiple machines without a distributed coordinator (reduce). Even the pedantic original version of smalltalk OOP has this problem. map-reduce and functions do not require coordination until reduction. map doesn't require a serialized mailbox (like actors), or worrying about ordering in time. Smalltalk requires a consistent state and the location of where to send messages is important. Distributed map is stateless. All versions of OOP concern themselves with state encapsulation, whereas map is state elimination. map is functional.
      - pjmlp 6y ago
        Smalltalk map reduce works across VM processes, there is nothing in CS literature that requires map reduce to be a cluster only algorithm.
cutler 6y ago
Functional paradigm closer to Python? I think you mean procedural as Guido has openly professed his distaste for all things functional to which Python's crippled lambdas bear witness. Ruby, on the other hand, with its procs, blocks and lambdas, supports an elegant functional style even if it is implemented in OO.
- pjmlp 6y ago
  All of which go back to Smalltalk blocks and collections. As for Python, Guido might not like it, but Python has enough features to do FP, in spite of crippled lambdas.
- abernard1 6y ago
  So I'm using a simple, deliberately not correct version of "functional" because a lot of people don't know the difference. I don't like it when people break out their monocle and pipe and complain about this or that being true functional :-). I really do just mean "procedural functions + return by value instead of mutation" here. It is completely normal to have a python script that is just ordinary functions, without state being enclosed in a class and accessed via `self`. This "My First Program" behavior scales extremely well, both from a parallelism perspective, as well from a complexity perspective. OO features can be applied iteratively on top of that. With Ruby--syntax aside--it is just very rare to see people only write functions or private_class_methods. The "Everything is a Class" mantra is very strong, and there's lots of rich behavior in Ruby attached onto what would be considered primitives or basic structs in Python. With data science specifically, this natural way of pushing simple procedures, primitives, structs from Python down into C has lots of performance benefits. Because those functions are "flat", you don't have to traverse the call stack routinely and lose performance there.

mumblemumble 6y ago

I don't think that would really be too bad, TBH. Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. All you really have to do to translate map and reduce into an OO paradigm is have the map and reduce operations take objects that implement specific interfaces instead of functions that have a certain signature.

abernard1 6y ago
> Don't forget that the first project to popularize distributed map-reduce, Hadoop, was a Java project. Yeah, but that required everybody learning a different style of programming that wasn't OO ;-). The problem is for "map", you have to get rid of shared state and the interleaving of state and time. Interleaving state and time is basically the only thing objects do beyond providing a namespace and polymorphism. There's also an inherent 1-ary behavior in most OO code. If you've got a complicated object graph, making something a bulk N-ary operation is non-trivial. A lot of ORMs have this problem when trying to optimize queries. When you're passing a vector of "state" to a function, it's much easier to make things bulk. (Anyone who's had to make legacy code provide a bulk API with an efficient DB insert operation knows what I'm talking about). What you usually get in Ruby is a bunch of code that's like Duck.quack() and a complex set of things below that interface. What you need for data processing is something like quack(ducks[]). The way of parallelizing real-deal objects keeping them objects is with actors, and then you need to worry about distributed state a lot more. Anyhoo, that's a long way of saying that the much-derided procedural or functional style of Python avoids this problem from the get-go. You can just slap the "map" step onto a stream processing system without cutting your graph into pieces that don't share state.
- pjmlp 6y ago
  Smalltalk collections have map reduce, and it is as OOP as it gets.
  - abernard1 6y ago
    Parallelize smalltalk collections across multiple machines without a distributed coordinator (reduce). Even the pedantic original version of smalltalk OOP has this problem. map-reduce and functions do not require coordination until reduction. map doesn't require a serialized mailbox (like actors), or worrying about ordering in time. Smalltalk requires a consistent state and the location of where to send messages is important. Distributed map is stateless. All versions of OOP concern themselves with state encapsulation, whereas map is state elimination. map is functional.
    - pjmlp 6y ago
      Smalltalk map reduce works across VM processes, there is nothing in CS literature that requires map reduce to be a cluster only algorithm.

cutler 6y ago

Functional paradigm closer to Python? I think you mean procedural as Guido has openly professed his distaste for all things functional to which Python's crippled lambdas bear witness. Ruby, on the other hand, with its procs, blocks and lambdas, supports an elegant functional style even if it is implemented in OO.

pjmlp 6y ago
All of which go back to Smalltalk blocks and collections. As for Python, Guido might not like it, but Python has enough features to do FP, in spite of crippled lambdas.
abernard1 6y ago
So I'm using a simple, deliberately not correct version of "functional" because a lot of people don't know the difference. I don't like it when people break out their monocle and pipe and complain about this or that being true functional :-). I really do just mean "procedural functions + return by value instead of mutation" here. It is completely normal to have a python script that is just ordinary functions, without state being enclosed in a class and accessed via `self`. This "My First Program" behavior scales extremely well, both from a parallelism perspective, as well from a complexity perspective. OO features can be applied iteratively on top of that. With Ruby--syntax aside--it is just very rare to see people only write functions or private_class_methods. The "Everything is a Class" mantra is very strong, and there's lots of rich behavior in Ruby attached onto what would be considered primitives or basic structs in Python. With data science specifically, this natural way of pushing simple procedures, primitives, structs from Python down into C has lots of performance benefits. Because those functions are "flat", you don't have to traverse the call stack routinely and lose performance there.