3 ms·

I want to cry... why distributed programming in Python? I mean, I understand that a lot of people want Pandas to run faster, and distributing computation would

I want to cry... why distributed programming in Python? I mean, I understand that a lot of people want Pandas to run faster, and distributing computation would help... but come on! There needs to be a line in the sand that a sane person would not cross. It doesn't matter how much effort you put into making Python distributed, it's not going to work unless you have the power to control the language itself. You know, I just thought of a metaphor for this. Maybe not 100%, but you'll get the idea. Suppose you live in a small country, something close to 10M population. And you want to buy pants. Well, you want to buy them from an offline store, so you don't get all the choices you have by shopping online. You don't care about the price, you can afford to buy one pair of pants for a price of ten. You just want a specific kind. Let's say, you want JNCO jeans. Tough luck. You go to every store that sells pants, but JNCO is just too niche for a small country to make sense for the retailers to import. So, every store that wants to make profits buys the same exact model of jeans that was advertised in the last year fashion catalogue. You can pay a little more or a little less for better or worse material, but all the pants on the offer have the same trashy design. You just cannot stand them. This is what's happening with Python. It's trash. But it's the model from the last year fashion catalogue. So any program you write, any problem you solve must be in Python, or else you lose the competition before you even enter into the contest.

acmiyaguchi 4y ago
The language is not a limiting factor here. Python is an excellent scripting language, and works plenty fine in distributed computation. The Python interface to Spark is a wrapper on the underlying Scala API. You don't lose out on performance when you're building up a lazy chain of computation that's executed by an engine written in a more performant language. Fugue is a layer to abstract out these distributed computation backends, and it looks like a nice programming interface.
- kvnkho 4y ago
  Well said! Python can push down to other languages like Rust and C to speed things up. Python can serve as a great end-user interface.
  - chrisjc 4y ago
    Trying to wrap my head around Fugue and the comment explaining how Python is a good wrapper. Does Fugue take advantage of each sublayer that already uses Arrow?
    - goodwanghan 4y ago
      Yes we do. We try to leverage Arrow as much as we can. And different from many other computing frameworks, Fugue doesn't invent new data types, it just uses Arrow data types.
- crabbone 4y ago
  Python is not excellent in any domain it's used. But, yes, it's a problem with the language, on which I'll comment later. First, present day users of Python need to understand how and why Python came under the spotlight. There was always a standoff between programmers who created unimaginative huge programs full of drool and red-taping, and programmers who wanted larger freedom of expression, less strings attached. The later group was usually the more savvy ones. In a way very similar to how an art student might be spending months studying a model, using a whole bunch of pencils starting from 10B and ending in 10H, various chalks, coal sticks and so on... and would still produce a... "study of a model #80907", which is ugly, anatomically incorrect and just boring. And there's an accomplished artist who can just stick her finger into the chimney, grab some soot, and in a matter of minutes make a great drawing, which will be lively, expressive, you name it. So... Python, and Perl before it were the soot. The junk languages a more experienced programmer would go to just to show those boring Java programmers "how it's done". But, Python programmers who came in the next wave thought that soot is the good tool to learn how to make good drawings. And, today, we have academies full of students trying very hard to draw models with materials and instruments which are very inappropriate for the task. (Unless you know anything about art education today, the example isn't that big of a stretch of what happened in it around 70s-80s.) --- I don't care if Python is a glue code for Scala or C or Rust: it doesn't matter. Python, as a language, is inadequate for dealing with concurrency. It needs to remove a bunch of stuff before it can start adding stuff that can be used to that end. It's a language with a lot of mutation semantics which are hard to interpret / implement correctly (what would that even mean?) in distributed context. It's a language with a lot of implicit stuff going on that is somewhat useful (but is not useful enough) if you want to have a quick and dirty "sketch" quality code, but will be devastating in distributed context. Things like decorators, context managers, imperative loops with break and continue, error handing mechanism, threading -- all of this must go before Python can start to think about becoming a decent language for distributed systems. But, probably more: I would need to research this in much depth to tell for sure if things like method calls would work well for example. It's a waste of time to try today to fit Python into distributed computation. You will either have to put a humongous effort purging a better half of the language (making all of the famed support libraries useless), or you will end up with a defective hodge-podge mess (which is all those famed support libraries are, including those which aim to do distributed programming in Python).

acmiyaguchi 4y ago

The language is not a limiting factor here. Python is an excellent scripting language, and works plenty fine in distributed computation. The Python interface to Spark is a wrapper on the underlying Scala API. You don't lose out on performance when you're building up a lazy chain of computation that's executed by an engine written in a more performant language. Fugue is a layer to abstract out these distributed computation backends, and it looks like a nice programming interface.

kvnkho 4y ago
Well said! Python can push down to other languages like Rust and C to speed things up. Python can serve as a great end-user interface.
- chrisjc 4y ago
  Trying to wrap my head around Fugue and the comment explaining how Python is a good wrapper. Does Fugue take advantage of each sublayer that already uses Arrow?
  - goodwanghan 4y ago
    Yes we do. We try to leverage Arrow as much as we can. And different from many other computing frameworks, Fugue doesn't invent new data types, it just uses Arrow data types.
crabbone 4y ago
Python is not excellent in any domain it's used. But, yes, it's a problem with the language, on which I'll comment later. First, present day users of Python need to understand how and why Python came under the spotlight. There was always a standoff between programmers who created unimaginative huge programs full of drool and red-taping, and programmers who wanted larger freedom of expression, less strings attached. The later group was usually the more savvy ones. In a way very similar to how an art student might be spending months studying a model, using a whole bunch of pencils starting from 10B and ending in 10H, various chalks, coal sticks and so on... and would still produce a... "study of a model #80907", which is ugly, anatomically incorrect and just boring. And there's an accomplished artist who can just stick her finger into the chimney, grab some soot, and in a matter of minutes make a great drawing, which will be lively, expressive, you name it. So... Python, and Perl before it were the soot. The junk languages a more experienced programmer would go to just to show those boring Java programmers "how it's done". But, Python programmers who came in the next wave thought that soot is the good tool to learn how to make good drawings. And, today, we have academies full of students trying very hard to draw models with materials and instruments which are very inappropriate for the task. (Unless you know anything about art education today, the example isn't that big of a stretch of what happened in it around 70s-80s.) --- I don't care if Python is a glue code for Scala or C or Rust: it doesn't matter. Python, as a language, is inadequate for dealing with concurrency. It needs to remove a bunch of stuff before it can start adding stuff that can be used to that end. It's a language with a lot of mutation semantics which are hard to interpret / implement correctly (what would that even mean?) in distributed context. It's a language with a lot of implicit stuff going on that is somewhat useful (but is not useful enough) if you want to have a quick and dirty "sketch" quality code, but will be devastating in distributed context. Things like decorators, context managers, imperative loops with break and continue, error handing mechanism, threading -- all of this must go before Python can start to think about becoming a decent language for distributed systems. But, probably more: I would need to research this in much depth to tell for sure if things like method calls would work well for example. It's a waste of time to try today to fit Python into distributed computation. You will either have to put a humongous effort purging a better half of the language (making all of the famed support libraries useless), or you will end up with a defective hodge-podge mess (which is all those famed support libraries are, including those which aim to do distributed programming in Python).