5 ms·

> What is the thing happening in computing making the largest demands on compute performance? Data science and ML. And what's the lingua franca of data science

by nilkn 6y ago

> What is the thing happening in computing making the largest demands on compute performance? Data science and ML. And what's the lingua franca of data science and ML? Not Java or Go, the managed compiled languages; no, it's Python. While ML might be the most demanding area in terms of computation, it's also a very small part of what developers in general do. My claim/prediction is more based on the notion that everything is becoming more compute-intensive. Where I work, we've probably incurred many, many millions of dollars of expenses (primarily hardware) due to the choice to write a lot of software in scripting languages over the years. We made that choice because C++ was so unwieldy in comparison that we figured we'd end up spending the same amount on C++ but in different ways. If we were starting from scratch today, I doubt we'd use scripting languages like Python for hardly any of our software, because there are choices available now that just didn't exist in the past. I'm not saying there's no place for something like Python, nor am I saying existing Python stuff should be rewritten. I think Python is going to continue being an extremely popular and important language. My claim/prediction is more that in the future we'll increasingly see Python chosen less for completely greenfield projects because there are now options that are high-level, well-designed, and extremely fast, and at least in my experience that has a lot of business value that is going to be harder and harder to ignore over time as it becomes more obvious.

tptacek 6y ago
Candidly, there once was a vocal subculture of developers that would aver as to how much more comfortable they were in C than in Python (or Tcl or Perl), and, I'm told, in the long long ago there were them that said the same thing about assembly language. As an engineering manager, your job, if done well, would be to mollify those developers without conceding any ground to them, because the higher-level language was practically always the better option. People will burn money on compute no matter what language things are built in; they'll ship on Fargate instead of EC2, on-demand instead of reserved instances, RDS instead of running their own Postgres, Postgres instead of just using Sleepycat, anything at all instead of running their own iron, and on and on down the list. (All of this is out the window if you're shipping a browser! Or a hypervisor! Or an OS kernel! Or a new router! Or a AAA game!) Most of the time, the most expensive thing you can burn is an hour of programmer time. And, with your fluency in Rust and other languages, you must see immediately why someone would believe Rust is more time-consuming than, say, Java, let alone Python. Paul Graham got this part, at least, right in "Beating The Averages", when he noted how much more worrisome a competitor would be if their job reqs were for Perl programmer than if they were for C developers. The Perl company is actually getting lots of stuff done.
- nilkn 6y ago
  You have two premises here, one explicit, one implicit: (1) [explicit]: a higher level language is practically always the better choice. (2) [implicit]: Rust is not high level (or at least not compared to Python). Our biggest disagreement is with (2). Rust is actually higher level and more expressive than Python in many ways. Of course, it also exposes some lower level concepts like lifetimes and ownership that Python hides. Your comparison doesn’t apply because there isn’t a simple hierarchy into which Rust and Python fall. Each language is both higher- and lower-level than the other in a number of ways.
  - tptacek 6y ago
    The word "expose" is doing a lot of work there. I've written a lot of Python (thankfully not so much anymore), and Rust is part of my day job. I am struggling to think of a circumstance in which something would be easier to write in Rust than in Python, factoring out "performance" --- just comparing the languages themselves. Maybe, you can provide examples, and we can see if we're working from the same premises.
    - nilkn 6y ago
      > The word "expose" is doing a lot of work there. I do not believe this is a statement that I can reply to in a meaningful way. You definitely believe Rust is much harder or more complicated than I think it is, and that’s basically where we disagree.
      - deleted 6y ago
        [deleted]
    - akiselev 6y ago
      > I am struggling to think of a circumstance in which something would be easier to write in Rust than in Python, factoring out "performance" --- just comparing the languages themselves. Most Python backend projects I've worked on would have been better off written in Rust (had it existed) the same way most of the Javascript projects I've worked on would have been better off written in Typescript. Maybe it's just the projects I've worked on (or their size) but maintenance and refactoring to deal with growth of or changes in business requirements were by far more time consuming than actually writing code. The amount of time lost on onboarding and writing trivial tests that could have been handled by the Rust compiler dwarfs true R&D time, let alone time spent debugging bugs that could have been prevented by encoding business logic into the type system. At my current job I often still pull out Python to prototype logic or Typescript to spin up a quick and dirty intranet app but for the stuff that needs to last, just work, or be maintained into the future, I often find myself yearning to use Rust. The last obstacles are organization talent pool and Rust's ecosystem.

by nilkn 6y ago

tptacek 6y ago

Candidly, there once was a vocal subculture of developers that would aver as to how much more comfortable they were in C than in Python (or Tcl or Perl), and, I'm told, in the long long ago there were them that said the same thing about assembly language. As an engineering manager, your job, if done well, would be to mollify those developers without conceding any ground to them, because the higher-level language was practically always the better option. People will burn money on compute no matter what language things are built in; they'll ship on Fargate instead of EC2, on-demand instead of reserved instances, RDS instead of running their own Postgres, Postgres instead of just using Sleepycat, anything at all instead of running their own iron, and on and on down the list. (All of this is out the window if you're shipping a browser! Or a hypervisor! Or an OS kernel! Or a new router! Or a AAA game!) Most of the time, the most expensive thing you can burn is an hour of programmer time. And, with your fluency in Rust and other languages, you must see immediately why someone would believe Rust is more time-consuming than, say, Java, let alone Python. Paul Graham got this part, at least, right in "Beating The Averages", when he noted how much more worrisome a competitor would be if their job reqs were for Perl programmer than if they were for C developers. The Perl company is actually getting lots of stuff done.

nilkn 6y ago
You have two premises here, one explicit, one implicit: (1) [explicit]: a higher level language is practically always the better choice. (2) [implicit]: Rust is not high level (or at least not compared to Python). Our biggest disagreement is with (2). Rust is actually higher level and more expressive than Python in many ways. Of course, it also exposes some lower level concepts like lifetimes and ownership that Python hides. Your comparison doesn’t apply because there isn’t a simple hierarchy into which Rust and Python fall. Each language is both higher- and lower-level than the other in a number of ways.
- tptacek 6y ago
  The word "expose" is doing a lot of work there. I've written a lot of Python (thankfully not so much anymore), and Rust is part of my day job. I am struggling to think of a circumstance in which something would be easier to write in Rust than in Python, factoring out "performance" --- just comparing the languages themselves. Maybe, you can provide examples, and we can see if we're working from the same premises.
  - nilkn 6y ago
    > The word "expose" is doing a lot of work there. I do not believe this is a statement that I can reply to in a meaningful way. You definitely believe Rust is much harder or more complicated than I think it is, and that’s basically where we disagree.
    - deleted 6y ago
      [deleted]
  - akiselev 6y ago
    > I am struggling to think of a circumstance in which something would be easier to write in Rust than in Python, factoring out "performance" --- just comparing the languages themselves. Most Python backend projects I've worked on would have been better off written in Rust (had it existed) the same way most of the Javascript projects I've worked on would have been better off written in Typescript. Maybe it's just the projects I've worked on (or their size) but maintenance and refactoring to deal with growth of or changes in business requirements were by far more time consuming than actually writing code. The amount of time lost on onboarding and writing trivial tests that could have been handled by the Rust compiler dwarfs true R&D time, let alone time spent debugging bugs that could have been prevented by encoding business logic into the type system. At my current job I often still pull out Python to prototype logic or Typescript to spin up a quick and dirty intranet app but for the stuff that needs to last, just work, or be maintained into the future, I often find myself yearning to use Rust. The last obstacles are organization talent pool and Rust's ecosystem.