3 ms·

I've come to the personal conclusion that the issue with compiled languages and machine learning is that the API surface is so big and the operations themselves

I've come to the personal conclusion that the issue with compiled languages and machine learning is that the API surface is so big and the operations themselves are very self-contained. I find that I am not typically writing complex "language-level" logic that would be expressible in any type of language and would benefit from the performance of compilation. I am guessing APIs and quickly testing ideas in totally unrelated colab notebooks. My productivity goes up when I can quickly answer questions like "how does this transpose impact the shape of the output" or "how do I find the mean of this tensor along only these dimensions." Compiled languages, and especially Rust, really require you to go in with a plan. I find most of the machine learning code I write is inspired by rapidly playing with ideas.

anentropic 3y ago
This seems to be explicitly targeted at deploying ML models rather than developing them
- ianbutler 3y ago
  Which makes sense their serving framework has a big python component right now, but most of it is written in rust. Given that one of the lead devs for the serving framework is also working on this, I'd guess they're going to look to remove that python portion.
- galangalalgol 3y ago
  I'm a novice at ml, would the performance benefit here be large compared to wonnx or the onnx bindings?
codeflo 3y ago
I wonder about this. Is ML just plugging together prebuilt blocks because that covers the entire solution space, or are we limiting the solution space to that because that’s the only thing you can realistically do in Python?
- robertlagrant 3y ago
  My sense is it's more the latter, at least when exploring how to solve things, but it's hard to say. What is the only thing you can realistically do in Python?
- brrrrrm 3y ago
  It's worth noting that these pre-built blocks are both convenient and incredibly flexible. Before I started getting my hands dirty, it was hard to really see this. For example, here's a masked gather operation along the innermost dimension of an arbitrary rank tensor A: A[..., idx[mask]] I'd hate to have to express that in serial code and then pray a compiler could lift the correct vectorized version out of it.
  - sanderjd 3y ago
    Sure, but any language can provide that kind of pre-built block in a library. (Indeed, I suspect the library we're discussing has this operation, though I didn't check.) But if you want to implement a new operation that performs well, not every language can do that. In python, people tend to drop down to C or C++ or increasingly Rust to do that. But it seems (to me) like it would be nice to be able to use a single language for both things.
    - brrrrrm 3y ago
      At least with the more common modern ML (LLMs etc) operations, no one drops down to Rust. It's really only good for IO/CPU work, but that's rarely a spot for new operations, just for data loading and high-latency network stuff. People typically use Triton or CUDA (usually the C++ flavor) to implement operations. It'd be cool if Rust had a CUDA dialect.
      - sanderjd 3y ago
        The point is that someone dropped down to implement those operations, at some point. It seems like you can target CUDA with Rust, because this project seems to be doing that?

anentropic 3y ago

This seems to be explicitly targeted at deploying ML models rather than developing them

ianbutler 3y ago
Which makes sense their serving framework has a big python component right now, but most of it is written in rust. Given that one of the lead devs for the serving framework is also working on this, I'd guess they're going to look to remove that python portion.
galangalalgol 3y ago
I'm a novice at ml, would the performance benefit here be large compared to wonnx or the onnx bindings?

codeflo 3y ago

I wonder about this. Is ML just plugging together prebuilt blocks because that covers the entire solution space, or are we limiting the solution space to that because that’s the only thing you can realistically do in Python?

robertlagrant 3y ago
My sense is it's more the latter, at least when exploring how to solve things, but it's hard to say. What is the only thing you can realistically do in Python?
brrrrrm 3y ago
It's worth noting that these pre-built blocks are both convenient and incredibly flexible. Before I started getting my hands dirty, it was hard to really see this. For example, here's a masked gather operation along the innermost dimension of an arbitrary rank tensor A: A[..., idx[mask]] I'd hate to have to express that in serial code and then pray a compiler could lift the correct vectorized version out of it.
- sanderjd 3y ago
  Sure, but any language can provide that kind of pre-built block in a library. (Indeed, I suspect the library we're discussing has this operation, though I didn't check.) But if you want to implement a new operation that performs well, not every language can do that. In python, people tend to drop down to C or C++ or increasingly Rust to do that. But it seems (to me) like it would be nice to be able to use a single language for both things.
  - brrrrrm 3y ago
    At least with the more common modern ML (LLMs etc) operations, no one drops down to Rust. It's really only good for IO/CPU work, but that's rarely a spot for new operations, just for data loading and high-latency network stuff. People typically use Triton or CUDA (usually the C++ flavor) to implement operations. It'd be cool if Rust had a CUDA dialect.
    - sanderjd 3y ago
      The point is that someone dropped down to implement those operations, at some point. It seems like you can target CUDA with Rust, because this project seems to be doing that?