14 ms·
Candle: Torch Replacement in Rust
- rochak 3y agoWould really wish something like that for Go. Rust is cool and all, but I just can’t be arsed to care about the language more than solving actual business problems.
- tomrod 3y agoOh man. I feel like this is a really big deal. I've been wondering what might move data scientists to Rust. Outside working through rustlings and trying new projects in Rust, I wonder if having this available will be it.
- spenczar5 3y agoI agree. Two years ago, I tried writing data-intensive components for Python as rust extension modules. It was painful, tedious, and confusing. I came back to it for a second try a month ago, and was blown away. PyO3 and maturin (especially maturin) are stunningly straightforward. I actually find maturin easier to use than “normal” Python packaging tools, to the extent such a thing even exists. I really hope we will see a move towards rust, and away from extension modules written in C/C++/Fortran, just because the development experience is so much more approachable. Which is not something I thought I would say about Rust!
- tomrod 3y agoYou've convinced me. I've been waiting to see when Julia or Rust would be able to start encroaching on Python's production lockup. Time to deep dive!
- deleted 3y ago[deleted]
- locuscoeruleus 3y agoHow can I effectively approach extending Python with Rust? I encountered some challenges while attempting to translate Python objects to Rust. My intuition suggests that it's more straightforward to work with arrays or dictionaries of primitive types. Could you share your approach and perhaps point me to some helpful examples or resources for better understanding?
- DannyBee 3y ago"I've been wondering what might move data scientists to Rust" I mean, writ large, the answer is "The availability of the ecosystem they need to use daily in Rust, without having to rewrite it themselves." That is more than just a torch replacement, so this won't do it for data scientists, even if Python is ... painful in a lot of ways to productionize or even use. Maybe it'll get some started on that ecosystem, though, so that a few years from now you see larger movement.
- appplication 3y agoMore than the ecosystem, it’s the speed and flexibility in prototyping python offers. I do not see rust becoming a replacement for that. The uncomfortable fact is that you really don’t need thread safety or static typing to prototype and build models effectively.
- ElectricalUnion 3y agoI guess the answer is when the ecosystem stops being composed of mostly experimentation, training and prototyping (data scientists) and turns into deployment and upkeep (ops) of said systems. Prototyping and training doesn't pay the bills. (unless you're Hugging Face, lol)
- tayo42 3y agoid imagine repl, junpyter notebooks are pretty useful and not doable in rust atm
- tomrod 3y agoRust in Jupyter Notebooks: https://users.rust-lang.org/t/getting-started-with-rust-jupyter-notebooks/39004 https://users.rust-lang.org/t/getting-started-with-rust-jupy... REPLit for Rust: https://replit.com/languages/rust https://replit.com/languages/rust For general REPL and rust appears there are a few options. I don't think getting rid of Python completely will happen. But current approaches _typically_ looks something like: - Jupyter Notebook for custom parts of EDA - Jupyter Notebook for testing new libraries or learning new methods - Pipeline (replaceable with Rust) for standard EDA - Pipeline (replaceable with Rust) for data quality - Application/pipeline for product development
- tayo42 3y agohmm ill look later, i thought they needed some compiler work done and miri to do it
- AlchemistCamp 3y ago> I've been wondering what might move data scientists to Rust. I think dynamic typing is the missing piece. Rust's safety guarantees are great and I like the language, but the type system is a barrier for learners, especially people who aren't full-time devs. Ease of experimentation and flexibility are key for this use case.
- CharlesW 3y ago> Rust's safety guarantees are great and I like the language, but the type system is a barrier for learners, especially people who aren't full-time devs. This surprises me. Hasn't everyone working with data (even at an "enthusiast" level) had to understand data types in the context of databases?
- rirze 3y agoYou underestimate the gusto of data engineers/scientists to just assume types.
- spenczar5 3y agoYes, but Rust’s type system is waaaaay more sophisticated than your typical SQL database. Talk about “generic associated types” and you can see people start to shrink back in fear. Plus, a surprisingly enormous fraction of the NoSQL hysteria was that “you don’t even need a schema!”
- josephg 3y ago> Plus, a surprisingly enormous fraction of the NoSQL hysteria was that “you don’t even need a schema!” The problem with this is that you have a schema regardless. In "schemaless" systems, you still have a schema. You just don't have it written down anywhere. And because its not written down, the database doesn't enforce your schema. So its very common for nosql databases to have records with missing fields or subtly corrupted data. That said, there's a reason why nosql became popular. I think a large part of it is that SQL databases are too inflexible. You have to define the entire schema ahead of time, and schema evolution is way too difficult. When I'm writing software, I don't define all of my columns ahead of time. I want to add and change stuff as I go. I wish databases were more flexible. Like, let me just add arbitrary extra fields to my records. Let the database recommend promoting a dynamic field to a static field. ("Hey, all your records have a name: string field. Wanna make that required?"). I love programming languages with similar features. I love typescript for prototyping because I can start with the type any (or wildcard fields in my interfaces - like [k: string]: any). And then I can formalize my types later when I understand my software better. I don't think schemas are a mistake for beginners. I think the opposite - they're amazing guard rails. But I agree with other commenters here. Rust's types include information about ownership, lifetime, containment (Box / Rc), traits and the stored data type all in one go. Its terrifying to beginners and exhausting for experts to use. I've been writing rust for years, and the Future trait still scares me.
- deleted 3y ago[deleted]
- huac 3y agoI sense that most folks will end up using this via a Python wrapper, if at all, much like how HF has Transformers written in Rust but with a Python wrapper. This library is targeting production, a much smaller subset of engineers. See the 'why candle' section: https://github.com/huggingface/candle#why-candle https://github.com/huggingface/candle#why-candle. So you could use PyTorch to build models and Candle to serve them.
- freilanzer 3y ago> I've been wondering what might move data scientists to Rust. If they didn't migrate to Julia they certainly won't migrate to Rust.
- tomrod 3y agoAs someone who has made that choice in this context, Rust is low-level but an increasing part of the stack, and Julia always had weird edge cases for me.
- alex_lav 3y ago> I've been wondering what might move data scientists to Rust. The two things that keep Python as the incumbent in this space is A.) It gets out of the way, sometimes to a fault, such that you can kinda write it without knowing what you're doing. This is a feature for most people, not a bug. And B.) All the math/science libraries that already exist for it. You can solve B, but until you solve A, Python stays put I think.
- jononor 3y agoAnd C) it is taught at every University that teaches ML/DS. I think unseating Python as the lingua franca will takes as long as it has taken to unseat MATLAB in engineering. 20+ years... Of course other things will also exist, and all languages will have decent ML support (especially for inference).
- jeroenhd 3y agoI tried to do some data analysis in Rust but I came back to Python quickly. There are Numpy alternatives for most features but they're all scarcely documented, not entirely compatible, or are missing methods and implementations that Numpy readily provides. With intensive Python processing being mostly C anyway, most data processing doesn't really benefit much from a Rust implementation. The actual native code implementation can probably be a lot better if that part switches over to Rust, with all of Rust's safeties and guarantees, but I don't think data science is going to switch to Rust for gluing together their actual calculations any time soon.
- ianpurton 3y agoI think the first Rust ML framework to gain massive traction will be the one that supports quantisation out of the box. At the moment there's no Rust ML library that does what llama.cpp or ggml can do.
- srush 3y agoNowhere near as neat as candle or ggml, but just released a 4-bit rust llama2 implementation with simd. Runs pretty fast. https://github.com/srush/llama2.rs/ https://github.com/srush/llama2.rs/
- 1ba9115454 3y agoThat's really cool.
- unshavedyak 3y agoThat is _really_ cool. Would be interested in some requirements being posted. Eg RAM required for the 70B model if CPU, VRAM required if GPU, etc. edit: I know it's memory mapping, but that still loads data in RAM to execute so if you had 512MiB of RAM it would likely be slow as hell.
- srush 3y agoYeah, it's CPU only, and it is using about 38g for 70B and 7g 7B. Guessing that is mostly from the large caches that it keeps around for efficiency. If you wanted to pay some computational cost, you could likely get that down by quantizing activations. Llama2 has some of these tricks built in automatically, for instance grouped query attention.
- FL33TW00D 3y agoThey already support quantization out of the box! https://github.com/huggingface/candle/pull/314/files https://github.com/huggingface/candle/pull/314/files
- microtonal 3y agoI am so happy about them releasing this. A few years ago I wrote a multi-task syntax annotator in Rust using Laurent Mazare's excellent Torch binding (tch-rs, it looks like he is also working on Candle): https://github.com/tensordot/syntaxdot https://github.com/tensordot/syntaxdot However, the deployment story was always quite difficult. The PyTorch C++ API is not stable, so a particular version of tch-rs will only work with a particular PyTorch version. So, anyone wanting to use SyntaxDot always had to get exactly the right version of libtorch (and set some environment variables) to build the project. The idea of making an abstraction over Torch and Rust ndarray (similar to Burn) crossed my mind several times, but there is only so much that I could do as a solo developer. So Candle would be a god-given if I was still working on this project. Seeing Candle wants to make me port curated-transformers to Candle for fun: https://github.com/explosion/curated-transformers https://github.com/explosion/curated-transformers
- adamnemecek 3y agothis couldn’t happen soon enough.
- brrrrrm 3y agoI've come to the personal conclusion that the issue with compiled languages and machine learning is that the API surface is so big and the operations themselves are very self-contained. I find that I am not typically writing complex "language-level" logic that would be expressible in any type of language and would benefit from the performance of compilation. I am guessing APIs and quickly testing ideas in totally unrelated colab notebooks. My productivity goes up when I can quickly answer questions like "how does this transpose impact the shape of the output" or "how do I find the mean of this tensor along only these dimensions." Compiled languages, and especially Rust, really require you to go in with a plan. I find most of the machine learning code I write is inspired by rapidly playing with ideas.
- anentropic 3y agoThis seems to be explicitly targeted at deploying ML models rather than developing them
- ianbutler 3y agoWhich makes sense their serving framework has a big python component right now, but most of it is written in rust. Given that one of the lead devs for the serving framework is also working on this, I'd guess they're going to look to remove that python portion.
- galangalalgol 3y agoI'm a novice at ml, would the performance benefit here be large compared to wonnx or the onnx bindings?
- codeflo 3y agoI wonder about this. Is ML just plugging together prebuilt blocks because that covers the entire solution space, or are we limiting the solution space to that because that’s the only thing you can realistically do in Python?
- robertlagrant 3y ago
- rdedev 3y agoLove the shoutut to dfdx. Haven't tried that crate yet but type checked matrix shapes sounds pretty good
- jstx1 3y agoIt's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rustling exercises, read Zero to Production in Rust and did the project there, and still it's not clicking - writing Rust is a slog where you have to jump through hoops and spend a lot more time thinking about programming language abstractions than thinking about the problem you're solving. So I'm not in a rush to rewrite anything written in Python just yet.
- lionkor 3y agoThis has been my experience, too. Somewhere in Rust, there is a beautiful, functional(ish), pragmatic and simple language, hiding in between all the crappy things. So it's just like C++ ;) except it has pattern matching and const by default.
- taminka 3y agowhat being a kitchen sink language does to a mf
- wokwokwok 3y agoYeah, but the goal of this isn’t “use this instead of Python to hack up a model” …it’s: > And simply removing Python from production workloads. Python can really add overhead in more complex workflows and the GIL is a notorious source of headaches. You know, those hugging face guys kindaaaaa do know what they’re talking about. Like, really. Python is a pain in the ass in production environments.
- ravenstine 3y ago> You know, those hugging face guys kindaaaaa do know what they’re talking about. > Like, really. Trust me, bro?
- 3y ago
- xorvoid 3y agoNeat. This is the library I’ve been waiting to see. Rust has some nice potential but big parts of the ecosystem are missing. Now let’s see if it actually gets traction.
- milliams 3y agoI have been occasionally checking https://www.arewelearningyet.com https://www.arewelearningyet.com for where these problems are being solved. It doesn't look like Candle is listed there yet.
- hackeromm 3y ago[dead]
- delijati 3y agoHow big are the executabels ? I would like to move our interence code from tensorflow to a executable
- josephg 3y agoI just compiled the llama example on my M1 mac. The resulting binary is 9.5mb in size. I assume that includes support for apple's ML acceleration but not CUDA.
- jiangplus 3y agoAnyone remember that Torch was first glued with Lua before PyTorch?
- GaggiX 3y agoI would love to see a single binary application with bleeding edge neural networks in it without all the complexity and bloat added when using Python in distributed software programs. I guess you would be able to achieve it even before but this library should help a lot.
- GaggiX 3y agoHopefully the library will support in the future the shapes of the tensors in the type system, that would be great to prevent many errors.
- liuliu 3y agoAfter a few years, I am coming to an opposite opinion on this. Yeah, shape checking is nice and all and you should be able to run them before-hand (might be a linter pass). But inside type-system while most people have LSP running in the background and bark at you on any type mismatch? It is not going to be a pleasant development experience.
- GaggiX 3y agoI cannot really see how it can be a worse experience than waiting for the model to run, fix the error found, wait a second time, fix another error; when you can just check instantly if the model architecture is sound.
- liuliu 3y agoBecause that's not my workflow looks like? * Fire up Jupyter notebook, create some PyTorch nn.Module, validate the output is expected. * Built out the model, validate the output is expected. * Check the data shape, do a eval() run with the data. * Move out of Jupyter notebook (or not), to actually run the trainer. Each of the "build out model" phase, shape inference at compile time would be kind of annoying especially if you have a LSP constantly running compilation.
- GaggiX 3y agoThat's also my workflow but the "Check the data shape, do a eval() run with the data." is the annoying process I was describing, the iterative "waiting for the model to run", "see that it doesn't work", "print random tensor shape trying the debug the code" etc... There is nothing annoying about having an oracle that can check it instantly unless you think that suffering should be part of the experience.
- timokoesters 3y agoThis looks like a great project! I have a question: Why is it necessary to specify the device for every Tensor? Wouldn't it be possible to set the device once and then all allocations are made to that device?
- polyrand 3y agoInteresting. The repo was owned by Laurent Mazare[0] a few days ago, I'm wondering if HuggingFace just hired them to work on candle full time? [0]: Also maintainer of https://github.com/LaurentMazare/tch-rs https://github.com/LaurentMazare/tch-rs
- skdotdan 3y agoLooks great! Rust is the wrong language for developing ML models, but it has a clear potential for being the best one for deploying them.
- sva_ 3y ago"Torch replacement"? Title should be "Candle: Minimalist ML framework for Rust" if anything.
- kvark 3y agoExciting tech, looking forward to see it running on wgpu! There is an issue about WebGPU although I don’t notice references to the native version of it. https://github.com/huggingface/candle/issues/344 https://github.com/huggingface/candle/issues/344
- hackeromm 3y ago[dead]
- bastardoperator 3y agoIn the video game Rust, you spawn with a Torch. They also just did a torch replacement in game. Maybe it's time for a break...
- steveklabnik 3y agoI've been playing Rust lately. Take from that what you will, hahah.
- Animats 3y agoOh, another Candle and Torch. There are also programs called "Candle" and "Torch" inside the WIX toolset, a widely used generator of Microsoft Windows installers. (Not to be confused with Wix.com, the heavily promoted low-end web hosting service.)
- nuccy 3y agoA logically connected name for a Torch-like project in Rust could have been Thermite Torch. You can make a thermite torch given that rust is iron oxide :)
- pc2g4d 3y agoWhat's the multithreading story? tch-rs's (and libtorch's) lack of real threading support was causing problems for my async code - at least, I haven't found a way to work around it yet, given my level of rust knowledge. Tangentially: I keep thinking an ML framework could be done based on Rust macros, so the automatic differentiation is done entirely at compile time. `let grad = autodiff!(x + y);` (The `autodiff` crate does this at runtime IIUC). Then the compiler's optimizations can apply to the static computation graph. I always thought that was the vision for TF in the early days, but clearly it moved in a more dynamic direction under PyTorch's influence.
- l-m-z 3y agoThe tensors should be Send and Sync so can be manipulated from multiple threads, the underlying data is protected by a RWLock to guard against data races. The heavy operations such as matrix multiplication will be run on multiple cores even without using some explicit threading.
- SushiHippie 3y agoCould this also run quantized models like llama.cpp?
- version_five 3y agoI'll say it though I expect violent downmoderation: Rust is way too political for me to ever consider using it for anything real. I've been starting to worry the same about python too (and really all of software) but Rust and Mozilla have that vibe of wanting to go way beyond software development and into identity politics, and it's dangerous to have dependencies with those priorities. Especially in AI because there's an overlap with the "safety" nonsense. New frameworks are always interesting but I wouldn't use rust seriously for anything until the ecosystem broadens enough to incorporate all viewpoints and get away from politics.
- serial_dev 3y agoDespite all the drama that happens every second month, I'm still "Rust curious". Working with Rust is fun, and I hope it will be my main language in my next job. You are not the first that noticed the overly political nature of the Rust "community", not a coincidence that we see a blue haired individual in this skit https://youtu.be/TGfQu0bQTKc https://youtu.be/TGfQu0bQTKc I didn't downvote you, but you might be downvoted not because you are wrong, but because it derails the conversation. Turning every Rust thread into a "The Foundation is too woke" can be annoying.
- version_five 3y agoI'm sure it's annoying, but it's better than staying quiet. If people like the language on it's technical merits, which is believe is the majority of Rust enthusiasts, it's important for them not to stand by and let ideologues hijack their software project for political aims.
- orangetuba 3y agoI couldn't agree more. This has turned me away from Rust as well.