6 ms·
Nx: Multi-dimensional tensors Elixir lib with multi-staged compilation (CPU/GPU)
- thibaut_barrere 6y agoAlso just above (https://github.com/elixir-nx/nx/tree/main/exla https://github.com/elixir-nx/nx/tree/main/exla), exla is an "Elixir client for Google's XLA (Accelerated Linear Algebra). It includes integration with the Nx library to compile numerical definitions (defn) to the CPU/GPU."
- losvedir 6y agoJose is so prolific on such varied kinds of projects. I expect he'll team up with John Carmack next to solve General AI... I know the newest BEAM release has a new JIT compiler for faster calculations. Is that work related to this at all? I'm guessing not, since it seems like the speed up here is from moving the calculation out of the BEAM.
- dnautics 6y agoit's not related to the new jit.
- themgt 6y agoWhat's really amazing to me is Nx is just installed as normal hex package. `defn` is a standard Elixir macro: https://github.com/elixir-nx/nx/blob/main/nx/lib/nx/defn.ex#L242-L283 https://github.com/elixir-nx/nx/blob/main/nx/lib/nx/defn.ex#...
- ajtulloch 6y agoA bit petty, but the first example is an unstable softmax implementation: defn softmax(t) do Nx.exp(t) / Nx.sum(Nx.exp(t)) end See https://ogunlao.github.io/2020/04/26/you_dont_really_know_softmax.html#numerical-stability-of-softmax https://ogunlao.github.io/2020/04/26/you_dont_really_know_so... etc.
- josevalim 6y agoThanks for the info and the reference, that was a good read!
- lawn 6y agoElixir as a language is really approachable. If you're looking to dive in I recommend the book `Elixir in Action` but the official docs are also quite good.
- 0b01 6y agoI don't find this useful in any way since Elixir is weakly typed and a worse Lisp than Python..
- lawn 6y agoHow exactly is Elixir "a worse Lisp than Python"? Elixir is a functional language and it even has Lisp-style macros.
- 0b01 6y agodeleted
- pmarreck 6y agodouches get downvotes https://games.greggman.com/game/dynamic-typing-static-typing/ https://games.greggman.com/game/dynamic-typing-static-typing... The hard evidence that static typing is an overall-win is lacking. Period. or, more succinctly, https://i.imgur.com/zbU1wwH.jpg https://i.imgur.com/zbU1wwH.jpg
- deleted 6y ago[deleted]
- moocowtruck 6y agohow is static typing lacking evidence? don't we use that in rust to prevent the existence of large swaths of errors vs say C? i know for sure when turning on typescript for some javascript projects at work we found many issues, those all seem like wins? Why when I look at some dynamic languages say for example clojure i see stuff like type annotations in their core libraries, and then spec's to help provide a lot of what a type system can do? do you have maybe a counter argument to this fellow https://www.youtube.com/watch?v=XTl7Jn_kmio https://www.youtube.com/watch?v=XTl7Jn_kmio
- fulafel 6y ago
- dnautics 6y agoThe Nx architecture is quite good. I started work on a backend that avoids requiring jvm/bazel/python toolchain (substituting in a zig toolchain): https://github.com/ityonemo/ez https://github.com/ityonemo/ez the two languages feel like they were made for each other. Despite the verbosity of doing some things in zig (the dynamic tensor broadcast function is a bit of a beast), due to both language's similar compile-time metaprogramming facilities, getting to the point of plugging in Nx's tensor addition + tensor multiplication is about 200 LoC.
- jeffreysmith 6y agoCool project. I'd never heard of Zig, but it looks like an interesting solution to the need you've identified. I'd definitely agree that it seems valuable to have an Nx version that comes with a smaller, simpler install for development purposes. XLA is amazing, but it's not obvious that you want to have that toolchain setup for local development in all scenarios. Would be great to see several different backends created for Nx as a way of exploring what feature sets are really needed and ensuring that the integration mechanisms are sufficiently simple and general to support multiple solutions.
- dnautics 6y agoWell another place is for elixir nerves. You probably don't want to figure out how to shipping the xla architecture to your embedded device (I think it has to JIT the code); with nerves you can trivially cross-compile the nif shared library with zigler and deploy to arm architecture. But kudos to José for building an architecture at a very good level of abstraction, you know, just in case xla falls out of favor (it is a Google project), nx won't have a hard dependency, and there will be trivial migration paths out. I think there could be some other interesting nx backends, like "remote computation", or "transpile elixir and run on a Julia worker node"
- cultofmetatron 6y agoAs if I needed another reason to just stay in elixir fulltime. Machine learning/deeplearning was one of the few niches I was looking outside elixir for.
- josevalim 6y agoHi everyone, co-creator of Nx here! I have just published an article on Dashbit's blog with a bit more context on Nx and the design decisions behind it: https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... It is hopefully a more in-depth reference than the README. If you have any questions, I will be glad to answer them!
- rdtsc 6y agoJust wanted to say thank you, and how much I appreciate your work for the Elixir and Erlang communities. A common complaint with Elixir and Erlang is that it isn't suitable for numerical computations, well now it is!
- nnadams 6y agoI'm excited to try this out, especially after seeing the XLA support. Where do you see Nx going in the future? I saw there's an MNIST example in the repo. Do you plan including higher-level features for neural networks, similar to Keras/PyTorch?
- josevalim 6y agoSean is looking into higher-level features for NN - it will likely be a separate library on top of Nx. For general direction, I briefly discuss that at the end of the article above. :) If something is unclear, please let me know!
- rhizome31 6y agoAs someone working in the weather field, the reference to xarray at the end of the article is very exciting. I imagine that eventuellay it will be possible to read NetCDF files and create dataframes based on Nx.
- elcritch 6y agoGreat initiative! One bit that sticks out to me is that the "standard" math functions like `exp` or `ln` aren't imported despite `defn` being quite maths specific. It seems that they could be imported as default functions inside the scope of `defn`. That leads me to a more important syntax question though. Do you have any thoughts or plans on operator syntax for elementwise operations vs matrix level operations? One big "win" for numerical Python was having `@` become an infix operator specifically for matrix dots [0, 1]. The syntax in Julia (which follows Matlab) is to have most math operators also have an element wise version. It's generalized to "broadcast" operations in Julia [2]. R appears to have ``, `%%, `%o%` infix operators for elementwise, inner, and outer matrix multiplication. Really writing any degree of matrix (ahem Tensor) maths and equations becomes very tedious and illegible if you have to use function calls to distinguish elementwise vs matrix operations. The pipe operator really does help significantly, even with simpler stats oriented code (I've tried a few different formats [4]). Still it's still harder to read if you're familiar with "broadcast" syntax from other mathematics oriented languages. 0: https://alysivji.github.io/python-matrix-multiplication-operator.html https://alysivji.github.io/python-matrix-multiplication-oper... 1: https://www.python.org/dev/peps/pep-0465/ https://www.python.org/dev/peps/pep-0465/ 2: https://julialang.org/blog/2018/05/extensible-broadcast-fusion/ https://julialang.org/blog/2018/05/extensible-broadcast-fusi... 3: https://www.statmethods.net/advstats/matrix.html https://www.statmethods.net/advstats/matrix.html 4: https://github.com/elcritch/matrex_numerix/blob/5835a9b477d8ea41bb9b862272a0997fe37c1236/lib/fft.ex#L81 https://github.com/elcritch/matrex_numerix/blob/5835a9b477d8...
- vfclists 6y agoI learned something new, a creature called the Numbat. I thought it was a pun on "wombat"
- ashton314 6y agoI too learned about the number from this project. I also learned that it’s endangered. The GitHub README has a link to where you can donate to help out.
- fouric 6y agoAs a programmer with no ML/GPU experience, this reminds me of the Spiral language: https://github.com/mrakgr/The-Spiral-Language https://github.com/mrakgr/The-Spiral-Language I wonder if Nx will succeed due being "just" a library for a popular language (Elixir) instead of being a separate language entirely. "Don't make a new language if your idea could be implemented as a library" is good advice indeed.
- 1_player 6y agoAs a regular full stack developer writing Elixir full time, what is the use of such a library? I have heard of Tensorflow, there is mention of tensors in this article, but what would be the practical use of such a library, in the real world? When would one write code like the one shown in the examples? I'm happy that Elixir is becoming more and more mainstream. It's the best language I've used in my 15 years in the business.
- rhizome31 6y agoAt a company I work for we manipulate a lot of weather data which comes in the form of big multi-dimensional arrays of numbers. We use Elixir but we delegate this data processing to Python using the xarray library. Nx is the foundation of an ecosystem that would allow to build similar libraries for Elixir.
- antpls 6y agoIf I may ask, why is your company using Elixir? Wouldn't it make more sense to use Python everywhere on your web stack if you do scientific work on weather data?
- rhizome31 6y agoIn fact the vast majority of our data processing is in Python but we also have some R and shell scripts. Elixir is great for orchestrating parallel data processing and building reactive UIs showing what's happening in real-time. The Erlang VM has interesting performance and introspection capabilities and Elixir being a younger language its tooling is a lot less messy.
- ashton314 6y agoI love how this is implemented on top of existing Elixir utilities: just some macros and fancy AST transforms. The fact that you can do all this with just a library and not any sort of super low-level language pragma speaks to the elegance of Elixir's core design. I think my dream job would be something like what José Valim does: designing a gorgeous, performant language that delights developers. Some day…
- jpf0 6y agoIf I understand correctly, Nx.defn effectively calls a backend compiler for that function, and that will presumably result in only one round-trip of data to the accelerator hardware (e.g. GPU). If you have two defn functions, do you 1) lose compilation between them and 2) round trip twice, or can you nest defn functions and get compilation of the composed function?
- dnautics 6y agoI believe that your understanding is correct: defn marks potential boundaries between "jit compiled" or even AOT compiled, and if you call a defn in a defn it will be fused.
- josevalim 6y agoCorrect! Consider when you call "defn" for the first time to be the entry-point. Everything inside that is fused.
- chrisweekly 6y agoTangential mini-rant: yes. naming things is hard. But we already have 'Nx' -- the monorepo tool.