6 ms·

Hi everyone, thanks for the enthusiasm so far! We are going to make the repository open source next week and, for those who haven't listened to the podcast, I

Hi everyone, thanks for the enthusiasm so far! We are going to make the repository open source next week and, for those who haven't listened to the podcast, I want to align everyone's expectations and say it is very early material. Expect bugs, breaking changes, etc. On the other hand, it is also a great time for anyone who wants to jump in and help shape things, so feedback will be very welcome!

dnautics 6y ago
(I'm the guy who guessed correctly and got early access). Sorry I have been busy with my main job and haven't been able to contrib much in the past couple of months or so, but I just took a look at it and it looks really good so far. I think some of the architectural things are well-considered, and also, one thing that I forsee is "calling out to a remote gpu", where the nx library itself can be used by a node that doesn't have a gpu, instead of having the node with a gpu expose an rpc to the gpuless node. I guess a relevant analogy here would be the difference between graphQL and REST, but possibly also allowing the gpuless node to gracefully degrade to cpu, if the gpu resource isn't available. The Nx API will likely present an opportunity to also make data gravity abstractions cleaner. I think the elixir/erlang way of thinking about "concurrency as a special case of distribution" will apply to Nx, except "gpu as a special case of distribution" which means there won't be a leaky abstraction for remote gpu. I tried to get the Julia folks to adopt this sort of an attitude towards the juliagpu, but I'm a bit of a nobody so it didn't take. Very excited! And I'm whipping up a little bit of something interesting related to Nx myself that hopefully I can release around the time of your talk, but I don't want to make any promises yet.
- rising-sky 6y ago
  > I guess a relevant analogy here would be the difference between graphQL and REST There is more than one difference between GraphQL and REST, could you clarify this analogy? Thanks
  - dnautics 6y ago
    Good point. The relevant analogy is that in GraphQL the client defines the query that gets run and in REST the server defines the query that gets run. So one way you could architect Nx is to have your elixir running on the gpu node expose a handful of prebaked functions and allow your client elixir node issue requests to those prebaked functions ("REST"). But I think a bit more exciting would be to have your gpu node running a relatively dumb Nx server, and have your gpu-less client running Nx, where its math instructions get serialized using a thin "network device" backend instead of a "gpu backend", over to the gpu node, then the gpu node does the math and comes back with the answer ("GraphQL"). For even longer-haul operations you would want to do things like figure out how to represent datasets with tokens that are cached remotely. I think this all is possible, and, relatively easy with the Nx architecture.
ssivark 6y ago
Very excited about this project, since I’ve been looking to wet my feet with Elixir and numerics are the perfect niche for me! Here’s something I’m confused about. For fast numerical code, it’s vey important to be disciplined about memory allocation. So in addition to considering of immutability -vs- mutable pre-allocation, it would seem that the scheduling model ought to be substantially different. Instead of constantly switching between concurrent processes to ensure none of them monopolize the processor, would it make more sense to finish one computation while the contents are in memory/cache, before cleaning it up and swapping to another process? (Especially for really heavy computations). Will process scheduling have to be micromanaged? Is the BEAM compatible with this mode of usage, or is this maybe not a concern?
- dmpk2k 6y ago
  I do not know what Nx is doing, but BEAM has "dirty NIF"s: long-running native functions that are scheduled differently so that they do not impact regular BEAM processes. They're transparent from the user space.
- deleted 6y ago
  [deleted]
- dnautics 6y ago
  If you're interested about the various modes of BEAM concurrency with nifs, I put together this video, but the code it is showcasing does (deliberately) elide over some of the finer details of beam reentrancy: https://youtu.be/l848TOmI6LI https://youtu.be/l848TOmI6LI
- rkangel 6y ago
  Memory allocation and scheduling are both important considerations but are separate problems. BEAM concurrency only involves memory allocation when the processes are created. Switching between them doesn't involve allocation and is very quick (don't forget we are talking about BEAM processes here not OS ones - it's all in userspace and very efficient). As long as you pre-create your pool of worker processes concurrency doesn't incur much allocation hit. You're right that immutability might do though. I am curious to see how this goes - the speed that Elixir normally runs at combined with some data sharing optimisations means that this issue doesn't exist normally. At the speed we care about when doing numerical work it's more of a concern. Or maybe we'll never get into the realms of full optimised C++ where things are sized for cache lines etc. (and that's ok).

dnautics 6y ago

(I'm the guy who guessed correctly and got early access). Sorry I have been busy with my main job and haven't been able to contrib much in the past couple of months or so, but I just took a look at it and it looks really good so far. I think some of the architectural things are well-considered, and also, one thing that I forsee is "calling out to a remote gpu", where the nx library itself can be used by a node that doesn't have a gpu, instead of having the node with a gpu expose an rpc to the gpuless node. I guess a relevant analogy here would be the difference between graphQL and REST, but possibly also allowing the gpuless node to gracefully degrade to cpu, if the gpu resource isn't available. The Nx API will likely present an opportunity to also make data gravity abstractions cleaner. I think the elixir/erlang way of thinking about "concurrency as a special case of distribution" will apply to Nx, except "gpu as a special case of distribution" which means there won't be a leaky abstraction for remote gpu. I tried to get the Julia folks to adopt this sort of an attitude towards the juliagpu, but I'm a bit of a nobody so it didn't take. Very excited! And I'm whipping up a little bit of something interesting related to Nx myself that hopefully I can release around the time of your talk, but I don't want to make any promises yet.

rising-sky 6y ago
> I guess a relevant analogy here would be the difference between graphQL and REST There is more than one difference between GraphQL and REST, could you clarify this analogy? Thanks
- dnautics 6y ago
  Good point. The relevant analogy is that in GraphQL the client defines the query that gets run and in REST the server defines the query that gets run. So one way you could architect Nx is to have your elixir running on the gpu node expose a handful of prebaked functions and allow your client elixir node issue requests to those prebaked functions ("REST"). But I think a bit more exciting would be to have your gpu node running a relatively dumb Nx server, and have your gpu-less client running Nx, where its math instructions get serialized using a thin "network device" backend instead of a "gpu backend", over to the gpu node, then the gpu node does the math and comes back with the answer ("GraphQL"). For even longer-haul operations you would want to do things like figure out how to represent datasets with tokens that are cached remotely. I think this all is possible, and, relatively easy with the Nx architecture.

ssivark 6y ago

Very excited about this project, since I’ve been looking to wet my feet with Elixir and numerics are the perfect niche for me! Here’s something I’m confused about. For fast numerical code, it’s vey important to be disciplined about memory allocation. So in addition to considering of immutability -vs- mutable pre-allocation, it would seem that the scheduling model ought to be substantially different. Instead of constantly switching between concurrent processes to ensure none of them monopolize the processor, would it make more sense to finish one computation while the contents are in memory/cache, before cleaning it up and swapping to another process? (Especially for really heavy computations). Will process scheduling have to be micromanaged? Is the BEAM compatible with this mode of usage, or is this maybe not a concern?

dmpk2k 6y ago
I do not know what Nx is doing, but BEAM has "dirty NIF"s: long-running native functions that are scheduled differently so that they do not impact regular BEAM processes. They're transparent from the user space.
deleted 6y ago
[deleted]
dnautics 6y ago
If you're interested about the various modes of BEAM concurrency with nifs, I put together this video, but the code it is showcasing does (deliberately) elide over some of the finer details of beam reentrancy: https://youtu.be/l848TOmI6LI https://youtu.be/l848TOmI6LI
rkangel 6y ago
Memory allocation and scheduling are both important considerations but are separate problems. BEAM concurrency only involves memory allocation when the processes are created. Switching between them doesn't involve allocation and is very quick (don't forget we are talking about BEAM processes here not OS ones - it's all in userspace and very efficient). As long as you pre-create your pool of worker processes concurrency doesn't incur much allocation hit. You're right that immutability might do though. I am curious to see how this goes - the speed that Elixir normally runs at combined with some data sharing optimisations means that this issue doesn't exist normally. At the speed we care about when doing numerical work it's more of a concern. Or maybe we'll never get into the realms of full optimised C++ where things are sized for cache lines etc. (and that's ok).