5 ms·
Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually
by modeless 3y ago
Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does.
I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch.
JavaScript is so much faster than Python, and the package management experience is so much better, that I really think a JS library would have a chance to take market share from PyTorch in the long term if it had those features.
- eduardoleao052 3y agoCould not agree more hahaha! I tried to work around it building methods like “torch.add(a, b)” for operator overloading and “torch.at(index)” for slicing. But it’s definitely not as seamless as these features you proposed.
- modeless 3y agoYou should do it! If you actually had a solution for operator overloading you'd really stand out from the other various JS deep learning libraries. Save me from pip and conda please :)
- eduardoleao052 3y agoI can try implementing it in the future lol It would surely be a quality of life improvement. But with the current tools, I tried my best to make the syntax as similar as possible to PyTorch’s!
- throwuwu 3y agoIf you do want to add evaluation of mathematical expressions you should check out Math.js since they provide a parser among other utilities. Please make it optional though, it would be a nightmare to debug if everything was written in strings. https://mathjs.org/docs/expressions/parsing.html https://mathjs.org/docs/expressions/parsing.html
- eduardoleao052 3y agoThanks for the tip, will look into it! Yes, I think it would always be better to leave a "vanilla" option available.
- noduerme 3y agoCurious, why do you need to construct these as class instances, like operation = new Exp() ? Seems like a lot of extra overhead constructing those objects. Why not just have Exp contain static methods for forwards and backwards? [edit] nevermind, I missed the cache step. Still not sure it wouldn't be more performant to centralize caches as plain objects somewhere rather than to call new() on every op...?
- eduardoleao052 3y agoI centralized the entire backpropagation around the Operation objects. They store the data about the forward prop in the cache, and serve as the connections in the graphs between tensors. Each tensor has a “parent”, “child” and “operation”. These store who generated the tensor, what tensors it generated, and how it was generated (what operation). I could store the backward function inside of each tensor instead of an Operation object, but I chose the slightly more verbose option because I think it is a little more interpretable and simpler to add new operations.
- sroussey 3y agoAgreed. My suggestion is for Bun to do that since they are already transforming typescript to JS. But you can do it —- there a babel plugin in the TC39 proposal: https://github.com/tc39/proposal-operator-overloading https://github.com/tc39/proposal-operator-overloading
- throwaway4aday 3y agoWhat's wrong with creating a function that does those things? It would be less surprising to people new to the library, would be self-documenting by having a name and an easily inspected declaration with named arguments, and it would be idiomatic JS. You could also have variants that are purely functional and return a new value or ones that mutate in place that you could use depending on your needs.
- QuadmasterXLII 3y agoIn python when I try a new gpu accelerated array library, to write norm_arr = arr / lib.sqrt( lib.sum( arr*2, axis=-1, keepdims=True) , I have to read documentation for sum to see whther they use axis or dim in sum. In javascript, to write the same thing I need to read documentation for a sum function, a broadcasted division function, a multiply function. I can probably assume that the sqrt function behaves and is named as I would expect
- 4hg4ufxhy 3y agoWhy can you make assumptions about operation overloads but not functions?
- foolswisdom 3y agoBecause there is nothing to make assumptions about. In the example code, both multiplication and division have a scalar on one side, there's no possible ambiguity of behavior. But there is the eternal question of terminology: do you specify dimensions by "axis" or "dim" and does your API actually use both terms in different places? (that's what I think the GP meant, anyway).
- throwaway4aday 3y agoIf each of those operators were implemented as functions then you'd have different names for different implementations in order to avoid confusion over what type of division or multiplication they were performing. It's more verbose but that's a good thing since it prevents you from making incorrect assumptions about what's going to happen when you do a * b.
- goatlover 3y agoWould JS be faster than Python when it comes to Pytorch? For example, I seriously doubt that would be the case for Numpy, since it's a wrapper for C code, with the ability to use Fortran libraries for optimization.
- throwaway4aday 3y agoThe benefit of having it in JS is not speed but portability and access to the JS ecosystem of tools. Having the code run in the browser without needing a complex setup is a huge benefit for sharing demos. Node.js provides a way to use native code as well and it's quite commonly used https://github.com/nodejs/node-gyp https://github.com/nodejs/node-gyp so there's no reason you couldn't use those same or similar libraries in a JS implementation.
- fulafel 3y agoVery much the opposite, since this is pure JS. PyTorch uses tuned native-code and GPU components for the heavy lifting and can in some cases compile Python code using PyTorch JIT / torch.compiler / torch.fx.
- _ea1k 3y agoIt'd be shocking if it was. PyTorch isn't particularly slow.
- modeless 3y agoC running on the CPU isn't fast enough for ML. You need to run on GPUs or TPUs if you're serious. Yes, most of the tensor operations in PyTorch do their math in native code. However, Python still does orchestration and other tasks like data loading and because it is so slow it still ends up causing a ton of overhead in many cases despite offloading most of the work. It's very common for the GPU to sit idle between kernels while Python spins. So JavaScript being faster could still be a big advantage.
- dagw 3y agoThere is often a lot of work that has to be done before and after PyTorch gets involved. For example the code I'm working on right now involves reading and parsing a bunch of files, filtering and extracting a bunch of data based on various criteria, formatting that data, passing it to a PyTorch model and then taking the results from PyTorch, validating it, reformatting it and then writing it to disk. The PyTorch part is probably as fast as it can get, but most of the overall runtime is spent doing all that other stuff and if you can speed that up then that is a clear win in many cases.
- mejutoco 3y agoYou could use eval to expand whatever syntax you like into function calls. It could be done once at the start of the program and act like a js preprocessor. I know eval is not kosher but this problem is also not real, so why not.
- eduardoleao052 3y agoThat's an interesting idea as well, could definitely see that working in some use cases.