6 ms·
Deep Learning in JavaScript
- treprinum 3y agoDo you plan to implement WebGPU acceleration to make it production-ready?
- eduardoleao052 3y agoI am currently studying Typescript and other JavaScript libraries to improve the library’s performance. So adding GPU support is in sight in the future.
- dleeftink 3y agoYou might already be familiar, but a GPU.js backend can provide some speedups via good old WebGL -- no need for WebGPU just yet! [0]: https://github.com/gpujs/gpu.js/ https://github.com/gpujs/gpu.js/
- eduardoleao052 3y agoThanks for the tip! I’ll definitely take a look. Adding GPU support is my next step!
- dleeftink 3y agoNot a problem, I've benchmarked some GPU backed js libraries here: [0]: https://observablehq.com/@dleeftink/ultimate-compute-show-down https://observablehq.com/@dleeftink/ultimate-compute-show-do...
- eduardoleao052 3y agoThank you! That’s going to help out a lot!
- pzo 3y agoFWIW also taichi is quite popular in python and seems has some javascript related implementation (I haven't used it though), taichi.js [0] [0] https://github.com/AmesingFlank/taichi.js https://github.com/AmesingFlank/taichi.js
- eduardoleao052 3y agoCool! Will take a look
- eduardoleao052 3y agoLearning Machine Learning, I’ve always been interested in PyTorch and its Automatic Backpropagation. In this project, I tried to reimplement most of PyTorch in Javascript. It is implemented from scratch in a well-documented, unit tested, and interpretable way, so it can help other JavaScript learners get into Machine Learning! Hope you enjoy!
- deleted 3y ago[deleted]
- krohling 3y agoThis is awesome! Congrats. Clearly a lot of work and a very cool library.
- eduardoleao052 3y agoThank you very much for the feedback! If you have any question about it, let me know.
- sroussey 3y agoYou might look into node api or ffi and add some native code to speed up some operations on the server. I helped with this project to do that and get prebuilt binaries: https://github.com/ashvardanian/SimSIMD https://github.com/ashvardanian/SimSIMD I haven't looked at ffi over node yet.
- eduardoleao052 3y agoThanks for the tip! I will look into it. Speed in the tensor calculations is definitely one of the largest challenges in this project.
- golergka 3y ago[flagged]
- BetterWhisper 3y ago[flagged]
- marviel 3y agoVery very cool to see!!! So yours is SO MUCH MORE fleshed out than mine, but I have a similar (again very fledgling) one going here that uses Typescript: https://github.com/Marviel/lab-grad https://github.com/Marviel/lab-grad I think there are some really neat opportunities to increase the approachability of ML algorithms when you take advantage of the Typescript templating system to explicitly ensure you're not accidentally composing tensors of incorrect shapes. Unfortunately, my above lib above doesn't yet handle tensors, it's still at the level of individual neurons. Hopefully someday I can bring it further along, use WASM or WebGPU, etc. :)
- eduardoleao052 3y agoThanks for the tip! I will look into your project, for sure! I am still a beginner on Typescript, but when I get the hang of it, I think I might refactor my package to Typescript. Again, thanks for the tip!
- pcthrowaway 3y agoTypescript is something you learn progressively over years, and from what others have posted it might not be possible to refactor your package in Typescript (some approaches to problems that are possible in JS aren't supported by TS). But type declarations at the very least should be possible
- eduardoleao052 3y agoFair enough, I’m currently studying what would be the best option, but I think you are right!
- GaggiX 3y agoThe nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disable the mask from attention. That said, very cool project.
- eduardoleao052 3y agoYou are absolutely right! Will solve that as soon as possible. Thanks for the feedback!
- _ea1k 3y agoIf you have the option to disable the mask, isn't it then a generic nn.Block?
- eduardoleao052 3y agoIt would be with some simple tweaks. For instance, the current block does not support Cross-Attention, just Self-Attention.
- GaggiX 3y agoThe ability to disable the mask in nn.MultiHeadSelfAttention, then having nn.DecoderBlock and a nn.EncoderBlock.
- eduardoleao052 3y agoI think that’s it. I’ll probably add that soon
- mhuffman 3y agoI like to see progress with multiple languages in the space. Julia, R, etc. It does seem though, that python really has a big and commanding lead. So much so that mojo is betting their business on it.
- eduardoleao052 3y agoThat’s true. I myself learned ML in python first, and I’m just now trying to implement some stuff in JavaScript. I guess it could be useful to make web integration with neural networks easier in this case, or at least as an educational package!
- keithalewis 3y agoWhat language is Python calling to do this? Hint: a language closer to the silicon.
- datascienced 3y agoC++, CUDA apis?
- csjh 3y agoWhy not using a faster language that compiles to WebAssembly?
- eduardoleao052 3y agoNow that I've tried Javascript, I could branch out to other languages in the future. I chose Javascript initially for its popularity and easy use on web browsers.
- datascienced 3y agoWouldn’t WebGPU be the key to unlocking speed?
- csjh 3y agoThat too, but WebGPU isn’t very well supported yet
- synergy20 3y agoIs the goal here to do everything in Javascript, or it will also be Javascript for the glue and c++/c/fortran/other-compiled-language for heavy lifting? This is more of a node-based effort, or it will work both frontend and backend.
- eduardoleao052 3y agoInitially I'll stick to Javascript (plus some libraries to improve performance), but I'll likely look into that in the future.
- modeless 3y agoSomeone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does. I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch. JavaScript is so much faster than Python, and the package management experience is so much better, that I really think a JS library would have a chance to take market share from PyTorch in the long term if it had those features.
- eduardoleao052 3y agoCould not agree more hahaha! I tried to work around it building methods like “torch.add(a, b)” for operator overloading and “torch.at(index)” for slicing. But it’s definitely not as seamless as these features you proposed.
- modeless 3y agoYou should do it! If you actually had a solution for operator overloading you'd really stand out from the other various JS deep learning libraries. Save me from pip and conda please :)
- eduardoleao052 3y agoI can try implementing it in the future lol It would surely be a quality of life improvement. But with the current tools, I tried my best to make the syntax as similar as possible to PyTorch’s!
- throwuwu 3y agoIf you do want to add evaluation of mathematical expressions you should check out Math.js since they provide a parser among other utilities. Please make it optional though, it would be a nightmare to debug if everything was written in strings. https://mathjs.org/docs/expressions/parsing.html https://mathjs.org/docs/expressions/parsing.html
- ralusek 3y agoMany people seem to be unaware of tensorflow.js, an official JS implementation of TF https://github.com/tensorflow/tfjs https://github.com/tensorflow/tfjs I'd love to see PyTorch in JS, but I think unless you get it running on the GPU it won't be able to do much.
- richrichie 3y agoWith PyTorch dominating the landscape, my guess was that tf will resurrect itself through tfjs. Seems may not.
- eduardoleao052 3y agoAdding GPU support soon is absolutely my goal in the future! I think a PyTorch-based JavaScript library could be useful, as PyTorch has been way more dominant than TensorFlow recently.
- ipsum2 3y agotfjs is dead, looking at the commit history. The standard now is to convert PyTorch to onnx, then use onnxruntime (https://github.com/microsoft/onnxruntime/tree/main/js/web https://github.com/microsoft/onnxruntime/tree/main/js/web) to run the model on the browser using webassembly/webGL (and nodejs if you wanted to, but why?).
- fire_lake 3y ago> nodejs if you wanted to, but why? Node.js is better backend than something like Flask.
- ipsum2 3y agoPerformance is a lot worse on NodeJS with a WebAssembly/WebGL backend versus Flask with a PyTorch/CUDA backend.
- throwaway4aday 3y agoIf you're using Node you can write whatever you want in C++ and then add a binding to call it from within your Node app. Don't need WebGL.
- sroussey 3y agoBTW: you might want to add support for typed arrays. See: https://github.com/xenova/transformers.js/blob/8804c36591d11d8456788d1bb4b16489121b3be2/src/utils/tensor.js#L45 https://github.com/xenova/transformers.js/blob/8804c36591d11... This is really old, but added as part of the shape of the vector as well: https://github.com/nicolaspanel/numjs/blob/master/src/dtypes.js https://github.com/nicolaspanel/numjs/blob/master/src/dtypes...
- jzombie 3y agoI agree. This advice should not be ignored; and if the OP is complaining about performance, the answer is lying here in plain sight.
- eduardoleao052 3y agoI had read about this as a possibility, but will definitely look more into it now, thank you for the tip!
- Sn0wCoder 3y agoThis is great. Code seems to have good comments / jsdoc from the bit studied, tests and all. Will take a closer look in the morning but congratulations on the release.
- eduardoleao052 3y agoThank you for the feedback! If you have any questions or suggestions looking it over tomorrow, I’d love to hear them!
- hexhowells 3y agoCool project! I worked on something similar a while ago to learn about automatic differentiation: https://github.com/hexhowells/onegrad.js https://github.com/hexhowells/onegrad.js Needs more layers to be really useful though. I always thought examples using browser extensions would be neat since their built in JS and you only need to download the model once.
- eduardoleao052 3y agoReally cool implementation. Thanks for the feedback as well!
- throwaway4aday 3y agoWow! Thank you for doing this. It looks like a great starting point for anyone approaching deep learning from the JS ecosystem. It is very plainly written and looks like it will be a joy to learn from. Thank you for adding JSDoc comments with type hints! Are you open to pull requests? If I have the time I'd love to contribute. I'm sure others would as well. You should write up a short article on this, even something really simple like one of the examples in the README but with some commentary and examples of output and then post it to a few places like https://dev.to/ https://dev.to/ or maybe https://hashnode.com/ https://hashnode.com/ or even Medium (even though I'm not a big fan). There aren't many newer implementations of PyTorch in JS and I've been looking for one to learn from for some time so I'm sure there are a lot of other JS/TS developers out there that would be interested. Getting to the front page of HN certainly helps but having an article somewhere will help everyone after this week find it through a Google search. Again, thanks so much for doing this work! It's really helpful to have everything spelled out in JS for those of us who haven't used Python much (I'm sure Python devs can relate when they think about JS projects).
- eduardoleao052 3y agoThank you so much for the feedback! I’m open to pull requests, absolutely. And about the article, that’s a great tip, I’ll definitely do that as well.
- eiriklv 3y agoEdit: Great work! I'd love to have a nice alternative to PyTorch in JavaScript :) Edit: formatting Making JavaScript look like Python in this case could potentially bite you in the ass. From the example: const torch = require("js-pytorch"); // Instantiate Tensors: x = torch.randn([8,4,5]); w = torch.randn([8,5,4], requires_grad = true); b = torch.tensor([0.2, 0.5, 0.1, 0.0], requires_grad = true); And: class Transformer extends nn.Module { constructor(vocab_size, hidden_size, n_timesteps, n_heads, p) { //..... this.b1 = new nn.Block(hidden_size, hidden_size, n_heads, n_timesteps, dropout_p=p); For both `requires_grad` and `dropout_p` you wouldn't be able to change the ordering + you're creating global variables. /** * All of the arguments for this function are positional * and cannot be provided in a different order than defined */ function performOperation(values, arg1 = false, arg2 = -1) { //.... } /** * This works, but only because of the order */ const result = performOperation([1, 2, 3], arg1 = true, arg2 = 10); /** * This does not work */ const result = performOperation([1, 2, 3], arg2 = 10, arg1 = true); /** \* What is actually happening \*/ arg1 = true; // global variable arg2 = 10; // global variable const result = performOperation([1, 2, 3], 10, true);
- eduardoleao052 3y agoThats true, it’s a limitation of working between these languages. I tried to mitigate it by using clear JSDoc, so that each variable pops up alongside an explanation when calling a function.
- eiriklv 3y agoI feel you - Python has much better (more flexible) argument support than JavaScript in this case. Converting the entire set of arguments into a keyed object is usually what happens, but then it wouldn't look like PyTorch anymore.
- fmiras 3y agonice work but why code is in closures using ugly `var` statement?
- yoaquim 3y agoThis is great! I was looking for something like this. Now I just need a guide that tackles the principles of this, but from a typescript/javascript perspective. PS: TyTorch (from Typescript) sounds nice!
- eduardoleao052 3y agoI plan on adding Typescript support soon! Thanks for the feedback
- ngcc_hk 3y agoOne of the learning tool is notebook. I even has a lisp backend to enabled it to be used. It will be nice if there is a notebook version and easy to run on web (like colab).
- eduardoleao052 3y agoI’m planning on creating a small article explaining the syntax and the functionalities of the Deep Learning library. I think that could be a useful learning tool, do you think it would help?