14 ms·

Tensorflow.js: Machine Learning in JavaScript

217 points by memexy 6y ago

ghj 6y ago
Any example of this being used in the wild? I feel like it could save with backend costs (especially for hobby projects) but the bandwidth/loading time might make it infeasible? Also if you have any novel model it will be trivial to reverse engineer it. You gotta send the weights over to the client and they can just run tensorflow.js themselves right?
- memexy 6y ago
  Yes, the idea is that the model is local to the user and runs in their browser (pushing the computation to the edge). Here's a project demonstrating pose estimation from Twilio's engineering blog: https://www.twilio.com/blog/pose-detection-video-tensorflow-js https://www.twilio.com/blog/pose-detection-video-tensorflow-.... Re: bandwidth. Do you mean the weights? If so then yes, you'd have to think about distilling large models to smaller ones. There are techniques for doing this. Here is an explanation of distillation from Floyd Hub's engineering blog: https://blog.floydhub.com/knowledge-distillation/ https://blog.floydhub.com/knowledge-distillation/.
- GistNoesis 6y ago
  I used it in the wild for some week-end project [1][2]. To get around the bandwidth issue, I used it as a browser extension. This means that weights are downloaded and installed only once during the setup. But I also used it in the past in a website for real-time audio processing when model are small enough [3]. Novel models will probably need to have some custom layers, which are quite painful to write. The weights for the web version will probably be a low quality of what you achieve on a more powerful machine. And you don't need to provide the code for the training, nor the dataset used, so you will still have an edge over people copying you. The main problem I have with tensorflow.js is that it's not production ready yet. It's probably 10 years ahead of it's time. Most people don't have recent GPU or recent video drivers. As it's doing some tricks with the GPU to have some acceleration, a fraction of your clients will encounter random bugs and crashes. I even got some bad review for causing reboots on Firefox :). For some people it will be so slow it will feel unresponsive, and others will find that it's unimpressive because you would have to use the minimal model. In a day and age where you need 99% satisfaction to exist, picking tensorflow.js is a mistake. [1] chrome : https://chrome.google.com/webstore/detail/colorify/pipnfjhpbdpejoemmkhoglekbjakmnkc https://chrome.google.com/webstore/detail/colorify/pipnfjhpb... [2] firefox : https://addons.mozilla.org/en-US/firefox/addon/colorify/ https://addons.mozilla.org/en-US/firefox/addon/colorify/ [3] Wisteria : https://gistnoesis.github.io/ https://gistnoesis.github.io/
  - ijpsud 6y ago
    I was playing with tf.js last year and there were some core components still being converted to WASM, which gave me some big performance boosts once I switched. I had some problems with WebGL-related bugs too. It seems that they're currently in the process of writing WebGPU-based back-ends. I'm hoping that when WebGPU is on-by-default in most browsers, and there's been a year or two to sort out all the major teething problems, we'll see some wide-spread uptake. Internet speeds are still increasing exponentially, so hopefully the model size problems will become less of an issue - perhaps aided by some CDN-served (and thus cached across domains) "base" models that are fine-tuned with some parameters downloaded from the server. I think I could start playing with it seriously if I could get the model sizes under 30mb or so. In a few years (with increasing internet speeds) that might be 50mb. I think huggingface's distilled GPT-2 model is a couple of hundred megabytes, for reference, so we're certainly not going to be doing anything revolutionary in the browser, but I have a bunch of neat little ideas that I think would be useful. For bringing the "big" stuff to the web we're probably stuck with APIs, like OpenAI's new GPT-3 offering. I think access to SOTA models on hard problems is going to be almost exclusively via APIs for the foreseeable future.
aphroz 6y ago
Will it be blocked by Chrome as resource heavy ? https://blog.chromium.org/2020/05/resource-heavy-ads-in-chrome.html?m=1 https://blog.chromium.org/2020/05/resource-heavy-ads-in-chro...
- foobarding 6y ago
  If used in a third party context, sure.
  - speedgoose 6y ago
    If the user interacts with the frame it should go well.
- onion2k 6y ago
  That's only relevant to ads. Hopefully no one is putting AI models in adverts.
  - paozac 6y ago
    I'm afraid that will happen soon
  - aphroz 6y ago
    How is an ad defined? In the case of mining i.e. they probably don't explicitly define it as an ad, no? Also I guess using clients' computing power for AI is as valuable as mining Monero.
jwilber 6y ago
Here’s something I made awhile back that uses it to generate Bob Ross drawings live from your sketches. (Never wrapped it up into an app but maybe I should): https://twitter.com/jdwlbr/status/1131244682317484032?s=20 https://twitter.com/jdwlbr/status/1131244682317484032?s=20
- gmanis 6y ago
  This is fantastic. Any way to see and learn from source?
aerovistae 6y ago
As a non-ML practitioner who has an interest in the field, I have a question I was hoping someone could answer. Looking at the page this links to, it gives pretrained models for "common use cases", among other things. Can you actually do anything interesting or novel with those? I feel like any meaningful new use of ML will not be possible with such "beginner building blocks" and you would just wind up with a weaker version of existing offerings. Could someone give me any examples of meaningful ways this can be used?
- sk0g 6y ago
  Re-training! The generalised models are... Generalised. They have learned to pick up details from a vast, varied data set. You can reuse that, and fine tune to the model to your specific data set, which could save you days worth of computation, and also decreases the data set size requirement on your end.
- memexy 6y ago
  There is a process called transfer learning where you take a model that has been trained on a large data set and then modify the last few layers of the neural network by training them on your specific/smaller data set. Not sure if there are any good resources for beginners but I learned about the technique from fast.ai videos.
- ml_basics 6y ago
  Here is an example thing I made using one of the pretrained mdoels (MediaPipe Facemesh): https://github.com/paruby/snake-face/ https://github.com/paruby/snake-face/ I'm sure that many people would not class this as interesting or novel, but it demonstrates that at least some of the building blocks can actually be used to make actual things.
- pabloem 6y ago
  I don't think pretrained models are "beginner building blocks". They provide a kind of "base model" that can be fine tuned to your specific need. The advantage of not training the full model means you can save many resources (in design time and computation). There are many boring but meaningful tasks in which this can be used. For example, I'm sure many industries could be benefit from image classification for very specific cases (e.g., fruit categorization). In those cases, you are not interested in the classification of "general" objects such as car, person, bike or horse (as provided by MobiletNet pretrained model), but you can use that model as base to classify different categories of fruit. Therefore, you are right that they might not be very useful to build new ML algorithms or network architectures. But they are useful to build specific (and novel) uses of current neural networks.
revskill 6y ago
Do you know how to consume model produced by python using tensorflow.js ?
- deleted 6y ago
  [deleted]
- ml_basics 6y ago
  I played around with tf.js a few weeks ago and learning to use python-trained models was the first hurdle. I made a minimal working example of this here, feel free to check it out: https://github.com/paruby/mnist https://github.com/paruby/mnist See model/mnist_js.ipynb to see how the model is trained and exported to a js readable format, and you can see in lines 13-15 in index.html how the model is loaded. Note: I have a decent amount of ML experience but almost no javascript experience, so YMMV
  - johnx123-up 6y ago
    TF Lite to TFJS might be a good option. But, this ticket is pending for a long time https://github.com/tensorflow/tfjs/issues/991 https://github.com/tensorflow/tfjs/issues/991 Do you have any solution for this?
- jaster 6y ago
  there is a specific tool to convert from various formats to a tensorflow.js model: https://github.com/tensorflow/tfjs/tree/master/tfjs-converter https://github.com/tensorflow/tfjs/tree/master/tfjs-converte...
ml_basics 6y ago
I recently wanted to learn how to use tf.js, so had a couple of mini-projects. The first is a minimal working example showing how to train a model in python and then load it in javascript. https://github.com/paruby/mnist https://github.com/paruby/mnist The second is the classic game of snake, controlled by pointing your head in the direction you want to steer the snake. https://github.com/paruby/snake-face/ https://github.com/paruby/snake-face/ This uses the MediaPipe Facemesh model with the device camera to work out which direction your head is pointing in.
- songzme 6y ago
  I have a hard time sorting through the ml logic vs the game logic, it would really help my learning (and others too) if the game logic was decoupled from the ml logic into 2 separate files.
jansan 6y ago
I will have to purchase a new PC soon. Provided I want to play around with Tensorflow.js, is there anything that I shoudl avoid o urgently include in the spec? Any specific graphics card or processor?
- 0x008 6y ago
  if you want to make use of your GPU you need an NVIDIA graphics card
- sk0g 6y ago
  I'd wait for the RTX 3000 series if possible, which would be around September or October. Seems like it will have better ML performance, and more VRAM, too.
- epmaybe 6y ago
  If you can wait, wait as the other commenter stated. However, if you're just interested in playing with machine learning, you don't need to get the latest and greatest tech! Even the raspberry pi can do some machine learning experiments or inferences.
- nl 6y ago
  If you just want to play with Tensorflow.js in browsers then any video card will do. If you want to use Node then I think you can get accelerated performance using a NVidia video card. If you want to train your own models then you'll need to use Python (or in theory Swift or something..) and you want an NVidia video card.
deleted 6y ago
[deleted]
Joe-Z-L 6y ago
Super noob question, I have a awesome use for image classification on my job, can I make it on Teachable Machine export as TensorFlow Js and make a simple page/app with simplistic UI just to drop/choose a image for the model to compare. What would be a roadmap or path to do this ? (I started programming literally yesterday With the intention to make what I wrote real)
- zzot 6y ago
  The Teachable Machine FAQ (https://teachablemachine.withgoogle.com/faq#Saving-&-Exporting https://teachablemachine.withgoogle.com/faq#Saving-&-Exporti...) says yes. See the "Can I use Teachable Machine models with other libraries and platforms?" question.
- ijpsud 6y ago
  Yep! It looks like that's exactly what the new version of Teachable Machine is for: https://teachablemachine.withgoogle.com/ https://teachablemachine.withgoogle.com/ I might have misunderstood your comment, but if you've just started learning to code, I'd probably start with simpler tutorials and projects - just to get a handle on the basics. Depending on your background it could take a good 6 months of daily practice to get comfortable with the basics. It took me and my friends at least that long - similar time frame to learning a spoken language like Japanese, say, to a basic level. Learning to code is absolutely worth it though - 6 months is a very cheap price in my opinion. I'd probably start with a tutorial like this one: https://www.youtube.com/watch?v=yPWkPOfnGsw https://www.youtube.com/watch?v=yPWkPOfnGsw Or the Khan Academy computer science course.
epmaybe 6y ago
For an image classification project I wanted to complete, I had to do a bunch of matrix/tensor manipulation, and I thought it would be interesting to do it all client side in JavaScript. Things like changing the color space, taking slices/patches of images, etc. I'd be happy to share some of that if people are truly interested!
- memexy 6y ago
  Yes, I'm interested. If you have notes please share. The more documented examples there are of how to use Tensorflow.js the better.
mrojecki 6y ago
Maybe here someone can help me with my struggle. I'm trying to tackle the topic of ML / AI for some time but I don't want to just use any existing tool and go with the flow - what I would like to do, is to understand how everything works, including underlying math. Just to give you an example, I would like to build my own image recogniton tool from scratch and understand every part of it. Where can I start? Any book recomendations would be greatly appreciated.
- frozenport 6y ago
  Bad idea, a performant optimization algorithm is difficult to make and will give you little insight into the thought process of modern machine learning practitioners.

217 points by memexy 6y ago

ghj 6y ago

Any example of this being used in the wild? I feel like it could save with backend costs (especially for hobby projects) but the bandwidth/loading time might make it infeasible? Also if you have any novel model it will be trivial to reverse engineer it. You gotta send the weights over to the client and they can just run tensorflow.js themselves right?

memexy 6y ago
Yes, the idea is that the model is local to the user and runs in their browser (pushing the computation to the edge). Here's a project demonstrating pose estimation from Twilio's engineering blog: https://www.twilio.com/blog/pose-detection-video-tensorflow-js https://www.twilio.com/blog/pose-detection-video-tensorflow-.... Re: bandwidth. Do you mean the weights? If so then yes, you'd have to think about distilling large models to smaller ones. There are techniques for doing this. Here is an explanation of distillation from Floyd Hub's engineering blog: https://blog.floydhub.com/knowledge-distillation/ https://blog.floydhub.com/knowledge-distillation/.
GistNoesis 6y ago
I used it in the wild for some week-end project [1][2]. To get around the bandwidth issue, I used it as a browser extension. This means that weights are downloaded and installed only once during the setup. But I also used it in the past in a website for real-time audio processing when model are small enough [3]. Novel models will probably need to have some custom layers, which are quite painful to write. The weights for the web version will probably be a low quality of what you achieve on a more powerful machine. And you don't need to provide the code for the training, nor the dataset used, so you will still have an edge over people copying you. The main problem I have with tensorflow.js is that it's not production ready yet. It's probably 10 years ahead of it's time. Most people don't have recent GPU or recent video drivers. As it's doing some tricks with the GPU to have some acceleration, a fraction of your clients will encounter random bugs and crashes. I even got some bad review for causing reboots on Firefox :). For some people it will be so slow it will feel unresponsive, and others will find that it's unimpressive because you would have to use the minimal model. In a day and age where you need 99% satisfaction to exist, picking tensorflow.js is a mistake. [1] chrome : https://chrome.google.com/webstore/detail/colorify/pipnfjhpbdpejoemmkhoglekbjakmnkc https://chrome.google.com/webstore/detail/colorify/pipnfjhpb... [2] firefox : https://addons.mozilla.org/en-US/firefox/addon/colorify/ https://addons.mozilla.org/en-US/firefox/addon/colorify/ [3] Wisteria : https://gistnoesis.github.io/ https://gistnoesis.github.io/
- ijpsud 6y ago
  I was playing with tf.js last year and there were some core components still being converted to WASM, which gave me some big performance boosts once I switched. I had some problems with WebGL-related bugs too. It seems that they're currently in the process of writing WebGPU-based back-ends. I'm hoping that when WebGPU is on-by-default in most browsers, and there's been a year or two to sort out all the major teething problems, we'll see some wide-spread uptake. Internet speeds are still increasing exponentially, so hopefully the model size problems will become less of an issue - perhaps aided by some CDN-served (and thus cached across domains) "base" models that are fine-tuned with some parameters downloaded from the server. I think I could start playing with it seriously if I could get the model sizes under 30mb or so. In a few years (with increasing internet speeds) that might be 50mb. I think huggingface's distilled GPT-2 model is a couple of hundred megabytes, for reference, so we're certainly not going to be doing anything revolutionary in the browser, but I have a bunch of neat little ideas that I think would be useful. For bringing the "big" stuff to the web we're probably stuck with APIs, like OpenAI's new GPT-3 offering. I think access to SOTA models on hard problems is going to be almost exclusively via APIs for the foreseeable future.

aphroz 6y ago

Will it be blocked by Chrome as resource heavy ? https://blog.chromium.org/2020/05/resource-heavy-ads-in-chrome.html?m=1 https://blog.chromium.org/2020/05/resource-heavy-ads-in-chro...

foobarding 6y ago
If used in a third party context, sure.
- speedgoose 6y ago
  If the user interacts with the frame it should go well.
onion2k 6y ago
That's only relevant to ads. Hopefully no one is putting AI models in adverts.
- paozac 6y ago
  I'm afraid that will happen soon
- aphroz 6y ago
  How is an ad defined? In the case of mining i.e. they probably don't explicitly define it as an ad, no? Also I guess using clients' computing power for AI is as valuable as mining Monero.

jwilber 6y ago

Here’s something I made awhile back that uses it to generate Bob Ross drawings live from your sketches. (Never wrapped it up into an app but maybe I should): https://twitter.com/jdwlbr/status/1131244682317484032?s=20 https://twitter.com/jdwlbr/status/1131244682317484032?s=20

gmanis 6y ago
This is fantastic. Any way to see and learn from source?

aerovistae 6y ago

As a non-ML practitioner who has an interest in the field, I have a question I was hoping someone could answer. Looking at the page this links to, it gives pretrained models for "common use cases", among other things. Can you actually do anything interesting or novel with those? I feel like any meaningful new use of ML will not be possible with such "beginner building blocks" and you would just wind up with a weaker version of existing offerings. Could someone give me any examples of meaningful ways this can be used?

sk0g 6y ago
Re-training! The generalised models are... Generalised. They have learned to pick up details from a vast, varied data set. You can reuse that, and fine tune to the model to your specific data set, which could save you days worth of computation, and also decreases the data set size requirement on your end.
memexy 6y ago
There is a process called transfer learning where you take a model that has been trained on a large data set and then modify the last few layers of the neural network by training them on your specific/smaller data set. Not sure if there are any good resources for beginners but I learned about the technique from fast.ai videos.
ml_basics 6y ago
Here is an example thing I made using one of the pretrained mdoels (MediaPipe Facemesh): https://github.com/paruby/snake-face/ https://github.com/paruby/snake-face/ I'm sure that many people would not class this as interesting or novel, but it demonstrates that at least some of the building blocks can actually be used to make actual things.
pabloem 6y ago
I don't think pretrained models are "beginner building blocks". They provide a kind of "base model" that can be fine tuned to your specific need. The advantage of not training the full model means you can save many resources (in design time and computation). There are many boring but meaningful tasks in which this can be used. For example, I'm sure many industries could be benefit from image classification for very specific cases (e.g., fruit categorization). In those cases, you are not interested in the classification of "general" objects such as car, person, bike or horse (as provided by MobiletNet pretrained model), but you can use that model as base to classify different categories of fruit. Therefore, you are right that they might not be very useful to build new ML algorithms or network architectures. But they are useful to build specific (and novel) uses of current neural networks.

revskill 6y ago

Do you know how to consume model produced by python using tensorflow.js ?

deleted 6y ago
[deleted]
ml_basics 6y ago
I played around with tf.js a few weeks ago and learning to use python-trained models was the first hurdle. I made a minimal working example of this here, feel free to check it out: https://github.com/paruby/mnist https://github.com/paruby/mnist See model/mnist_js.ipynb to see how the model is trained and exported to a js readable format, and you can see in lines 13-15 in index.html how the model is loaded. Note: I have a decent amount of ML experience but almost no javascript experience, so YMMV
- johnx123-up 6y ago
  TF Lite to TFJS might be a good option. But, this ticket is pending for a long time https://github.com/tensorflow/tfjs/issues/991 https://github.com/tensorflow/tfjs/issues/991 Do you have any solution for this?
jaster 6y ago
there is a specific tool to convert from various formats to a tensorflow.js model: https://github.com/tensorflow/tfjs/tree/master/tfjs-converter https://github.com/tensorflow/tfjs/tree/master/tfjs-converte...

ml_basics 6y ago

I recently wanted to learn how to use tf.js, so had a couple of mini-projects. The first is a minimal working example showing how to train a model in python and then load it in javascript. https://github.com/paruby/mnist https://github.com/paruby/mnist The second is the classic game of snake, controlled by pointing your head in the direction you want to steer the snake. https://github.com/paruby/snake-face/ https://github.com/paruby/snake-face/ This uses the MediaPipe Facemesh model with the device camera to work out which direction your head is pointing in.

songzme 6y ago
I have a hard time sorting through the ml logic vs the game logic, it would really help my learning (and others too) if the game logic was decoupled from the ml logic into 2 separate files.

jansan 6y ago

I will have to purchase a new PC soon. Provided I want to play around with Tensorflow.js, is there anything that I shoudl avoid o urgently include in the spec? Any specific graphics card or processor?

0x008 6y ago
if you want to make use of your GPU you need an NVIDIA graphics card
sk0g 6y ago
I'd wait for the RTX 3000 series if possible, which would be around September or October. Seems like it will have better ML performance, and more VRAM, too.
epmaybe 6y ago
If you can wait, wait as the other commenter stated. However, if you're just interested in playing with machine learning, you don't need to get the latest and greatest tech! Even the raspberry pi can do some machine learning experiments or inferences.
nl 6y ago
If you just want to play with Tensorflow.js in browsers then any video card will do. If you want to use Node then I think you can get accelerated performance using a NVidia video card. If you want to train your own models then you'll need to use Python (or in theory Swift or something..) and you want an NVidia video card.

deleted 6y ago

[deleted]

Joe-Z-L 6y ago

Super noob question, I have a awesome use for image classification on my job, can I make it on Teachable Machine export as TensorFlow Js and make a simple page/app with simplistic UI just to drop/choose a image for the model to compare. What would be a roadmap or path to do this ? (I started programming literally yesterday With the intention to make what I wrote real)

zzot 6y ago
The Teachable Machine FAQ (https://teachablemachine.withgoogle.com/faq#Saving-&-Exporting https://teachablemachine.withgoogle.com/faq#Saving-&-Exporti...) says yes. See the "Can I use Teachable Machine models with other libraries and platforms?" question.
ijpsud 6y ago
Yep! It looks like that's exactly what the new version of Teachable Machine is for: https://teachablemachine.withgoogle.com/ https://teachablemachine.withgoogle.com/ I might have misunderstood your comment, but if you've just started learning to code, I'd probably start with simpler tutorials and projects - just to get a handle on the basics. Depending on your background it could take a good 6 months of daily practice to get comfortable with the basics. It took me and my friends at least that long - similar time frame to learning a spoken language like Japanese, say, to a basic level. Learning to code is absolutely worth it though - 6 months is a very cheap price in my opinion. I'd probably start with a tutorial like this one: https://www.youtube.com/watch?v=yPWkPOfnGsw https://www.youtube.com/watch?v=yPWkPOfnGsw Or the Khan Academy computer science course.

epmaybe 6y ago

For an image classification project I wanted to complete, I had to do a bunch of matrix/tensor manipulation, and I thought it would be interesting to do it all client side in JavaScript. Things like changing the color space, taking slices/patches of images, etc. I'd be happy to share some of that if people are truly interested!

memexy 6y ago
Yes, I'm interested. If you have notes please share. The more documented examples there are of how to use Tensorflow.js the better.

mrojecki 6y ago

Maybe here someone can help me with my struggle. I'm trying to tackle the topic of ML / AI for some time but I don't want to just use any existing tool and go with the flow - what I would like to do, is to understand how everything works, including underlying math. Just to give you an example, I would like to build my own image recogniton tool from scratch and understand every part of it. Where can I start? Any book recomendations would be greatly appreciated.

frozenport 6y ago
Bad idea, a performant optimization algorithm is difficult to make and will give you little insight into the thought process of modern machine learning practitioners.