4 ms·

Show HN: TensorSpace.js – Neural network 3D visualization framework

167 points by syt123450 8y ago

ParanoidShroom 8y ago
Wow, this looks amazing. Going to check out how to process my model from keras to use with this tool. Doesn't like that complicated. Gonna give this a shot :) Great work !
- syt123450 8y ago
  Thanks. Visualize Keras model is easier than other kinds of model, as TensorSpace's API is designed based on Keras, you can easily use keras-like TensorSpace API to build TensorSpace model and visualize it.
nerush 8y ago
Always wanted to visualize CNNs like this. I wonder whether it is possible to visualize RNNs at all.
- syt123450 8y ago
  I am trying to design RNN layers, however, RNN is not the same as CNN layers. Now we use 3D to visualize CNN layer, as RNN has a time level information, it required 4D to visualize it, but I will try my best to add this attractive feature to TensorSpace!
deleted 8y ago
[deleted]
yters 8y ago
And use an Occulus rift for full immersive visualization!
- mcc- 8y ago
  Well, take a look at this :) https://github.com/NCBI-Hackathons/deVoReaNN https://github.com/NCBI-Hackathons/deVoReaNN Totally independently, we had the same idea of visualizing a DL in VR using an oculus -- a few months ago. We worked on it last Friday and Saturday in our hackathon: https://www.u-hackmed.org/ https://www.u-hackmed.org/ https://www.u-hackmed.org/2018teams/team-3 https://www.u-hackmed.org/2018teams/team-3 Here is a demo: https://youtu.be/E_VWewj_jX8 https://youtu.be/E_VWewj_jX8 Really cool ideas going around in this thread. I really like thoughts around visualizing the changes during training -- not just activation. As these tools currently stand, I agree that they are mostly for educational purposes. Taken to the nth degree, however, I think they can make a DL expert put on a VR headset when they get to work.
ericjang 8y ago
This is neat! I hope you don't mind a bit of constructive criticism here, but early on in my research career I also thought it would be a good idea to "visualize the neural network connectome" in 3D (I implemented a very rudimentary version of your Browser-based visualizer in QT + OpenGL, no training frontend). And then I followed up with an early TensorFlow visualizer https://github.com/ericjang/tdb https://github.com/ericjang/tdb It turns out that while such tools seem useful at first glance, they turn out to not be that helpful to power users. For models bigger than LeNet, things get really ugly to visualize. And once you understand a high-level module and can take its training for granted, there isn't a need anymore to really look at it anymore. It can also be kind of annoying to tumble around in 3D when you just want to look at some activation maps. What does the 3D aspect of the visualization buy you here? Tools like TensorBoard + Jupyter notebooks for inspecting weights and ad-hoc visualizations (e.g. VizDom) seem to strike the right balance. If you want to continue pushing in this direction, I highly encourage embarking on an actual Deep Learning research project using your tool. In ML it's so important to dogfood your own software!
- mendeza 8y ago
  Are there any good guides, tutorials, or research papers that investigate or advise how to inspect weights during training for debugging. The only things I read are to watch out for vanishing gradients, and when fine-tuning the most change in layers are seen toward the end of the network, not the beginning layers.
  - ericjang 8y ago
    Yes, a recent exciting phenomena of interest to researchers is how and why the spectrum of the Hessian appears to separate into 2 parts - a "bulk" part that changes very slowly and "outliers" that change quickly. This suggests that only a few weights in the model actually change during training. If one could determine which weights these are, it might lend to faster and more efficient learning algorithms that don't have to backprop to all the parameters in a large neural network. https://arxiv.org/pdf/1706.04454.pdf https://arxiv.org/pdf/1706.04454.pdf https://openreview.net/forum?id=ByeTHsAqtX https://openreview.net/forum?id=ByeTHsAqtX
breatheoften 8y ago
Is convolution transpose supported in this/tensorflow.js? Might be fun to visualize fully convolutional networks this way...
- syt123450 8y ago
  Yes, TensorSpace provides TransposeConv2d (https://tensorspace.org/html/docs/layerTranspose.html https://tensorspace.org/html/docs/layerTranspose.html)
brian_herman__ 8y ago
This is awesome!
- syt123450 8y ago
  Thanks!
mendeza 8y ago
Would love to see this be vizualized in Augmented reality as well, like an ARKit app.
- syt123450 8y ago
  Augmented reality is really an amazing idea! As Three.js support web vr, I will dig into it. Hoping I can add this feature to TensorSpace soon.
  - mendeza 8y ago
    You should look into 8th wall, they are a startup that implemented really advanced AR on the browser!https://8thwall.com/products-web.html https://8thwall.com/products-web.html
jamesonthecrow 8y ago
This looks really neat and it's definitely fun to play around with. I can't resist playing around with tools like this for a few minutes, but I've never really figured out what they're good for. What am I supposed to learn from them? What is the actionable information? That's not really a criticism, I just feel like I'm missing something.
- syt123450 8y ago
  TensorSpace provides structure information and some basic metric for deep learning, and we use these famous network to show what TensorSpace can do, and how to use TensorSpace API to construct networks. For deep learning experts, these famous network's information is really basic, but we hope TensorSpace can help others to understand existing model, or help to present customized models.
p1esk 8y ago
To provide some feedback: 1. display weights and gradients (in addition to activations) 2. display how activations/weights/gradients change during training. For example, I should be able to point to checkpoint directory where my model is saved every epoch (or every few iterations), and then hit 'play' button to see how a particular feature map or weights filter is changing as I move between epochs/iterations. 3. display which inputs contribute the most to the activations, at every layer. These additional features would be very useful for debugging, for model compression, for identifying information flows (e.g. in a DenseNet), for saliency analysis, and probably for other things as well.
- syt123450 8y ago
  Thanks for your feedback, I am planing to visualize weights in next version of TensorSpace. These days, I add a "live loader" for TensorSpace, user can use this API to visualize online training process. And for others features, we are discussing how to integrate them with TensorSpace, looking forward to your further suggestions.
andreyk 8y ago
Quite neat! Some constructive feedback - I think in addition to mouse controls, having keyboard 'spaceship' flying around is pretty important. I would also try to add model interprability tools (SmoothGrad) as an option, and perhaps contact some of those researchers to see how this can best be used for model understanding (I would guess visualizing grad flows / weights is more interesting than just layers). PS I actually worked on something similar, was quite fun... http://www.andreykurenkov.com/projects/major_projects/KerasJS3D/ http://www.andreykurenkov.com/projects/major_projects/KerasJ...
- syt123450 8y ago
  Thanks so much for your suggestions. As you say, visualizing weights may be really helpful, and I get many feedbacks for adding visualizing weights. I will try to add this feature in later version of TensorSpace. Temporarily, TensorSpace integrate TrackballControl which is an plugin for Three.js to handle "space exploration", and I will try to add more keyboard event for TensorSpace. I have viewed your blog, and it is really fun. However, it is a pity that the online demo is outdated. Could you tell me what technique you use to construct 3D scene? Thanks.
quickthrower2 8y ago
Minority report style!
- syt123450 8y ago
  Thanks!
webmaven 8y ago
Pretty cool. If you're looking for ideas on what sort of information to display in the visualizations, this article is just chock full of them: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/
- syt123450 8y ago
  Thanks for your recommendation, I can not wait to add more interesting idea into TensorSpace!

167 points by syt123450 8y ago

Wow, this looks amazing. Going to check out how to process my model from keras to use with this tool. Doesn't like that complicated. Gonna give this a shot :) Great work !

syt123450 8y ago
Thanks. Visualize Keras model is easier than other kinds of model, as TensorSpace's API is designed based on Keras, you can easily use keras-like TensorSpace API to build TensorSpace model and visualize it.

nerush 8y ago

Always wanted to visualize CNNs like this. I wonder whether it is possible to visualize RNNs at all.

syt123450 8y ago
I am trying to design RNN layers, however, RNN is not the same as CNN layers. Now we use 3D to visualize CNN layer, as RNN has a time level information, it required 4D to visualize it, but I will try my best to add this attractive feature to TensorSpace!

deleted 8y ago

[deleted]

yters 8y ago

And use an Occulus rift for full immersive visualization!

mcc- 8y ago
Well, take a look at this :) https://github.com/NCBI-Hackathons/deVoReaNN https://github.com/NCBI-Hackathons/deVoReaNN Totally independently, we had the same idea of visualizing a DL in VR using an oculus -- a few months ago. We worked on it last Friday and Saturday in our hackathon: https://www.u-hackmed.org/ https://www.u-hackmed.org/ https://www.u-hackmed.org/2018teams/team-3 https://www.u-hackmed.org/2018teams/team-3 Here is a demo: https://youtu.be/E_VWewj_jX8 https://youtu.be/E_VWewj_jX8 Really cool ideas going around in this thread. I really like thoughts around visualizing the changes during training -- not just activation. As these tools currently stand, I agree that they are mostly for educational purposes. Taken to the nth degree, however, I think they can make a DL expert put on a VR headset when they get to work.

ericjang 8y ago

This is neat! I hope you don't mind a bit of constructive criticism here, but early on in my research career I also thought it would be a good idea to "visualize the neural network connectome" in 3D (I implemented a very rudimentary version of your Browser-based visualizer in QT + OpenGL, no training frontend). And then I followed up with an early TensorFlow visualizer https://github.com/ericjang/tdb https://github.com/ericjang/tdb It turns out that while such tools seem useful at first glance, they turn out to not be that helpful to power users. For models bigger than LeNet, things get really ugly to visualize. And once you understand a high-level module and can take its training for granted, there isn't a need anymore to really look at it anymore. It can also be kind of annoying to tumble around in 3D when you just want to look at some activation maps. What does the 3D aspect of the visualization buy you here? Tools like TensorBoard + Jupyter notebooks for inspecting weights and ad-hoc visualizations (e.g. VizDom) seem to strike the right balance. If you want to continue pushing in this direction, I highly encourage embarking on an actual Deep Learning research project using your tool. In ML it's so important to dogfood your own software!

mendeza 8y ago
Are there any good guides, tutorials, or research papers that investigate or advise how to inspect weights during training for debugging. The only things I read are to watch out for vanishing gradients, and when fine-tuning the most change in layers are seen toward the end of the network, not the beginning layers.
- ericjang 8y ago
  Yes, a recent exciting phenomena of interest to researchers is how and why the spectrum of the Hessian appears to separate into 2 parts - a "bulk" part that changes very slowly and "outliers" that change quickly. This suggests that only a few weights in the model actually change during training. If one could determine which weights these are, it might lend to faster and more efficient learning algorithms that don't have to backprop to all the parameters in a large neural network. https://arxiv.org/pdf/1706.04454.pdf https://arxiv.org/pdf/1706.04454.pdf https://openreview.net/forum?id=ByeTHsAqtX https://openreview.net/forum?id=ByeTHsAqtX

breatheoften 8y ago

Is convolution transpose supported in this/tensorflow.js? Might be fun to visualize fully convolutional networks this way...

syt123450 8y ago
Yes, TensorSpace provides TransposeConv2d (https://tensorspace.org/html/docs/layerTranspose.html https://tensorspace.org/html/docs/layerTranspose.html)

brian_herman__ 8y ago

This is awesome!

syt123450 8y ago
Thanks!

mendeza 8y ago

Would love to see this be vizualized in Augmented reality as well, like an ARKit app.

syt123450 8y ago
Augmented reality is really an amazing idea! As Three.js support web vr, I will dig into it. Hoping I can add this feature to TensorSpace soon.
- mendeza 8y ago
  You should look into 8th wall, they are a startup that implemented really advanced AR on the browser!https://8thwall.com/products-web.html https://8thwall.com/products-web.html

jamesonthecrow 8y ago

This looks really neat and it's definitely fun to play around with. I can't resist playing around with tools like this for a few minutes, but I've never really figured out what they're good for. What am I supposed to learn from them? What is the actionable information? That's not really a criticism, I just feel like I'm missing something.

syt123450 8y ago
TensorSpace provides structure information and some basic metric for deep learning, and we use these famous network to show what TensorSpace can do, and how to use TensorSpace API to construct networks. For deep learning experts, these famous network's information is really basic, but we hope TensorSpace can help others to understand existing model, or help to present customized models.

p1esk 8y ago

To provide some feedback: 1. display weights and gradients (in addition to activations) 2. display how activations/weights/gradients change during training. For example, I should be able to point to checkpoint directory where my model is saved every epoch (or every few iterations), and then hit 'play' button to see how a particular feature map or weights filter is changing as I move between epochs/iterations. 3. display which inputs contribute the most to the activations, at every layer. These additional features would be very useful for debugging, for model compression, for identifying information flows (e.g. in a DenseNet), for saliency analysis, and probably for other things as well.

syt123450 8y ago
Thanks for your feedback, I am planing to visualize weights in next version of TensorSpace. These days, I add a "live loader" for TensorSpace, user can use this API to visualize online training process. And for others features, we are discussing how to integrate them with TensorSpace, looking forward to your further suggestions.

andreyk 8y ago

Quite neat! Some constructive feedback - I think in addition to mouse controls, having keyboard 'spaceship' flying around is pretty important. I would also try to add model interprability tools (SmoothGrad) as an option, and perhaps contact some of those researchers to see how this can best be used for model understanding (I would guess visualizing grad flows / weights is more interesting than just layers). PS I actually worked on something similar, was quite fun... http://www.andreykurenkov.com/projects/major_projects/KerasJS3D/ http://www.andreykurenkov.com/projects/major_projects/KerasJ...

syt123450 8y ago
Thanks so much for your suggestions. As you say, visualizing weights may be really helpful, and I get many feedbacks for adding visualizing weights. I will try to add this feature in later version of TensorSpace. Temporarily, TensorSpace integrate TrackballControl which is an plugin for Three.js to handle "space exploration", and I will try to add more keyboard event for TensorSpace. I have viewed your blog, and it is really fun. However, it is a pity that the online demo is outdated. Could you tell me what technique you use to construct 3D scene? Thanks.

quickthrower2 8y ago

Minority report style!

syt123450 8y ago
Thanks!

webmaven 8y ago

Pretty cool. If you're looking for ideas on what sort of information to display in the visualizations, this article is just chock full of them: https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/

syt123450 8y ago
Thanks for your recommendation, I can not wait to add more interesting idea into TensorSpace!