4 ms·

The consensus in the machine learning / deep learning research communities is that this is a colourful "nothing" visualization. Little to no information is conv

by Smerity 10y ago

The consensus in the machine learning / deep learning research communities is that this is a colourful "nothing" visualization. Little to no information is conveyed, it doesn't represent the architectures in any sane way that researchers or the broader non-technical community can use / understand, and it primarily seems a vehicle to use the hype surrounding AI to advertise Graphcore. I'm not the only one[1] within the research community to think this. If you find any insights from this, I'd honestly first be surprised and then second be interested to know what insights you gleamed from it. Background: researcher who publishes papers in deep learning. [1]: https://twitter.com/jackclarkSF/status/834461913262157824 https://twitter.com/jackclarkSF/status/834461913262157824 (thread containing a member of OpenAI who specializes in communicating complex machine learning topics to the media and a primary developer of PyTorch / member of Facebook's AI Research lab)

maldeh 10y ago
One really contrived usecase may be "flame graphs" for high level diagnostics of learned models in a production pipeline, as you collect more data or modify features. But I don't imagine it'd be very good. - the images and false colors need to show some semblance of stability for a given network between epochs; and it needs to be robust against changing data or input structure. - requiring visual inspection doesn't give you something you can automate with, unlike an evaluation score. - if there is indeed a significant deviation in "MRI"-like scans between batches, its diagnostic utility ends there - it tells you nothing about what caused a change.
Kiro 10y ago
It looks like a brain and that's enough for me to jump on the hype train. I don't really care if it does nothing more.
- make3 10y ago
  well shit I'm sure I can take a RAM dump and soup up fancy graph code to make it look like a brain .. it still won't mean anything
  - kowdermeister 10y ago
    Great, the do it, then we will A/B I'm in love with data visualization / generative art.
deepnet 10y ago
Aesthetically enjoyable and with an organic feel, the intricate detail suggests a lot of information is represented, yet Graphcore's diagram does not seem immediately apprehensible. In the OP article's masthead, the clusters are labelled, this is AlexNet's computational graph from a Tensorflow description (depicted in full lower down the page). On the right is "Conv1 11x11 forward [3 in, 64 out]" which suggests a Convolutional Layer with 3 inputs and 64 outputs. Forward, presumably, the direction of tensors flowing through the layer. Alexnet's Layer 1 is Convolutional [a] : • Images: 227x227x3 • F (receptive field size): 11 • S (stride) = 4 • Convlayer output: 55x55x96 Compare Colah's 2D Convolutional NN depiction: http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ or CS231n's page containing both structural and activation diagrams: http://cs231n.github.io/convolutional-networks/ http://cs231n.github.io/convolutional-networks/ In reply to your linked tweet [1], Chintala asserts the links are " compute/mem activity on their cores" and "connections between clusters are memory transfers/activity" Does Chintala mean compute cores, Graphcore's IPUs ? From Graphcore's page: "computational graphs are made up of vertices (think neurons) for the compute elements, connected by edges (think synapses), which describe the communication paths between vertices." What do Graphcore's colours represent ? What is an IPU ? Is an IPU hardware like Google's TPU ? [edit] Graphcore explains: "Our Poplar graph compiler has converted a description of the network into a computational graph of 18.7 million vertices and 115.8 million edges. This graph represents AlexNet as a highly-parallel execution plan for the IPU. The vertices of the graph represent computation processes and the edges represent communication between processes. The layers in the graph are labelled with the corresponding layers from the high level description of the network. The clearly visible clustering is the result of intensive communication between processes in each layer of the network, with lighter communication between layers." So it is a compiled computational graph just like Tensorflow's. I cannot fathom it further and am unsure how or if to proceed. Counting the little dots and lines in cluster Conv1 and relating this to to 11x11 [3 in, 64 out] & Alexnet's 1st Layer could be a place to start an inquiry. [a] http://vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet_tugce_kyunghee.pdf http://vision.stanford.edu/teaching/cs231b_spring1415/slides...
ThomPete 10y ago
I dont think thats important. What is important is that there might be insights. Who know what might come out of experimenting with it.
- dkersten 10y ago
  GP's point was that there are no insights here, just pretty pictures. We can see that they're complex, but that's about it. At least, that seems to be the agreement on this.
  - ThomPete 10y ago
    And my point is that there might be something coming out of trying to visualize this that we don't know of yet.
chestervonwinch 10y ago
> The consensus ... is ... > I'm not the only one within the research community to think this. I should hope not if it's indeed the consensus :) Anyhow, I agree with your sentiment. Comparisons with organic intelligence? Check. Vague descriptions of new technologies? Check. Flashy, uninformative graphics? Check. Maybe I'm being too cynical.
- make3 10y ago
  no you're exactly right
- ythn 10y ago
  "IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories? Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver." [1] http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-on-deep-learning http://spectrum.ieee.org/automaton/robotics/artificial-intel...
  - unityByFreedom 10y ago
    I wish this practical understanding of AI were understood by the public. It seems that, when educating the public about AI in order to advance the field, we must first go through this phase where people do not understand and we risk taking a step back.
thedonkeycometh 10y ago
Yeah, these graphs may encourage people to look into the subject, but they're built on subjectivity.

by Smerity 10y ago

maldeh 10y ago

One really contrived usecase may be "flame graphs" for high level diagnostics of learned models in a production pipeline, as you collect more data or modify features. But I don't imagine it'd be very good. - the images and false colors need to show some semblance of stability for a given network between epochs; and it needs to be robust against changing data or input structure. - requiring visual inspection doesn't give you something you can automate with, unlike an evaluation score. - if there is indeed a significant deviation in "MRI"-like scans between batches, its diagnostic utility ends there - it tells you nothing about what caused a change.

Kiro 10y ago

It looks like a brain and that's enough for me to jump on the hype train. I don't really care if it does nothing more.

make3 10y ago
well shit I'm sure I can take a RAM dump and soup up fancy graph code to make it look like a brain .. it still won't mean anything
- kowdermeister 10y ago
  Great, the do it, then we will A/B I'm in love with data visualization / generative art.

deepnet 10y ago

Aesthetically enjoyable and with an organic feel, the intricate detail suggests a lot of information is represented, yet Graphcore's diagram does not seem immediately apprehensible. In the OP article's masthead, the clusters are labelled, this is AlexNet's computational graph from a Tensorflow description (depicted in full lower down the page). On the right is "Conv1 11x11 forward [3 in, 64 out]" which suggests a Convolutional Layer with 3 inputs and 64 outputs. Forward, presumably, the direction of tensors flowing through the layer. Alexnet's Layer 1 is Convolutional [a] : • Images: 227x227x3 • F (receptive field size): 11 • S (stride) = 4 • Convlayer output: 55x55x96 Compare Colah's 2D Convolutional NN depiction: http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ or CS231n's page containing both structural and activation diagrams: http://cs231n.github.io/convolutional-networks/ http://cs231n.github.io/convolutional-networks/ In reply to your linked tweet [1], Chintala asserts the links are " compute/mem activity on their cores" and "connections between clusters are memory transfers/activity" Does Chintala mean compute cores, Graphcore's IPUs ? From Graphcore's page: "computational graphs are made up of vertices (think neurons) for the compute elements, connected by edges (think synapses), which describe the communication paths between vertices." What do Graphcore's colours represent ? What is an IPU ? Is an IPU hardware like Google's TPU ? [edit] Graphcore explains: "Our Poplar graph compiler has converted a description of the network into a computational graph of 18.7 million vertices and 115.8 million edges. This graph represents AlexNet as a highly-parallel execution plan for the IPU. The vertices of the graph represent computation processes and the edges represent communication between processes. The layers in the graph are labelled with the corresponding layers from the high level description of the network. The clearly visible clustering is the result of intensive communication between processes in each layer of the network, with lighter communication between layers." So it is a compiled computational graph just like Tensorflow's. I cannot fathom it further and am unsure how or if to proceed. Counting the little dots and lines in cluster Conv1 and relating this to to 11x11 [3 in, 64 out] & Alexnet's 1st Layer could be a place to start an inquiry. [a] http://vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet_tugce_kyunghee.pdf http://vision.stanford.edu/teaching/cs231b_spring1415/slides...

ThomPete 10y ago

I dont think thats important. What is important is that there might be insights. Who know what might come out of experimenting with it.

dkersten 10y ago
GP's point was that there are no insights here, just pretty pictures. We can see that they're complex, but that's about it. At least, that seems to be the agreement on this.
- ThomPete 10y ago
  And my point is that there might be something coming out of trying to visualize this that we don't know of yet.

chestervonwinch 10y ago

> The consensus ... is ... > I'm not the only one within the research community to think this. I should hope not if it's indeed the consensus :) Anyhow, I agree with your sentiment. Comparisons with organic intelligence? Check. Vague descriptions of new technologies? Check. Flashy, uninformative graphics? Check. Maybe I'm being too cynical.

make3 10y ago
no you're exactly right
ythn 10y ago
"IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories? Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver." [1] http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-on-deep-learning http://spectrum.ieee.org/automaton/robotics/artificial-intel...
- unityByFreedom 10y ago
  I wish this practical understanding of AI were understood by the public. It seems that, when educating the public about AI in order to advance the field, we must first go through this phase where people do not understand and we risk taking a step back.

thedonkeycometh 10y ago

Yeah, these graphs may encourage people to look into the subject, but they're built on subjectivity.