10 ms·

Inside an AI 'brain' – What does machine learning look like?

124 points by breck 10y ago

partycoder 10y ago
I wonder what is the motivation behind rendering the network inside a circle, and how to interpret from these charts... like: which neural ensemble is connected to which, etc.
- isoprophlex 10y ago
  Beautiful images for sure, but what do we learn from them? Without any explanation of the questions you raise, this page is 99% marketing speak, and to me, next to useless.
  - partycoder 10y ago
    Didn't want to sound as severe but yes... it's hard to decipher why they want it to make like a petri dish under a microscope.
- posterboy 10y ago
  I suppose the circle with its ideal properties (ie. smallest area to circumference ratio) allows for denser packing, or at least for easier calculation of packing because there are literally no corner cases
Smerity 10y ago
The consensus in the machine learning / deep learning research communities is that this is a colourful "nothing" visualization. Little to no information is conveyed, it doesn't represent the architectures in any sane way that researchers or the broader non-technical community can use / understand, and it primarily seems a vehicle to use the hype surrounding AI to advertise Graphcore. I'm not the only one[1] within the research community to think this. If you find any insights from this, I'd honestly first be surprised and then second be interested to know what insights you gleamed from it. Background: researcher who publishes papers in deep learning. [1]: https://twitter.com/jackclarkSF/status/834461913262157824 https://twitter.com/jackclarkSF/status/834461913262157824 (thread containing a member of OpenAI who specializes in communicating complex machine learning topics to the media and a primary developer of PyTorch / member of Facebook's AI Research lab)
- maldeh 10y ago
  One really contrived usecase may be "flame graphs" for high level diagnostics of learned models in a production pipeline, as you collect more data or modify features. But I don't imagine it'd be very good. - the images and false colors need to show some semblance of stability for a given network between epochs; and it needs to be robust against changing data or input structure. - requiring visual inspection doesn't give you something you can automate with, unlike an evaluation score. - if there is indeed a significant deviation in "MRI"-like scans between batches, its diagnostic utility ends there - it tells you nothing about what caused a change.
- Kiro 10y ago
  It looks like a brain and that's enough for me to jump on the hype train. I don't really care if it does nothing more.
  - make3 10y ago
    well shit I'm sure I can take a RAM dump and soup up fancy graph code to make it look like a brain .. it still won't mean anything
    - kowdermeister 10y ago
      Great, the do it, then we will A/B I'm in love with data visualization / generative art.
- deepnet 10y ago
  Aesthetically enjoyable and with an organic feel, the intricate detail suggests a lot of information is represented, yet Graphcore's diagram does not seem immediately apprehensible. In the OP article's masthead, the clusters are labelled, this is AlexNet's computational graph from a Tensorflow description (depicted in full lower down the page). On the right is "Conv1 11x11 forward [3 in, 64 out]" which suggests a Convolutional Layer with 3 inputs and 64 outputs. Forward, presumably, the direction of tensors flowing through the layer. Alexnet's Layer 1 is Convolutional [a] : • Images: 227x227x3 • F (receptive field size): 11 • S (stride) = 4 • Convlayer output: 55x55x96 Compare Colah's 2D Convolutional NN depiction: http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ or CS231n's page containing both structural and activation diagrams: http://cs231n.github.io/convolutional-networks/ http://cs231n.github.io/convolutional-networks/ In reply to your linked tweet [1], Chintala asserts the links are " compute/mem activity on their cores" and "connections between clusters are memory transfers/activity" Does Chintala mean compute cores, Graphcore's IPUs ? From Graphcore's page: "computational graphs are made up of vertices (think neurons) for the compute elements, connected by edges (think synapses), which describe the communication paths between vertices." What do Graphcore's colours represent ? What is an IPU ? Is an IPU hardware like Google's TPU ? [edit] Graphcore explains: "Our Poplar graph compiler has converted a description of the network into a computational graph of 18.7 million vertices and 115.8 million edges. This graph represents AlexNet as a highly-parallel execution plan for the IPU. The vertices of the graph represent computation processes and the edges represent communication between processes. The layers in the graph are labelled with the corresponding layers from the high level description of the network. The clearly visible clustering is the result of intensive communication between processes in each layer of the network, with lighter communication between layers." So it is a compiled computational graph just like Tensorflow's. I cannot fathom it further and am unsure how or if to proceed. Counting the little dots and lines in cluster Conv1 and relating this to to 11x11 [3 in, 64 out] & Alexnet's 1st Layer could be a place to start an inquiry. [a] http://vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet_tugce_kyunghee.pdf http://vision.stanford.edu/teaching/cs231b_spring1415/slides...
YeGoblynQueenne 10y ago
Historically, purpose-built processor architectures were always surpassed by general-purpose architectures. Think of Lisp Machines and x86, for a very relevant example of an architecture specifically designed for AI applications, that was quickly rendered obsolete by a general-purpose architecture. Why graphcore is going to be any different is anybody's guess. Although, I admit the concept sounds cool on paper and the graph plots look pretty- I'd hang one on my wall for sure.
- sdenton4 10y ago
  We're in a different historical moment, though. GPUs are specialized hardware for linear algebra, and have kind of taken over. TPUs are similarly specialized. The big difference is that Moore's law seems to be stalled out for standard, general purpose processing; as a result there's a lot to be gained in specializing. Now whether these particular professors are worth anything is a different question...
  - posterboy 10y ago
    *processors
- deepnotderp 10y ago
  General purpose always won because a) the demands wasn't great enough b) the steamroller of moore's law helped general purpose better and c) previous special purpose chips were maybe 10x better, not 100-250X.
fxj 10y ago
Here is some explanation: https://www.graphcore.ai/blog/graph-computing-for-machine-intelligence-with-poplar https://www.graphcore.ai/blog/graph-computing-for-machine-in...
zebrafish 10y ago
Can somebody explain this to me? Unlike a scalar CPU or a vector GPU, the Graphcore Intelligent Processing Unit (IPU) is a graph processor. Is this just marketing mumbo-jumbo? I don't understand how a "graph processor" would look any different than a vector processor.
- posterboy 10y ago
  a gpu is optimized for 2 and 3d, graphs might represent higher topologies?
- deepnotderp 10y ago
  Founder of a different deep learning chip startup here. They're talking about deep learning's computational graphs. The only differences between that and a general purpose program graph is that a) the (general) absence of control flow and b)the operations are all arithmetic and generally predictably so. So basically yeah, it's marketing.
gnipgnip 10y ago
Seems like PR bait. Tim Davis' page is much much nicer. http://www.cise.ufl.edu/research/sparse/matrices/synopsis/ http://www.cise.ufl.edu/research/sparse/matrices/synopsis/ I think Trefethen too has nice visualization like this (or maybe that was the spectral thing).
jlebrech 10y ago
Wow that looks organic, it looks like a bacteria learning and adapting to its surroundings.

124 points by breck 10y ago

I wonder what is the motivation behind rendering the network inside a circle, and how to interpret from these charts... like: which neural ensemble is connected to which, etc.

isoprophlex 10y ago
Beautiful images for sure, but what do we learn from them? Without any explanation of the questions you raise, this page is 99% marketing speak, and to me, next to useless.
- partycoder 10y ago
  Didn't want to sound as severe but yes... it's hard to decipher why they want it to make like a petri dish under a microscope.
posterboy 10y ago
I suppose the circle with its ideal properties (ie. smallest area to circumference ratio) allows for denser packing, or at least for easier calculation of packing because there are literally no corner cases

Smerity 10y ago

The consensus in the machine learning / deep learning research communities is that this is a colourful "nothing" visualization. Little to no information is conveyed, it doesn't represent the architectures in any sane way that researchers or the broader non-technical community can use / understand, and it primarily seems a vehicle to use the hype surrounding AI to advertise Graphcore. I'm not the only one[1] within the research community to think this. If you find any insights from this, I'd honestly first be surprised and then second be interested to know what insights you gleamed from it. Background: researcher who publishes papers in deep learning. [1]: https://twitter.com/jackclarkSF/status/834461913262157824 https://twitter.com/jackclarkSF/status/834461913262157824 (thread containing a member of OpenAI who specializes in communicating complex machine learning topics to the media and a primary developer of PyTorch / member of Facebook's AI Research lab)

maldeh 10y ago
One really contrived usecase may be "flame graphs" for high level diagnostics of learned models in a production pipeline, as you collect more data or modify features. But I don't imagine it'd be very good. - the images and false colors need to show some semblance of stability for a given network between epochs; and it needs to be robust against changing data or input structure. - requiring visual inspection doesn't give you something you can automate with, unlike an evaluation score. - if there is indeed a significant deviation in "MRI"-like scans between batches, its diagnostic utility ends there - it tells you nothing about what caused a change.
Kiro 10y ago
It looks like a brain and that's enough for me to jump on the hype train. I don't really care if it does nothing more.
- make3 10y ago
  well shit I'm sure I can take a RAM dump and soup up fancy graph code to make it look like a brain .. it still won't mean anything
  - kowdermeister 10y ago
    Great, the do it, then we will A/B I'm in love with data visualization / generative art.
deepnet 10y ago
Aesthetically enjoyable and with an organic feel, the intricate detail suggests a lot of information is represented, yet Graphcore's diagram does not seem immediately apprehensible. In the OP article's masthead, the clusters are labelled, this is AlexNet's computational graph from a Tensorflow description (depicted in full lower down the page). On the right is "Conv1 11x11 forward [3 in, 64 out]" which suggests a Convolutional Layer with 3 inputs and 64 outputs. Forward, presumably, the direction of tensors flowing through the layer. Alexnet's Layer 1 is Convolutional [a] : • Images: 227x227x3 • F (receptive field size): 11 • S (stride) = 4 • Convlayer output: 55x55x96 Compare Colah's 2D Convolutional NN depiction: http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ http://colah.github.io/posts/2014-07-Conv-Nets-Modular/ or CS231n's page containing both structural and activation diagrams: http://cs231n.github.io/convolutional-networks/ http://cs231n.github.io/convolutional-networks/ In reply to your linked tweet [1], Chintala asserts the links are " compute/mem activity on their cores" and "connections between clusters are memory transfers/activity" Does Chintala mean compute cores, Graphcore's IPUs ? From Graphcore's page: "computational graphs are made up of vertices (think neurons) for the compute elements, connected by edges (think synapses), which describe the communication paths between vertices." What do Graphcore's colours represent ? What is an IPU ? Is an IPU hardware like Google's TPU ? [edit] Graphcore explains: "Our Poplar graph compiler has converted a description of the network into a computational graph of 18.7 million vertices and 115.8 million edges. This graph represents AlexNet as a highly-parallel execution plan for the IPU. The vertices of the graph represent computation processes and the edges represent communication between processes. The layers in the graph are labelled with the corresponding layers from the high level description of the network. The clearly visible clustering is the result of intensive communication between processes in each layer of the network, with lighter communication between layers." So it is a compiled computational graph just like Tensorflow's. I cannot fathom it further and am unsure how or if to proceed. Counting the little dots and lines in cluster Conv1 and relating this to to 11x11 [3 in, 64 out] & Alexnet's 1st Layer could be a place to start an inquiry. [a] http://vision.stanford.edu/teaching/cs231b_spring1415/slides/alexnet_tugce_kyunghee.pdf http://vision.stanford.edu/teaching/cs231b_spring1415/slides...

YeGoblynQueenne 10y ago

Historically, purpose-built processor architectures were always surpassed by general-purpose architectures. Think of Lisp Machines and x86, for a very relevant example of an architecture specifically designed for AI applications, that was quickly rendered obsolete by a general-purpose architecture. Why graphcore is going to be any different is anybody's guess. Although, I admit the concept sounds cool on paper and the graph plots look pretty- I'd hang one on my wall for sure.

sdenton4 10y ago
We're in a different historical moment, though. GPUs are specialized hardware for linear algebra, and have kind of taken over. TPUs are similarly specialized. The big difference is that Moore's law seems to be stalled out for standard, general purpose processing; as a result there's a lot to be gained in specializing. Now whether these particular professors are worth anything is a different question...
- posterboy 10y ago
  *processors
deepnotderp 10y ago
General purpose always won because a) the demands wasn't great enough b) the steamroller of moore's law helped general purpose better and c) previous special purpose chips were maybe 10x better, not 100-250X.

fxj 10y ago

Here is some explanation: https://www.graphcore.ai/blog/graph-computing-for-machine-intelligence-with-poplar https://www.graphcore.ai/blog/graph-computing-for-machine-in...

zebrafish 10y ago

Can somebody explain this to me? Unlike a scalar CPU or a vector GPU, the Graphcore Intelligent Processing Unit (IPU) is a graph processor. Is this just marketing mumbo-jumbo? I don't understand how a "graph processor" would look any different than a vector processor.

posterboy 10y ago
a gpu is optimized for 2 and 3d, graphs might represent higher topologies?
deepnotderp 10y ago
Founder of a different deep learning chip startup here. They're talking about deep learning's computational graphs. The only differences between that and a general purpose program graph is that a) the (general) absence of control flow and b)the operations are all arithmetic and generally predictably so. So basically yeah, it's marketing.

gnipgnip 10y ago

Seems like PR bait. Tim Davis' page is much much nicer. http://www.cise.ufl.edu/research/sparse/matrices/synopsis/ http://www.cise.ufl.edu/research/sparse/matrices/synopsis/ I think Trefethen too has nice visualization like this (or maybe that was the spectral thing).

jlebrech 10y ago

Wow that looks organic, it looks like a bacteria learning and adapting to its surroundings.