7 ms·

Hello! I'm one of the authors; we'd be happy to answer any questions. Make sure to check out our library and the colab notebooks, which allow you to reproduce

by colah3 9y ago

Hello! I'm one of the authors; we'd be happy to answer any questions. Make sure to check out our library and the colab notebooks, which allow you to reproduce our results in your browser, on a free GPU, without any setup: https://github.com/tensorflow/lucid#notebooks https://github.com/tensorflow/lucid#notebooks I think that there's something very exciting about this kind of reproducibility. It means that there's continuous spectrum of engaging with the paper: Reading <> Interactive Diagrams <> Colab Notebooks <> Projects based on Lucid My colleague Ludwig calls it "enthusiastic reproducibility and falsifiability" because we're putting lots of effort into making it easy.

ludwigschubert 9y ago
We need to go from it being special to open source code for a paper to making it almost seamless to interact with a presented idea, to try it in new contexts, to build upon it, and, yes, to falsify it. When we introduce examples into an article we do so to make it easier to understand our message, but this opens us up to criticism of cherry picking our examples. We take a first step by allowing readers to choose from a set of examples, but real progress is open sourcing the code behind each diagram, and making it seamless for readers to run. If you primarily care about finding truth, then you don't only need to make reproduction and falsification possible, you need to make it easy.
- Nzen 9y ago
  While I agree in principle, in practice this implies a maintenance burden. Who should the burden fall on? Some researchers leave to industry positions. Some leave for family obligations. J Scoggins published an expert system wavefront scheduler [0][1] six years ago. Now, he's working on an expert system processor simulator [2]. Should requests on his earlier work take time from his current work? If he were to release a docker image, or whatever, of his setup while writing these tools is that enough? Obviously, there's a spot to shoot for between tossing binaries over the wall and a month long sabbatical to write documentation and example code for something no one may ever use. I have to ask, because I didn't pursue academia. Would you, as a grad student (per your github), reject or welcome a policy that insisted that the last week, or whatever, of the development window is to document the tools of the study? [0] https://github.com/DrItanium/WavefrontScheduling https://github.com/DrItanium/WavefrontScheduling [1] https://cpp-primo.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=01CALS_ALMA71380013420002901&context=L&vid=01CALS_PUP&lang=en_US&search_scope=EVERYTHING&adaptor=Local%20Search%20Engine&tab=everything&query=any,contains,Scoggins,%20Joshua&sortby=rank&offset=0 https://cpp-primo.hosted.exlibrisgroup.com/primo-explore/ful... [2] https://github.com/DrItanium/maya https://github.com/DrItanium/maya
jashkenas 9y ago
Hi Chris! (Apologies if it's putting you too much on the spot, but I'm actually curious in hearing about this...) My question for you is how you personally navigate the ethical issues that are raised by today's news about Google working with the U.S. military to help classify imagery from drone footage. (1) (2) In your tweet about this paper, you write: It qualitatively changes the questions we can ask. One basic example: neuron 426 fired ~= useless [floppy ear] fired = very interesting! But one can just as easily imagine: neuron 183 fired ~= useless [young man fleeing] fired = very interesting! Are there conversations within Google Brain about the appropriate applications of this research you can share with us? Do you feel like this class of military technology is going to be developed one way or the other, so there's no use in worrying about speeding it along? 1: https://gizmodo.com/google-is-helping-the-pentagon-build-ai-for-drones-1823464533 https://gizmodo.com/google-is-helping-the-pentagon-build-ai-... 2: https://theintercept.com/2018/03/06/google-is-quietly-providing-ai-technology-for-drone-strike-targeting-project/ https://theintercept.com/2018/03/06/google-is-quietly-provid...
- colah3 9y ago
  Hi Jash! I focus my work on technical research directions I believe increase the probability of advances in AI being outstanding for all of humanity. I'm fortunate that Google provides me resources to do so, with broad autonomy to work on what I think is important. My main focus is on building techniques that allow humans to understand how neural networks make decisions, because I think that will be a powerful tool in making sure increasingly automated systems are aligned with society's values -- for example, being fair -- and are making decisions for reasons we endorse. I also spend a lot of time thinking about accident risk and safety. I believe this work is robustly good for the world. Of course, lots of concerns about AI can't be addressed by the kind of technical research I work on. I think technical research into interpretability, robustness, safety, fairness, etc... is necessary but not sufficient for the kind of world we want to see. There are also tough questions of public policy, international governance, and scientific ethics -- such as issues around the abuse of AI systems. On these kind of issues, I'd largely defer to people who think deeply about these topics, like Allan Dafoe (Oxford), Michael Page (OpenAI), and Tim Hwang (Ethics & Governance of AI Fund).
taliesinb 9y ago
Hello! I'm a fan of this latest publication, distill.pub, and your previous work. I have three questions: 1. Flexible interfaces based on the kinds of grammars you describe in this post feel like their time have come, as an important part of scientific communication, but also as a tool to create, test, and update hypotheses themselves. Entire experiments could be written in the language of these grammars, where transformations and operations directly produce the artifacts that support scientific conclusions: things like means and variances, distributions, statistical tests, scatter plots, and the like. What are your thoughts on that? Is that a direction in which Lucid is interested in going? 2. I'm interested in how to automate the creation of such linked interfaces, whether they be for convnets, language models, seq2seq type models, etc. The idea would be to have the user write a query in a DSL that explicitly connects specific interior activations (on a given axis/axes) with gradients and visualization techniques to produce a linked interface. I haven't yet played around with Lucid -- is Lucid already such a DSL for tensorflow? Or does it rather collect various instantiations of the general grammar that have proved useful? 3. What are the best examples you know of linked interfaces and grammar-like approaches to visualization inspiring important discoveries? Within deep learning, it feels like the slam-dunk example is OpenAI's sentiment neuron, although that was a fairly simple form of visualization. I'd be curious to find more examples, both inside and outside the field of deep learning.
cs702 9y ago
Hi Chris! -- I have four comments: First, as always, I have to say THANK YOU, to you, and to the other authors, and to your various helpers, for putting this together, with no obvious goal other than the public good. :-) Second, your idea of (a) taking all activation values in a trained DNN in response to a particular sample, (b) reshaping all these values into a single giant matrix, and (c) factorizing this giant matrix to identify "neuron groups" (i.e., some kind of low-rank approximation) that most closely explain the behavior of the DNN for each particular sample... is a brilliant idea. In hindsight, it seems like an obvious thing to do, but I don't think I've seen anyone else do it before. I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks. Third, I think this barely scratches the surface of the kinds of UI-driven "interpretation tools" that ultimately will be needed to enable (non-technical) human beings be able to associate or attribute DNN behavior to "salient ideas," including those salient ideas for which human beings currently lack descriptive terminology. This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future. Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it. For example, what if the number of "salient ideas" current AI systems can discover vastly exceeds the number of distinct salient ideas a human brain can distinguish and work with? What is your view or opinionated guess on this?
- colah3 9y ago
  > factorizing this giant matrix to identify "neuron groups" ... is a brilliant idea. As with many ideas in this article, Alex deserves all the credit. He's been doing that trick internally at Google for years. (In my experience, if Alex is excited about an idea, >50% odds you'll realize it's a super important idea 2 years later. :P ) I think there's I saw an instance of someone doing PCA on conv net activations to make heat maps. We should try to dig that up and cite it. > I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks. Absolutely! (I think that most of the ideas in our article are pretty general. :) ) > This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future. Thanks! We're super excited as well! > Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it... What is your view or opinionated guess on this? Oh my. That's a super interesting and deep question. Wild wild speculation ahead. Please take everything I say with a big grain of salt. My first comment is that a model can be hard to understand because it's too stupid, in addition to being hard to understand because it's exceeding us. I suspect that if one could have an "interpretability curve" of how easy models are to understand vs model performance -- it's kind of hard to make this sensical, because both variables are actually these really nuance high-dimensional things, but imagine you could -- it would go up as you approach human performance and actually peak after it, before eventually declining again. This intuition is driven by the thought that early superhuman performance for tasks humans are already good at is probably largely about having really crisp, much more statistically precise, versions of our abstractions. Those crisp versions of our abstractions are probably easier to understand than the confused ones. But, of course, that one dimensional view is a gross simplification. I suspect that something that will be important in future thought about this is "alien abstractions" vs "refined abstractions." By a "refined abstraction" I mean something like ears in GoogLeNet. You see, GoogLeNet has dozens of detectors for different kinds of ear -- a much richer vocabulary of ear types than I can articulate -- and knows a lot about how those should influence class probabilities. Although I can't understand the detailed nuance of each ear detector, I can get the general gist and verify that it has reasonable consequences. Conversely, by an "alien abstraction" I mean a feature that I don't have a corresponding idea for. These are much harder for us to deal with. Both "refined" and "alien" abstractions could give superhuman performance. We're in a much better state when refined ones dominate. For visual tasks that humans are already good at, I expect refined abstractions to dominate for a while. In other domains, I have a lot less confidence. I think there's a lot of nuance about where interpretability will easily scale vs have severe challenges is super subtle. I have a draft essay floating around on the topic. Hopefully I'll get it out there someday.
brylie 9y ago
I couldn't read the article on a mobile screen. The text rendered at about 3px high. Please consider using a mobile-friendly CSS design.
nmca 9y ago
Hey - I thought that this was interesting work, but just wanted to jump in and say thanks so much for the effort in reproducibility. This should be the new standard, but putting in the effort to do it now is improving the world at relatively little personal benefit. So thanks!

by colah3 9y ago

ludwigschubert 9y ago

We need to go from it being special to open source code for a paper to making it almost seamless to interact with a presented idea, to try it in new contexts, to build upon it, and, yes, to falsify it. When we introduce examples into an article we do so to make it easier to understand our message, but this opens us up to criticism of cherry picking our examples. We take a first step by allowing readers to choose from a set of examples, but real progress is open sourcing the code behind each diagram, and making it seamless for readers to run. If you primarily care about finding truth, then you don't only need to make reproduction and falsification possible, you need to make it easy.

Nzen 9y ago
While I agree in principle, in practice this implies a maintenance burden. Who should the burden fall on? Some researchers leave to industry positions. Some leave for family obligations. J Scoggins published an expert system wavefront scheduler [0][1] six years ago. Now, he's working on an expert system processor simulator [2]. Should requests on his earlier work take time from his current work? If he were to release a docker image, or whatever, of his setup while writing these tools is that enough? Obviously, there's a spot to shoot for between tossing binaries over the wall and a month long sabbatical to write documentation and example code for something no one may ever use. I have to ask, because I didn't pursue academia. Would you, as a grad student (per your github), reject or welcome a policy that insisted that the last week, or whatever, of the development window is to document the tools of the study? [0] https://github.com/DrItanium/WavefrontScheduling https://github.com/DrItanium/WavefrontScheduling [1] https://cpp-primo.hosted.exlibrisgroup.com/primo-explore/fulldisplay?docid=01CALS_ALMA71380013420002901&context=L&vid=01CALS_PUP&lang=en_US&search_scope=EVERYTHING&adaptor=Local%20Search%20Engine&tab=everything&query=any,contains,Scoggins,%20Joshua&sortby=rank&offset=0 https://cpp-primo.hosted.exlibrisgroup.com/primo-explore/ful... [2] https://github.com/DrItanium/maya https://github.com/DrItanium/maya

jashkenas 9y ago

Hi Chris! (Apologies if it's putting you too much on the spot, but I'm actually curious in hearing about this...) My question for you is how you personally navigate the ethical issues that are raised by today's news about Google working with the U.S. military to help classify imagery from drone footage. (1) (2) In your tweet about this paper, you write: It qualitatively changes the questions we can ask. One basic example: neuron 426 fired ~= useless [floppy ear] fired = very interesting! But one can just as easily imagine: neuron 183 fired ~= useless [young man fleeing] fired = very interesting! Are there conversations within Google Brain about the appropriate applications of this research you can share with us? Do you feel like this class of military technology is going to be developed one way or the other, so there's no use in worrying about speeding it along? 1: https://gizmodo.com/google-is-helping-the-pentagon-build-ai-for-drones-1823464533 https://gizmodo.com/google-is-helping-the-pentagon-build-ai-... 2: https://theintercept.com/2018/03/06/google-is-quietly-providing-ai-technology-for-drone-strike-targeting-project/ https://theintercept.com/2018/03/06/google-is-quietly-provid...

colah3 9y ago
Hi Jash! I focus my work on technical research directions I believe increase the probability of advances in AI being outstanding for all of humanity. I'm fortunate that Google provides me resources to do so, with broad autonomy to work on what I think is important. My main focus is on building techniques that allow humans to understand how neural networks make decisions, because I think that will be a powerful tool in making sure increasingly automated systems are aligned with society's values -- for example, being fair -- and are making decisions for reasons we endorse. I also spend a lot of time thinking about accident risk and safety. I believe this work is robustly good for the world. Of course, lots of concerns about AI can't be addressed by the kind of technical research I work on. I think technical research into interpretability, robustness, safety, fairness, etc... is necessary but not sufficient for the kind of world we want to see. There are also tough questions of public policy, international governance, and scientific ethics -- such as issues around the abuse of AI systems. On these kind of issues, I'd largely defer to people who think deeply about these topics, like Allan Dafoe (Oxford), Michael Page (OpenAI), and Tim Hwang (Ethics & Governance of AI Fund).

taliesinb 9y ago

Hello! I'm a fan of this latest publication, distill.pub, and your previous work. I have three questions: 1. Flexible interfaces based on the kinds of grammars you describe in this post feel like their time have come, as an important part of scientific communication, but also as a tool to create, test, and update hypotheses themselves. Entire experiments could be written in the language of these grammars, where transformations and operations directly produce the artifacts that support scientific conclusions: things like means and variances, distributions, statistical tests, scatter plots, and the like. What are your thoughts on that? Is that a direction in which Lucid is interested in going? 2. I'm interested in how to automate the creation of such linked interfaces, whether they be for convnets, language models, seq2seq type models, etc. The idea would be to have the user write a query in a DSL that explicitly connects specific interior activations (on a given axis/axes) with gradients and visualization techniques to produce a linked interface. I haven't yet played around with Lucid -- is Lucid already such a DSL for tensorflow? Or does it rather collect various instantiations of the general grammar that have proved useful? 3. What are the best examples you know of linked interfaces and grammar-like approaches to visualization inspiring important discoveries? Within deep learning, it feels like the slam-dunk example is OpenAI's sentiment neuron, although that was a fairly simple form of visualization. I'd be curious to find more examples, both inside and outside the field of deep learning.

cs702 9y ago

Hi Chris! -- I have four comments: First, as always, I have to say THANK YOU, to you, and to the other authors, and to your various helpers, for putting this together, with no obvious goal other than the public good. :-) Second, your idea of (a) taking all activation values in a trained DNN in response to a particular sample, (b) reshaping all these values into a single giant matrix, and (c) factorizing this giant matrix to identify "neuron groups" (i.e., some kind of low-rank approximation) that most closely explain the behavior of the DNN for each particular sample... is a brilliant idea. In hindsight, it seems like an obvious thing to do, but I don't think I've seen anyone else do it before. I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks. Third, I think this barely scratches the surface of the kinds of UI-driven "interpretation tools" that ultimately will be needed to enable (non-technical) human beings be able to associate or attribute DNN behavior to "salient ideas," including those salient ideas for which human beings currently lack descriptive terminology. This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future. Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it. For example, what if the number of "salient ideas" current AI systems can discover vastly exceeds the number of distinct salient ideas a human brain can distinguish and work with? What is your view or opinionated guess on this?

colah3 9y ago
> factorizing this giant matrix to identify "neuron groups" ... is a brilliant idea. As with many ideas in this article, Alex deserves all the credit. He's been doing that trick internally at Google for years. (In my experience, if Alex is excited about an idea, >50% odds you'll realize it's a super important idea 2 years later. :P ) I think there's I saw an instance of someone doing PCA on conv net activations to make heat maps. We should try to dig that up and cite it. > I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks. Absolutely! (I think that most of the ideas in our article are pretty general. :) ) > This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future. Thanks! We're super excited as well! > Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it... What is your view or opinionated guess on this? Oh my. That's a super interesting and deep question. Wild wild speculation ahead. Please take everything I say with a big grain of salt. My first comment is that a model can be hard to understand because it's too stupid, in addition to being hard to understand because it's exceeding us. I suspect that if one could have an "interpretability curve" of how easy models are to understand vs model performance -- it's kind of hard to make this sensical, because both variables are actually these really nuance high-dimensional things, but imagine you could -- it would go up as you approach human performance and actually peak after it, before eventually declining again. This intuition is driven by the thought that early superhuman performance for tasks humans are already good at is probably largely about having really crisp, much more statistically precise, versions of our abstractions. Those crisp versions of our abstractions are probably easier to understand than the confused ones. But, of course, that one dimensional view is a gross simplification. I suspect that something that will be important in future thought about this is "alien abstractions" vs "refined abstractions." By a "refined abstraction" I mean something like ears in GoogLeNet. You see, GoogLeNet has dozens of detectors for different kinds of ear -- a much richer vocabulary of ear types than I can articulate -- and knows a lot about how those should influence class probabilities. Although I can't understand the detailed nuance of each ear detector, I can get the general gist and verify that it has reasonable consequences. Conversely, by an "alien abstraction" I mean a feature that I don't have a corresponding idea for. These are much harder for us to deal with. Both "refined" and "alien" abstractions could give superhuman performance. We're in a much better state when refined ones dominate. For visual tasks that humans are already good at, I expect refined abstractions to dominate for a while. In other domains, I have a lot less confidence. I think there's a lot of nuance about where interpretability will easily scale vs have severe challenges is super subtle. I have a draft essay floating around on the topic. Hopefully I'll get it out there someday.

brylie 9y ago

I couldn't read the article on a mobile screen. The text rendered at about 3px high. Please consider using a mobile-friendly CSS design.

nmca 9y ago

Hey - I thought that this was interesting work, but just wanted to jump in and say thanks so much for the effort in reproducibility. This should be the new standard, but putting in the effort to do it now is improving the world at relatively little personal benefit. So thanks!