13 ms·

Visual Information Theory (2015)

543 points by less_penguiny 7y ago

tw1010 7y ago
This guy seems to have peaked in terms of creative output while he was on the thiel fellowship, and then abruptly stop once he was sucked up by google.
- tyingq 7y ago
  Based on a ~20 minute or less reading/reflection of this post, or something more?
  - tw1010 7y ago
    At least in terms of public output. He hasn't published anything in particular in the last few years, whereas the years when his income came from the fellowship (judging by blog output, his twitter, and his academic CV). Can't say what he's doing inside google of course, but it's a shame it's not ideas and knowledge that can be enjoyed by us, the lowly outsiders.
    - lonelappde 7y ago
      Please stop reposting the same false information.
- Voloskaya 7y ago
  Maybe you weren't aware he moved to https://distill.pub/ https://distill.pub/ ?
  - tw1010 7y ago
    Not trying to make a judgement about the lack of output or anything. I appreciate the work he did and the ideas he published. Mostly making a comment on the observation that outsider researchers being bought by google doesn't seem to produce the greatest effect on the open source idea-sphere. But he's free to do whatever he wants of course.
    - Voloskaya 7y ago
      Didn't think you were making a judgement, but I don't understand your comment about him abrubptly stopping, since he never stopped.
- kgraves 7y ago
  Agreed, very unfortunate and such a waste, extraordinary talent should go towards startups and open source software not the big FAANG companies.
- pm90 7y ago
  This actually happens to a lot of folks. I don’t necessarily blame them. It seems a bit unreasonable to expect a person to keep putting out good material forever. But I can only imagine that Google provides enough opportunities to express ones creative outputs internally to suppress the need to publish it externally...
- typo_hunter 7y ago
  What a toxic comment. Many people would consider DeepDream and Inceptionism to be the spark that ignited a lot of creative work in deep learning, which Christopher Olah wrote on while at Google AI.
  - kgraves 7y ago
    Disagree, I don't find this a "toxic" comment. The parent makes a point. Such potential in my opinion from these researchers/engineers and instead gets scooped up by the big tech companies, hoarding the talent such that nobody can compete with them. Happy for Chris's decision to be at the big G doing his research, but it seems every researcher/engineer is obsessed with being at the big tech companies being on their resume, which is such a shame.
    - seanmcdirmid 7y ago
      Working research for a big tech means you are much less subject to the “publish or perish” req of a university. It often isn’t a must have resume point, but it actually gives you a different kind of freedom.
      - kgraves 7y ago
        Given that he was awarded a Thiel fellowship, this gives him more freedom to research what he wants and make an impact than big tech and can literally go anywhere. But obviously chose Google.
        colah3 7y ago
        Hello! The Thiel Fellowship was a wonderful period for me, and gave me a lot of freedom to work on different topics. I'm very grateful for it. Unfortunately, the Fellowship only lasts for two years. Towards the end, it was actually super stressful. How could I support myself while continuing to work on what I thought was important? And if I wanted to live in the Bay Area, how could I event get immigration status to work and live there? Getting to work at Google Brain at this point let me continue to do what I was doing before -- many of my blog posts, including the one being discussed here, were written while I was at Brain.
- b3kart 7y ago
  It’s sad that people have less time to contribute to public knowledge when hired by big companies, but saying that someone “has peaked” is just mean. I am sure you’d find it easy to say no to Google money.
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
- rrherr 7y ago
  Chris Olah is first author on these papers in distill.pub: - https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ - https://distill.pub/2017/feature-visualization/ https://distill.pub/2017/feature-visualization/ - https://distill.pub/2016/augmented-rnns/ https://distill.pub/2016/augmented-rnns/ He explains his personal mission, doing research distillation to reduce research debt, in this essay: https://distill.pub/2017/research-debt/ https://distill.pub/2017/research-debt/
  - behnamoh 7y ago
    I wish more people were interested in research distillation. Recently I've been searching for intuitive explanations of different areas of math, and while there are some great blogs that do so, the lack of such materials is indeed obvious. I don't think academia is going to contribute to this mission as it's mostly governed by a make-it-look-ridiculously-difficult-so-people-won't-find-it-approachable-and-thus-give-you-much-credit-for-doing-impressive-work culture.
    - lonelappde 7y ago
      Your hostile attitude to the people putting the work in is disrespectful and inappropriate. It's like the people who make entitled demands of open source software authors. Academics don't get rewarded for confusing each other, they have to create research that their peers appreciate, understand, and value.
      - feral 7y ago
        That's naive. Academia isn't some perfect system with perfect incentives. There's plenty of incentive to dress something up as more complex and impressive and using more high powered machinary than needed, e.g. impressing reviewers, qualifying for a particular conference, convincing the funding body that your pet technique can actually be useful etc I don't think you'd find many academics that dispute this, though hopefully they'd think a lot of people are trying to do good research. It'd be like open source software if it was done as a hobby; a lot of academics get public funding.
- colah3 7y ago
  Hello! Thanks for pointing out that my CV is out of date. You're right that none of my work over the last two years is listed there, and I should probably fix that. It feels a bit awkward to chime in, but it feels a bit important because I think there's a way in which the narrative is a little off (although I'm very grateful for everyone's kind comments): Working at Google wasn't a trade off where I got financial support in exchange for not publishing. Rather, working at Google was amazing enabler, which supported me doing work I believed advanced important public interests for five years. That actually includes the blog posts being discussed here, which I wrote while an intern on Brain and received feedback from many researchers there. To be concrete, some of the things I did while at Google included: * Writing expository blog posts explaining important ideas in machine learning (eg. this blog post). * A five year thread of research on neural network interpretability. How humans can understand how neural networks make decisions? (eg. DeepDream, Feature Vis, Lucid, Building Blocks) * Cofoudning a scientific journal for machine learning that values exposition and allows interactive diagrams. * Working on early TensorFlow and writing some of the initial tutorials on it. * Articulating concrete safety concerns about modern machine learning systems, organizing a cross-instutional paper on it, and later representing Google on the Partnership on AI's Safety-Critical AI working group. * Designing and teaching the introductory course for the Google Brain/AI Residency. * Miscellaneous ML research, all published in open access venues. That's basically the majority of what I did at Google. I always felt empowered to work on things that I thought were important for the world, and in many cases felt like had leverage I wouldn't have had individually. I think that was exceptionally generous of Google, and also speaks a lot to the environment Jeff created for those of us on Brain. (I realize Google is a large company, and other people's experience may be very different.) I do think you're right that the number of short expository blog posts I wrote declined, especially after my first two years at Google. A big part of that is that I focused more on a small number of more ambitious projects (eg. Distill, Building Blocks). Another big part is that I started doing more non-individual work: teaching, mentoring, editing, debugging social issues. To be clear, I actually left Google last fall. I now work at OpenAI where I lead the new Clarity team, which works neural network interpretability -- basically, can we take a trained neural network and turn it into something like code that a human can understand? Leading a team means that I do even less individual research and writing. Sometimes I miss being an individual contributor, but I think it's the right call. I get to build an environment where others can focus on a kind of research I think is really important for the world and teach and support them.
chrisweekly 7y ago
Nice post: author has good communication / teaching style, and firm grasp of the material. The visuals help make some of the concepts more intuitive. Bookmarked.
- ThouYS 7y ago
  incredible guy, he really is the 10x everything type: https://colah.github.io/cv.pdf https://colah.github.io/cv.pdf
  - applecrazy 7y ago
    Wow, his internship host was Jeff Dean? I knew @colah was amazing (his neural net articles come to mind), but this is a whole other level of awesome.
    - Koshkin 7y ago
      > Wow WTH is Jeff Dean and WSIC.
- z2 7y ago
  On an unrelated tangent, the way the page formats when printing is one of the best I've seen. No weird navigation cruft, and it seems there's even a style defined that sets book-style margins, such that the left and right margins alternate in length.
  - colah3 7y ago
    Thanks! This particular article went through pretty extensive feedback with colleagues where I'd print the article out, share it, and get hand written feedback. This necessitated investing in print formatting. :)
- lonelappde 7y ago
  It's no accident. Olah is living a life and leading a movement dedicated to doing what you noticed. https://distill.pub/2017/research-debt/ https://distill.pub/2017/research-debt/
alberto_ol 7y ago
previous submission https://news.ycombinator.com/item?id=10388799 https://news.ycombinator.com/item?id=10388799
kensai 7y ago
The referenced 1948 paper was recently cited by Max Hodak at the presentation of Neuralink recently. Pretty amazing piece of work!
- glup 7y ago
  There's a followup on properties of English which is also good fun: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
Rainymood 7y ago
Love this article. It makes statistics enjoyable and accessible. Most of Olah's old stuff is also really good, especially the one on manifolds and neural networks [1] [1] https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
- m3at 7y ago
  You might like to check distill [1], it's a journal with a limited amount of content for now, but of very high quality, and C. Olah is one of the editor. [1] https://distill.pub/ https://distill.pub/
lonelappde 7y ago
The visual presentation in this article is very similar with how small children are taught the theorems of multiplication and distributivity and simple series sums like triangle numbers.
samch93 7y ago
Nice article. For those who are more interested in mosaic plots, statisticians have already done a lot of work on this issue. For R there are many nice solutions, e.g. the strucplot framework which allows to visualize complicated relationships between multiple qualitative variables (https://www.jstatsoft.org/article/view/v017i03 https://www.jstatsoft.org/article/view/v017i03).
bcheung 7y ago
Love how the probability distributions are presented. I wish those diagrams were in the material when I was first learning probability. Would have communicated the concepts so much faster and easier.
boazbarak 7y ago
Love this blog post! One minor nitpick: the event that it rains next week is probably rather correlated with the event that it rains this week (in particular it's correlated with the season), so I don't think this is a great example of independent variables. Maybe you can separate by distance: the event that you wear t-shirt vs the event that it rains in city Y vs the event that it rains in city Z.
meuk 7y ago
This visualization is also very useful when trying to understand Bayes' theorem.
171243 7y ago
bookmarked

543 points by less_penguiny 7y ago

This guy seems to have peaked in terms of creative output while he was on the thiel fellowship, and then abruptly stop once he was sucked up by google.

tyingq 7y ago
Based on a ~20 minute or less reading/reflection of this post, or something more?
- tw1010 7y ago
  At least in terms of public output. He hasn't published anything in particular in the last few years, whereas the years when his income came from the fellowship (judging by blog output, his twitter, and his academic CV). Can't say what he's doing inside google of course, but it's a shame it's not ideas and knowledge that can be enjoyed by us, the lowly outsiders.
  - lonelappde 7y ago
    Please stop reposting the same false information.
Voloskaya 7y ago
Maybe you weren't aware he moved to https://distill.pub/ https://distill.pub/ ?
- tw1010 7y ago
  Not trying to make a judgement about the lack of output or anything. I appreciate the work he did and the ideas he published. Mostly making a comment on the observation that outsider researchers being bought by google doesn't seem to produce the greatest effect on the open source idea-sphere. But he's free to do whatever he wants of course.
  - Voloskaya 7y ago
    Didn't think you were making a judgement, but I don't understand your comment about him abrubptly stopping, since he never stopped.
kgraves 7y ago
Agreed, very unfortunate and such a waste, extraordinary talent should go towards startups and open source software not the big FAANG companies.
pm90 7y ago
This actually happens to a lot of folks. I don’t necessarily blame them. It seems a bit unreasonable to expect a person to keep putting out good material forever. But I can only imagine that Google provides enough opportunities to express ones creative outputs internally to suppress the need to publish it externally...
typo_hunter 7y ago
What a toxic comment. Many people would consider DeepDream and Inceptionism to be the spark that ignited a lot of creative work in deep learning, which Christopher Olah wrote on while at Google AI.
- kgraves 7y ago
  Disagree, I don't find this a "toxic" comment. The parent makes a point. Such potential in my opinion from these researchers/engineers and instead gets scooped up by the big tech companies, hoarding the talent such that nobody can compete with them. Happy for Chris's decision to be at the big G doing his research, but it seems every researcher/engineer is obsessed with being at the big tech companies being on their resume, which is such a shame.
  - seanmcdirmid 7y ago
    Working research for a big tech means you are much less subject to the “publish or perish” req of a university. It often isn’t a must have resume point, but it actually gives you a different kind of freedom.
    - kgraves 7y ago
      Given that he was awarded a Thiel fellowship, this gives him more freedom to research what he wants and make an impact than big tech and can literally go anywhere. But obviously chose Google.
      - colah3 7y ago
        Hello! The Thiel Fellowship was a wonderful period for me, and gave me a lot of freedom to work on different topics. I'm very grateful for it. Unfortunately, the Fellowship only lasts for two years. Towards the end, it was actually super stressful. How could I support myself while continuing to work on what I thought was important? And if I wanted to live in the Bay Area, how could I event get immigration status to work and live there? Getting to work at Google Brain at this point let me continue to do what I was doing before -- many of my blog posts, including the one being discussed here, were written while I was at Brain.
b3kart 7y ago
It’s sad that people have less time to contribute to public knowledge when hired by big companies, but saying that someone “has peaked” is just mean. I am sure you’d find it easy to say no to Google money.
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
rrherr 7y ago
Chris Olah is first author on these papers in distill.pub: - https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ - https://distill.pub/2017/feature-visualization/ https://distill.pub/2017/feature-visualization/ - https://distill.pub/2016/augmented-rnns/ https://distill.pub/2016/augmented-rnns/ He explains his personal mission, doing research distillation to reduce research debt, in this essay: https://distill.pub/2017/research-debt/ https://distill.pub/2017/research-debt/
- behnamoh 7y ago
  I wish more people were interested in research distillation. Recently I've been searching for intuitive explanations of different areas of math, and while there are some great blogs that do so, the lack of such materials is indeed obvious. I don't think academia is going to contribute to this mission as it's mostly governed by a make-it-look-ridiculously-difficult-so-people-won't-find-it-approachable-and-thus-give-you-much-credit-for-doing-impressive-work culture.
  - lonelappde 7y ago
    Your hostile attitude to the people putting the work in is disrespectful and inappropriate. It's like the people who make entitled demands of open source software authors. Academics don't get rewarded for confusing each other, they have to create research that their peers appreciate, understand, and value.
    - feral 7y ago
      That's naive. Academia isn't some perfect system with perfect incentives. There's plenty of incentive to dress something up as more complex and impressive and using more high powered machinary than needed, e.g. impressing reviewers, qualifying for a particular conference, convincing the funding body that your pet technique can actually be useful etc I don't think you'd find many academics that dispute this, though hopefully they'd think a lot of people are trying to do good research. It'd be like open source software if it was done as a hobby; a lot of academics get public funding.
colah3 7y ago
Hello! Thanks for pointing out that my CV is out of date. You're right that none of my work over the last two years is listed there, and I should probably fix that. It feels a bit awkward to chime in, but it feels a bit important because I think there's a way in which the narrative is a little off (although I'm very grateful for everyone's kind comments): Working at Google wasn't a trade off where I got financial support in exchange for not publishing. Rather, working at Google was amazing enabler, which supported me doing work I believed advanced important public interests for five years. That actually includes the blog posts being discussed here, which I wrote while an intern on Brain and received feedback from many researchers there. To be concrete, some of the things I did while at Google included: * Writing expository blog posts explaining important ideas in machine learning (eg. this blog post). * A five year thread of research on neural network interpretability. How humans can understand how neural networks make decisions? (eg. DeepDream, Feature Vis, Lucid, Building Blocks) * Cofoudning a scientific journal for machine learning that values exposition and allows interactive diagrams. * Working on early TensorFlow and writing some of the initial tutorials on it. * Articulating concrete safety concerns about modern machine learning systems, organizing a cross-instutional paper on it, and later representing Google on the Partnership on AI's Safety-Critical AI working group. * Designing and teaching the introductory course for the Google Brain/AI Residency. * Miscellaneous ML research, all published in open access venues. That's basically the majority of what I did at Google. I always felt empowered to work on things that I thought were important for the world, and in many cases felt like had leverage I wouldn't have had individually. I think that was exceptionally generous of Google, and also speaks a lot to the environment Jeff created for those of us on Brain. (I realize Google is a large company, and other people's experience may be very different.) I do think you're right that the number of short expository blog posts I wrote declined, especially after my first two years at Google. A big part of that is that I focused more on a small number of more ambitious projects (eg. Distill, Building Blocks). Another big part is that I started doing more non-individual work: teaching, mentoring, editing, debugging social issues. To be clear, I actually left Google last fall. I now work at OpenAI where I lead the new Clarity team, which works neural network interpretability -- basically, can we take a trained neural network and turn it into something like code that a human can understand? Leading a team means that I do even less individual research and writing. Sometimes I miss being an individual contributor, but I think it's the right call. I get to build an environment where others can focus on a kind of research I think is really important for the world and teach and support them.

chrisweekly 7y ago

Nice post: author has good communication / teaching style, and firm grasp of the material. The visuals help make some of the concepts more intuitive. Bookmarked.

ThouYS 7y ago
incredible guy, he really is the 10x everything type: https://colah.github.io/cv.pdf https://colah.github.io/cv.pdf
- applecrazy 7y ago
  Wow, his internship host was Jeff Dean? I knew @colah was amazing (his neural net articles come to mind), but this is a whole other level of awesome.
  - Koshkin 7y ago
    > Wow WTH is Jeff Dean and WSIC.
z2 7y ago
On an unrelated tangent, the way the page formats when printing is one of the best I've seen. No weird navigation cruft, and it seems there's even a style defined that sets book-style margins, such that the left and right margins alternate in length.
- colah3 7y ago
  Thanks! This particular article went through pretty extensive feedback with colleagues where I'd print the article out, share it, and get hand written feedback. This necessitated investing in print formatting. :)
lonelappde 7y ago
It's no accident. Olah is living a life and leading a movement dedicated to doing what you noticed. https://distill.pub/2017/research-debt/ https://distill.pub/2017/research-debt/

alberto_ol 7y ago

previous submission https://news.ycombinator.com/item?id=10388799 https://news.ycombinator.com/item?id=10388799

kensai 7y ago

The referenced 1948 paper was recently cited by Max Hodak at the presentation of Neuralink recently. Pretty amazing piece of work!

glup 7y ago
There's a followup on properties of English which is also good fun: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf

Rainymood 7y ago

Love this article. It makes statistics enjoyable and accessible. Most of Olah's old stuff is also really good, especially the one on manifolds and neural networks [1] [1] https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

m3at 7y ago
You might like to check distill [1], it's a journal with a limited amount of content for now, but of very high quality, and C. Olah is one of the editor. [1] https://distill.pub/ https://distill.pub/

lonelappde 7y ago

The visual presentation in this article is very similar with how small children are taught the theorems of multiplication and distributivity and simple series sums like triangle numbers.

samch93 7y ago

Nice article. For those who are more interested in mosaic plots, statisticians have already done a lot of work on this issue. For R there are many nice solutions, e.g. the strucplot framework which allows to visualize complicated relationships between multiple qualitative variables (https://www.jstatsoft.org/article/view/v017i03 https://www.jstatsoft.org/article/view/v017i03).

bcheung 7y ago

Love how the probability distributions are presented. I wish those diagrams were in the material when I was first learning probability. Would have communicated the concepts so much faster and easier.

boazbarak 7y ago

Love this blog post! One minor nitpick: the event that it rains next week is probably rather correlated with the event that it rains this week (in particular it's correlated with the season), so I don't think this is a great example of independent variables. Maybe you can separate by distance: the event that you wear t-shirt vs the event that it rains in city Y vs the event that it rains in city Z.

meuk 7y ago

This visualization is also very useful when trying to understand Bayes' theorem.

171243 7y ago

bookmarked