7 ms·

> Scholarship >Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainl

> Scholarship >Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. Then publish your damn results in a damn open journal, dammit.

mathetic 9y ago
That's not always an option. Your career very much depends on consistently publishing in "A" venues (conference/journal). So it depends on your subcommunity's opinion on open access. For example, programming language people as a community are super keen on it, so most top venues are becoming open access. If that's not the case for your field, it's basically academic suicide unless you already have an amazing reputation.
- traverseda 9y ago
  You see, I read that clause in particular as "The difference between research and tinkering is whether you belong to an organisation that's willing to spend 30k+ a year on journal access". And fuck that. If that's the attitude that academia is going to take, clearly we need to kill it off, tinker for a while, and start fresh. Maybe we can refresh some standards for academic writing while we're at it.
  - 52-6F-62 9y ago
    I've heard some prominent academics speak to the same end, or farther, in the rare access I've had. (Yet with displaying no less passion for their chosen subjects. Who would have guessed!)
    - neuromantik8086 9y ago
      Freeman Dyson is actually quite vocal about his opposition to the current structure of academia, so I'm not surprised that you'd be able to find other academics who were equally self-critical.
      - 52-6F-62 9y ago
        I probably shouldn't be so surprised— but the number of esteemed academics I've had the chance to encounter I can count on one hand. Though many seem to share that same opinion. The number of pretentious title-holders (even at the lowest minimum levels) I've met seems unending. I could get into that one... but I won't. Instead you've just reminded me of yet more subjects to stack on my reading list.
- togelius 9y ago
  This is all true. However, the large publishers (IEEE, ACM, Springer) these days all allow you to self-archive on your own webpage and typically also on ArXiv. So you can (and should) make your papers freely available. I have done that since the start of my career. It's just a matter of looking.
sloreti 9y ago
Also the video discusses previous work starting at 4:40, so I'm confused what OP is complaining about unless the MarI/O guy re-uploaded with this attribution. https://youtu.be/qv6UVOQ0F44?t=4m40s https://youtu.be/qv6UVOQ0F44?t=4m40s
- frankmcsherry 9y ago
  Obvs he's complaining that the video didn't cite his work. :) One of the differences between researchers and tinkerers unmentioned in the article: vanity.
- dyarosla 9y ago
  After seeing this, it completely discredits all of the 'scholarship' part of the author's post. Looks like the author should do his research first.
- togelius 9y ago
  He mentions NEAT, which is good, but not any of the work on playing Mario, much less any work on evolving neural nets to play Mario. There's quite a few papers on the topic, not just mine. Note that I don't say that SethBling is wrong. Seen as tinkering, this is perfectly fine, and he did a great job with the video. No hard feelings. I just use it as an example of how it is not research, because of the lack of scholarship.
- fwilliams 9y ago
  This is hardly a discussion of previous work. The author of the video states the paper he implemented. In any semi-decent peer reviewed venue, you would cite a wide variety of papers that solve a similar or related problem or introduce a concept related to the method in the paper. The related work section of a paper is one of the most important parts since it puts the research into context. By citing other work, the authors explain what has already been done, and what contribution their work makes. A related work section should also illustrate the downsides, limitations, and differences of other cited research. Limitations of other works are often poorly understood since very few people have had the time to evaluate them beyond the initial experiments done before first publication. Research is not simply about presenting new techniques but also understanding the trade-offs that arise when choosing different solutions to a problem.
Karrot_Kream 9y ago
> Then publish your damn results in a damn open journal, dammit. Discoverability is hard. I mean, forget academic papers, take a look at just the web, and how many years it took Google to create a search engine that made discoverability easy, and that exploited (originally) the graph structure of the web. Papers do cite other papers, but how do you check a paper's relevance? Moreover, papers often describe something very specific, a specific approach, and searching for that approach is a lot harder than looking for a page that is "Haskell tutorial". An open journal isn't the answer here (it is the answer for other things though imo) and I'm not sure why you're getting so angry.
- neuromantik8086 9y ago
  This is one of those things that makes me deeply skeptical of the most zealous folks within the open science / open access movements. The emphasis seems to be much more on just throwing your data / paper up on the web sans context and much less on properly adding metadata and documentation for how to perform an experiment or how data can be reused. I've probably said this before elsewhere, but Josh Carp's "The Secret Lives of fMRI Experiments" really underscores this point. There are two equally catastrophic issues in the social (and to an extent) natural sciences and solving one (openness) is only part of the puzzle when the other involves blatant sloppiness in bookkeeping, pure and simple. If science is to overcome its current setbacks and lots of confidence, it needs to emphasize rigor alongside transparency.
  - traverseda 9y ago
    All the metadata stuff is good, but I'm not in academia. I can't make use of it if it's not available. That's why I'm worried about the information being available first and foremost.
    - neuromantik8086 9y ago
      > I can't make use of it if it's not available You also can't make use of data if it doesn't have appropriate metadata indicating how it can be used. Metadata is more than just "good"- it's essential for actually being able to run an analysis. Take fMRI for example- someone could just throw a brain image up on the internet. In fact, one particularly uninformed individual (who possibly broke Virginia state law) did so here: https://github.com/dcunited001/mri-scans https://github.com/dcunited001/mri-scans However, the raw data is not useful without knowing what scan sequence was used, slice-timing order, etc. Some of this is encoded in the data itself (the example above uploaded DICOM, which has somewhat rich metadata, although by no means is it comprehensive enough), but other data that is necessary is not (Phillips in particular is somewhat infamous for doing non-standard things with their scanners that aren't recorded in the DICOM headers, and in general it's a Herculean task getting MRI vendors to agree on standards). You could try performing an analysis with contextless data, but you're likely just going to make really spotty conclusions. > I'm not in academia Why should that matter? The circumstances of data collection are also important for business intelligence data or financial modeling data. I'm certain that the lack of this also contributes to poorly formed conclusions that cause real hurt to businesses out there.
togelius 9y ago
Like basically all researchers in CS these days, I keep all my papers freely downloadable from my webpage. Easily findable via Google Scholar, or just a regular Google search, for anyone who cares to look.
- oldandtired 9y ago
  Yet, before today, I had never heard of you or your work. I download and read from a broad selection of papers and fields. Mayhaps, you need to do a bit more "youtubing" to get your stuff out there and more accessible. A couple of my current favourites on youtube is 3Blue1Brown and Mathologer. They are both entertaining as well as being informative and have set me on investigatory paths for other research in various fields.
  - AlotOfReading 9y ago
    Why should they be familiar to you? There are hundreds of thousands of academics around the world working in thousands of distinct fields. Promotion on YouTube can be useful, but not nearly so much as you're thinking.
    - oldandtired 9y ago
      My point is that even though he puts his stuff out there, if he is not heard of then it matters not.
fwilliams 9y ago
Most researchers in my field make all their publications freely available online after they have been submitted. In machine learning and computer vision, the default is to put your work on ArXiv immediately upon completion. There is a lot of openly available research depending on the field. I find most fields in computer science are good for this. In the case of the Mario example, I very much doubt that the author was not able to find other work because of closed journals. Research can seem non-transparent to non-researchers because when problems are new, they are often poorly understood. Academic papers discuss novel problems and contextualize them based on other cutting edge work. Reading and understanding these papers requires a lot of context and takes time. Research is a skill that takes years to learn. After some time has passed and we collectively gain a better understanding of a problem, academic papers may seem abstruse and overly complicated, but when these works were first published, this was the best way to understand them. For somebody looking for a recipe solution to a problem, an academic paper is likely not an the ideal place to look, which is why we write books, blog posts, etc... as we come to better understand a problem and its solutions.

mathetic 9y ago

That's not always an option. Your career very much depends on consistently publishing in "A" venues (conference/journal). So it depends on your subcommunity's opinion on open access. For example, programming language people as a community are super keen on it, so most top venues are becoming open access. If that's not the case for your field, it's basically academic suicide unless you already have an amazing reputation.

traverseda 9y ago
You see, I read that clause in particular as "The difference between research and tinkering is whether you belong to an organisation that's willing to spend 30k+ a year on journal access". And fuck that. If that's the attitude that academia is going to take, clearly we need to kill it off, tinker for a while, and start fresh. Maybe we can refresh some standards for academic writing while we're at it.
- 52-6F-62 9y ago
  I've heard some prominent academics speak to the same end, or farther, in the rare access I've had. (Yet with displaying no less passion for their chosen subjects. Who would have guessed!)
  - neuromantik8086 9y ago
    Freeman Dyson is actually quite vocal about his opposition to the current structure of academia, so I'm not surprised that you'd be able to find other academics who were equally self-critical.
    - 52-6F-62 9y ago
      I probably shouldn't be so surprised— but the number of esteemed academics I've had the chance to encounter I can count on one hand. Though many seem to share that same opinion. The number of pretentious title-holders (even at the lowest minimum levels) I've met seems unending. I could get into that one... but I won't. Instead you've just reminded me of yet more subjects to stack on my reading list.
togelius 9y ago
This is all true. However, the large publishers (IEEE, ACM, Springer) these days all allow you to self-archive on your own webpage and typically also on ArXiv. So you can (and should) make your papers freely available. I have done that since the start of my career. It's just a matter of looking.

sloreti 9y ago

Also the video discusses previous work starting at 4:40, so I'm confused what OP is complaining about unless the MarI/O guy re-uploaded with this attribution. https://youtu.be/qv6UVOQ0F44?t=4m40s https://youtu.be/qv6UVOQ0F44?t=4m40s

frankmcsherry 9y ago
Obvs he's complaining that the video didn't cite his work. :) One of the differences between researchers and tinkerers unmentioned in the article: vanity.
dyarosla 9y ago
After seeing this, it completely discredits all of the 'scholarship' part of the author's post. Looks like the author should do his research first.
togelius 9y ago
He mentions NEAT, which is good, but not any of the work on playing Mario, much less any work on evolving neural nets to play Mario. There's quite a few papers on the topic, not just mine. Note that I don't say that SethBling is wrong. Seen as tinkering, this is perfectly fine, and he did a great job with the video. No hard feelings. I just use it as an example of how it is not research, because of the lack of scholarship.
fwilliams 9y ago
This is hardly a discussion of previous work. The author of the video states the paper he implemented. In any semi-decent peer reviewed venue, you would cite a wide variety of papers that solve a similar or related problem or introduce a concept related to the method in the paper. The related work section of a paper is one of the most important parts since it puts the research into context. By citing other work, the authors explain what has already been done, and what contribution their work makes. A related work section should also illustrate the downsides, limitations, and differences of other cited research. Limitations of other works are often poorly understood since very few people have had the time to evaluate them beyond the initial experiments done before first publication. Research is not simply about presenting new techniques but also understanding the trade-offs that arise when choosing different solutions to a problem.

Karrot_Kream 9y ago

> Then publish your damn results in a damn open journal, dammit. Discoverability is hard. I mean, forget academic papers, take a look at just the web, and how many years it took Google to create a search engine that made discoverability easy, and that exploited (originally) the graph structure of the web. Papers do cite other papers, but how do you check a paper's relevance? Moreover, papers often describe something very specific, a specific approach, and searching for that approach is a lot harder than looking for a page that is "Haskell tutorial". An open journal isn't the answer here (it is the answer for other things though imo) and I'm not sure why you're getting so angry.

neuromantik8086 9y ago
This is one of those things that makes me deeply skeptical of the most zealous folks within the open science / open access movements. The emphasis seems to be much more on just throwing your data / paper up on the web sans context and much less on properly adding metadata and documentation for how to perform an experiment or how data can be reused. I've probably said this before elsewhere, but Josh Carp's "The Secret Lives of fMRI Experiments" really underscores this point. There are two equally catastrophic issues in the social (and to an extent) natural sciences and solving one (openness) is only part of the puzzle when the other involves blatant sloppiness in bookkeeping, pure and simple. If science is to overcome its current setbacks and lots of confidence, it needs to emphasize rigor alongside transparency.
- traverseda 9y ago
  All the metadata stuff is good, but I'm not in academia. I can't make use of it if it's not available. That's why I'm worried about the information being available first and foremost.
  - neuromantik8086 9y ago
    > I can't make use of it if it's not available You also can't make use of data if it doesn't have appropriate metadata indicating how it can be used. Metadata is more than just "good"- it's essential for actually being able to run an analysis. Take fMRI for example- someone could just throw a brain image up on the internet. In fact, one particularly uninformed individual (who possibly broke Virginia state law) did so here: https://github.com/dcunited001/mri-scans https://github.com/dcunited001/mri-scans However, the raw data is not useful without knowing what scan sequence was used, slice-timing order, etc. Some of this is encoded in the data itself (the example above uploaded DICOM, which has somewhat rich metadata, although by no means is it comprehensive enough), but other data that is necessary is not (Phillips in particular is somewhat infamous for doing non-standard things with their scanners that aren't recorded in the DICOM headers, and in general it's a Herculean task getting MRI vendors to agree on standards). You could try performing an analysis with contextless data, but you're likely just going to make really spotty conclusions. > I'm not in academia Why should that matter? The circumstances of data collection are also important for business intelligence data or financial modeling data. I'm certain that the lack of this also contributes to poorly formed conclusions that cause real hurt to businesses out there.

togelius 9y ago

Like basically all researchers in CS these days, I keep all my papers freely downloadable from my webpage. Easily findable via Google Scholar, or just a regular Google search, for anyone who cares to look.

oldandtired 9y ago
Yet, before today, I had never heard of you or your work. I download and read from a broad selection of papers and fields. Mayhaps, you need to do a bit more "youtubing" to get your stuff out there and more accessible. A couple of my current favourites on youtube is 3Blue1Brown and Mathologer. They are both entertaining as well as being informative and have set me on investigatory paths for other research in various fields.
- AlotOfReading 9y ago
  Why should they be familiar to you? There are hundreds of thousands of academics around the world working in thousands of distinct fields. Promotion on YouTube can be useful, but not nearly so much as you're thinking.
  - oldandtired 9y ago
    My point is that even though he puts his stuff out there, if he is not heard of then it matters not.

fwilliams 9y ago

Most researchers in my field make all their publications freely available online after they have been submitted. In machine learning and computer vision, the default is to put your work on ArXiv immediately upon completion. There is a lot of openly available research depending on the field. I find most fields in computer science are good for this. In the case of the Mario example, I very much doubt that the author was not able to find other work because of closed journals. Research can seem non-transparent to non-researchers because when problems are new, they are often poorly understood. Academic papers discuss novel problems and contextualize them based on other cutting edge work. Reading and understanding these papers requires a lot of context and takes time. Research is a skill that takes years to learn. After some time has passed and we collectively gain a better understanding of a problem, academic papers may seem abstruse and overly complicated, but when these works were first published, this was the best way to understand them. For somebody looking for a recipe solution to a problem, an academic paper is likely not an the ideal place to look, which is why we write books, blog posts, etc... as we come to better understand a problem and its solutions.