27 ms·

The differences between tinkering and research (2016)

324 points by hardmaru 9y ago

ChuckMcM 9y ago
I like this differentiation. Early on in the 3d printer craze I would often say "A lot of people are building 3D printers but only a few people are engineering them." which was much the same argument. When you "engineer" a thing you do so understanding the constraints and design it to reliably within those constraints. When you just build something, as soon as it works once you're done.
- Xunxi 9y ago
  This reminds me of the quote from one of my favorite shows, MythBusters. “Remember kids, the only difference between Science and screwing around is writing it down.” One is a crude and improvised or temporary solution to a problem, designed to be more functional and timely than precise. Something done with little forethought, organization, planning or precision. Whereas the other is meticulously documented to note the choreographed nuances.
- Animats 9y ago
  Engineering has a specific meaning in that area. It means you have a design which can be built using standard techniques and will then work.
  - ChuckMcM 9y ago
    agreed as long as you also append 'to specifications under all acceptable conditions.' It was the latter bit that really challenged 3D printer makers. So many 3D printers would work one time and not the next because there were so many parameters that they didn't specifically account for or respond to.
    - Animats 9y ago
      Here's one that's very robust.[1] [1] https://www.youtube.com/watch?v=srVHzKsBguM https://www.youtube.com/watch?v=srVHzKsBguM
commandlinefan 9y ago
> It is easy to miss that someone tackled the same problem as you (or had the same idea as you) last year ... and hard to resist the temptation to go jump out the window when you find out that the thing that you've been trying to figure out for a year has already been figured out...
- jacobolus 9y ago
  There are independent benefits to struggling with something for a while by yourself before adopting someone else’s solution. For one thing, you will understand the problem and their solution a whole lot better.
  - colmvp 9y ago
    Pretty much. There are for example studies that show that people who attempt a problem before finding a solution learn a lot more than people who just look at the solution.
joe_the_user 9y ago
A really excellent summary of the distinctions. One thing I'd note is that before that Internet era, the scholarship needed to discover who had done X previously took a really significant amount of time. The Internet has changed that a lot, at the same that it has tempted people into tinkering and overall wingnuttery. I remember in 80s, having professor who functioned as little more than a search-engine, tell him an idea and, after a couple of puffs of his pipe, he'd give you a list of people who'd worked on it. He was marvelous despite apparently never having done much more than providing this kind of information. Another thing to note that scholarship implies a "community of researchers". If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. The article kind of says this but also consider that the people who have worked on this stuff exist, they have publications, conferences, meetups, etc and you can go to these or look at these and find-out whether or not your idea was considered and rejected, think about what's different between your idea and the idea the "field" rejected and so-forth.
- jamesrcole 9y ago
  > If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly: - each of the earlier people were approaching that problem from a specific angle, and new people might be approaching it from an untried angle - the earlier people might have had limited time to really investigate the matter, so there may be unturned stones - advances since the earlier work may give newer researchers a leg up. Note that all of these apply even if (as is likely) the earlier researchers were very smart.
  - joe_the_user 9y ago
    >>If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. > At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly Well, if you use such an approach of considering previous work, you will have the tools to show you why your work shouldn't be dismissed (whether you are believed or not is a different matter). If you aren't looking at earlier work, not only do people have good ammunition for dismissing your position but it really is hard to be sure you haven't just repeated something.
jacobolus 9y ago
There are also some unfortunate features of academic research. A big one is that the product the researcher cares about is papers and citations, not working software per se. Papers you only write once. You can cherry-pick examples, work around bugs, do some facile analysis, publish the paper and then move on never to look at the code again. This means that code and data is typically hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use (or often not publicly inspectable at all). Often the folks doing the grunt work are graduate students, who leave after a few years, leaving a new batch of graduate students to deal with the code without much guidance, so it tends to accrete features but seldom get any refactoring love. Nobody is paying for bug fixes or software maintenance or documentation. Someone coming from industry (or even just a tinkerer) who wants to use the same idea in a production environment often realizes halfway through reimplementation that it isn’t actually better than existing alternatives, and that the analysis/benchmarks in the paper were limited or misleading for a general context. Sorting the wheat from the chaff can be pretty demanding, ideas can sometimes be obfuscated by abstruse mathematical formalism, and academic researchers don’t typically go out of their way to make their output usable for non-academics.
- 3pt14159 9y ago
  True, but sometimes you commercialize something with some insights you bring from industry that never would have occurred to the original academic that made the piece in the first place. Basically what I and another guy did in 2012. We were stupid though, took the first real acquisition offer. That's the problem with being in your twenties you want money ASAP. "$1m today or $10m three years from now? Take the $1m, we'll figure out how to get the other millions later." I think it's another reason why the rich keep getting richer. Muuuuuch easier to turn down $1m when daddy is worth $50m.
- joe_the_user 9y ago
  Well, the thing with academic research is that it involves a "community of researcher" whose activities involve "evolving an idea". And in such a situation, the result is primarily expressed by its text/human-language representation while code is secondary to this. While this is problematic to verifying the results, it should be noted that code as a result has significant problems. Code doesn't have a long history of being read, interrogated and modified while human-language does. As the article effectively says, don't underestimate the significant value of putting a result in historical context. A human-language description can give a clue whether X has been done before, perfectly functioning code might not. This question touches on "programming literacy", which in the extreme asks, "how can computer programs also search as _documents_ in the fashion scientific papers have historically served as documents." So far, I don't think any answer has really appeared. Perhaps that problem itself is "AI complete", present either a terrible or green pastures for research.
snarfy 9y ago
"The difference between screwing around and science is writing it down." - tpai
ggm 9y ago
As a self-defined tinkerer, I have tried the transition to a peer-review recognised researcher. It is very hard. Science is a very harsh mistress, and what you want your work to be saying is quite distinct from strong tests into what it actually says. The game-like qualities of surviving peer-review are just that: a set of pretty arbitrary hurdles to cross. If you work in a narrow enough field, your set of blind review peers is tiny and their critique, even if masked, can seem very personal at times. I don't like that much. I don't overall judge my attempt a success. I think this is an endevour best started early. Late stage career, trying to demonstrate you understand the rules of science, is hard.
- GuiA 9y ago
  The best way to get your work accepted in a peer reviewed venue (journal, conference, etc) is to voraciously consume everything that venue has published before, and make your work match the style observed + copiously quote works from that venue. Obviously the intrinsic quality of your work matters. But you’ll more often see work of marginal value that properly follows the form than work of great value that does not follow the form. In fact, in my field (academic HCI), I have never seen the latter, yet see plenty of the former at every conference I attend.
  - ThrustVectoring 9y ago
    >copiously quote works from that venue This is more important than you'd initially think. Incentives matter, and professors are rewarded for publishing papers that get cited by other papers. So if you cite your reviewer's paper, it's in your reviewer's best interest to get your paper through review. Hence, quote a ton of works from that venue and hope to get lucky like that.
  - f1notformula1 9y ago
    Replying here because this matched my train of thought well. I'm actually really curious to know if anyone has first-hand knowledge of a tinkerer (by this blog's definition) getting their work published. As the GP said, understanding the rules is one thing, but demonstrating their understanding is another. And as you said, matching the style seems to be more important than the value of work that is demonstrated. I see the value in matching terms, jargon, style etc just so the reviewers can standardize their thought processes. But as someone who's been a tinkerer for ages now it's hard to change styles for no immediate benefit. Maybe some examples will inspire me to try :)
hmwhy 9y ago
I tend to disagree with those distinctions and I'm inclined to think the difference is simply that, to a "researcher", doing things is a job; and to a "tinkerer", doing things is a joy. ## Scholarship To begin with, there are many "tinkerers" out there who practise good scholarship and at acknowledge prior art, as well as provide references. Anyone who is courteous and has the awareness that the world doesn't revolve around her would honestly document these things even if she is a "tinkerer". And just as there are "tinkerers" who practise good scholarship, there are "researchers" who practise bad scholarship as has been pointed out by @jacobolus. ## Testing I'm not sure what makes the author thinks that rigorous testing is only a "researcher" quality. The article just insulted all the "tinkerers" out there by saying: > "Here's another big thing. A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens." A "tinkerer" doesn't just make something and puts it out there. A serious "tinkerer" would not want to put something that doesn't work out there—it's embarrassing and it's just, wrong. To make something (reasonably complex) that works usually involves a lot of testing, even if the goal isn't to publish a scientific paper it would still involve a lot of testing. ## Goals This bit really lost me. It sounds like a bunch of self-contradictory, "researcher"-glorifying statements one after another. > Usually, goals in research are not just goals, but ambitious goals. The reason we don't know what the results of a research project will be is that the project is ambitious; no-one (as far we know) has attempted what we do before so our best guesses at what will happen are just that: guesses. A "tinkerer" can have ambitious goals. A tinkerer may do something that nobody has attempted before and, naturally, in that case she would just have to make guesses. If, hypothetically, I set a goal to investigate the possibility of using CSS to make high-quality 3D games that can run at 60 FPS in web browsers, does that make me a "tinkerer" or a "researcher"? What if, in the process of doing so, I practise good scholarship, document everything carefully in a well-organised format, but just simply have no interest of publishing it in a scientific journal—does that make me a "tinkerer" or a "researcher"? On this: > However, if you read a scientific paper those are usually not the stated reasons for embarking on the research work presented in the paper. Usually, the work is said to be motivated by some scientific problem (e.g. optimizing real-value vectors in high-dimensional spaces, identifying faces in a crowd, generating fun game levels for Super Mario Bros). And that is often the truth, or at least part of the truth, from a certain angle. The stated reasons in scientific paper is usually after written after the research has been carried out and they are designed to tell a reader why the works is important. In a space where everyone is struggling to keep up with publishing so that they can keep their job, who in the right mind would begin a paper with "our research group embarked on [insert amazing thing] because it seemed fun to do so and has never been done before"? Actually, I would love to read papers like that, because most of the time you know it's the usual standard crap when academics start a paper by stating a problem—it probably means that they haven't solved the problem and are not even close. Also, I would like to draw your attention to the last, contradictory sentence. It seems that the author isn't quite sure either. ## Persistence If a someone spends 10 years modifying and perfecting something that has never been attempted before, but does not care for its scientific value, does that make her a "tinkerer" or a "researcher"? Bottom line, anyone who appreciates, and wants to do, good work will naturally do all of those things above—"tinkerer" or not. To throw away sentences like: > Probably the most importance difference between tinkering and research is scholarship. > A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens. > While tinkering can be (and often is) done for the hell of it, research is meant to have some kind of goal. > Tinkerers are content to release something and then forget about it. Researchers carry out sustained efforts over a long time, where individual experiments and papers are part of the puzzle. ... is just simply conceited.
Jedi72 9y ago
I wonder if the author would classify the Wright brothers as 'tinkerers'. Overall I find the tone pretty insulting, nobody believes they are the bees-knees quite like academics.
mmilano 9y ago
The word research is pretty vague in comparison to the context the author interprets its meaning. He over-reaches on his explanation of what a researcher is, and leaves too much of a gap between researcher and tinkerer.
- hatmatrix 9y ago
  Maybe the distinction he is trying to make is between academic work and non-academic work (hence the emphasis on scholarship). Non-academics can do research but what makes it academic is its placement in the broader context of civilized endeavors.
traverseda 9y ago
> Scholarship >Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. Then publish your damn results in a damn open journal, dammit.
- mathetic 9y ago
  That's not always an option. Your career very much depends on consistently publishing in "A" venues (conference/journal). So it depends on your subcommunity's opinion on open access. For example, programming language people as a community are super keen on it, so most top venues are becoming open access. If that's not the case for your field, it's basically academic suicide unless you already have an amazing reputation.
  - traverseda 9y ago
    You see, I read that clause in particular as "The difference between research and tinkering is whether you belong to an organisation that's willing to spend 30k+ a year on journal access". And fuck that. If that's the attitude that academia is going to take, clearly we need to kill it off, tinker for a while, and start fresh. Maybe we can refresh some standards for academic writing while we're at it.
    - 52-6F-62 9y ago
      I've heard some prominent academics speak to the same end, or farther, in the rare access I've had. (Yet with displaying no less passion for their chosen subjects. Who would have guessed!)
      - neuromantik8086 9y ago
        Freeman Dyson is actually quite vocal about his opposition to the current structure of academia, so I'm not surprised that you'd be able to find other academics who were equally self-critical.
1auralynn 9y ago
As someone who spends a fair amount of time with software developed by researchers (mostly biochemistry visualization software), that for all the "research into what has been done", I think there is a huge amount of effort being duplicated all over the place. In my experience this is due to the relatively short timelines of grants, the fact that you have to spend most of your grant money demonstrating that what you created has some kind of scientific impact vs actually implementing the thing, and, as others have pointed out, the fact that a lot of the core work is done by diamond-in-the-rough superstar grad students who will move on after 2-4 years.
brownbat 9y ago
> First of all, it's not about whether you work at a university and have a PhD. People can do excellent research without a PhD, and certainly not everything that a PhD-holder does deserves to called research. I appreciate this dismissed as a straw man, but there are definitely online communities (and individuals I know) who reflexively discount comments from people outside academia. I understand the instinct, it's a reasonable immune response when everyone thinks they're an expert because they read a blog post or a pop sci best seller. Sometimes, though, the antibodies kill perfectly healthy tissue too, like valid contributions from bright generalists.
- the_cat_kittles 9y ago
  another factor is that you are very defensive if you spent a bunch of time getting a phd and someone didnt and is still able to do research as well as you. and im sure its not very common, since getting good at something requires lots of time and energy. but i bet it does sometimes, and thats probably one of the reasons why some of the healthy tissue is killed
  - DataWorker 9y ago
    And they have families to feed and their income depends on thier PhD being enough of a barrier to entry that they can pay thier bills. So much human progress has been lost because the economic model is broken. We seemed to make more progress when science was restricted to a smaller group of practitioners, many of whom were wealthy enough that they could be guided by the science rather than the funding.
    - newsbinator 9y ago
      Also in the early days there was more low-hanging fruit. You could make scientific discoveries by locking yourself in a room for a week and thinking deeply about something you read recently. Then come out and test it with items from your kitchen. Now it's harder to make breakthroughs without specialized equipment, and the area of interest has narrowed to a tiny sliver.
togelius 9y ago
I'm the author, and I'm somewhat amused by the negative tone of many of the comments. Seems there are plenty of people with resentments towards research here ;)
- strken 9y ago
  I think it's because it treats hackers as deficient academics who can't be bothered to cite their sources properly or do any research, while there's an equal and opposite argument that academics are deficient hackers who can't be bothered to put their code on github or write any documentation. In reality, neither is really deficient, they're just aiming at different audiences which need different things.
  - togelius 9y ago
    Oh yes, I'm a terrible hacker, I rarely write code anymore. An unfortunate byproduct of academia is that you get "promoted out of the job", and all the hands-on work is done by your lab members. I didn't mean to imply that tinkering was inferior to research - the whole premise was just to tease out how they're different, with different audiences, as you say. Interestingly, the discussion here has been dominated by people who think that I look down on them. People who've discussed it in other fora have not read the post that way.
    - deepnet 9y ago
      Togelius I believe you are in error in Seth Bling's case. The Show More links under his youtube MarI/O video link directly to the Stanley Miikkulainen NEAT paper. and he links the wikipedia page on Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution which cites two [1][2] of your papers among others : yet you say : " it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. " a bit harsh perhaps ? At least mentioning the foundational NEAT paper is not a complete lack of scholarship. I feel you have overlooked that Seth does try and provide good citations to his audience and it would kind if you mentioned him by name in your article rather than " some guy " He did inspire other twitch streamers to experiment with neuroevolution and neural nets and benchmark many SNES games.[3] although downstream this follow up does goes uncited by yourself. [1] "Neuroevolution in Games: State of the Art and Open Challenges" https://arxiv.org/pdf/1410.7326v3.pdf https://arxiv.org/pdf/1410.7326v3.pdf [2] "Countering poisonous inputs with memetic neuroevolution" (PDF)https://www.academia.edu/download/30945872/poison.pdf https://www.academia.edu/download/30945872/poison.pdf [3] Mario Bros mari/o https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat https://www.youtube.com/watch?v=bRxUQNFxAWc https://www.youtube.com/watch?v=bRxUQNFxAWc Mari/o kart winterbunny https://clips.twitch.tv/HilariousPolishedZucchiniHassanChop https://clips.twitch.tv/HilariousPolishedZucchiniHassanChop - mario kart RNN mariflow
thethirdone 9y ago
> Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. I think its important to note that that he does reference NEAT (and the paper it comes from) and the emulator he used. This is building on what others have already done. While the 2009 paper from the post author would definitely be worth note, I feel "atrocious" is a bit harsh.
cornelis 9y ago
Thank you OP for your very insightful blog post. You have been better than any other academic at articulating exactly what happens when I (or one of my colleagues) interact with the academic world and you made me realize that the academic world has a set of values that might be good but which I should not even attempt to incorporate in my own professional life. I have come to my personal conclusion that there is a difference between 'science' and 'the academic world' in much the same way as there is a difference between 'religion' and 'the church'. One can be very religious without going to the church and doing all the rites that are required by the church. In much the same way, a 'tinkerer' (or 'heathen') can be scientific without being academic. I believe that wiki summarizes it quite well for me: "Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe". There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing. I want to solve real world problems and the way I want to do that is in a structural and repeatable approach. Science helps me doing this. Of course I'm not living in a vacuum, so I will look up as much as I can about the subject at hand as is reasonable within the available amount of time. Interestingly enough, I almost never end up with a research paper, but almost always with blog posts, books, tutorials et cetera. Products don't sell themselves and neither do research papers apparently. I think there is something seriously wrong if that is not perceived as a crisis in the academic world.
- iagovar 9y ago
  Research papers are typically about very specific topics. It would be strange to end up reading one on the Internet unless you already know the field keywords and use google scholar or any other science-oriented browser.
  - jacobolus 9y ago
    Indeed, google scholar and similar citation graphs are amazing. It makes tracking down vaguely relevant papers much easier than in the past. I can skim a few dozen papers one day, and then more closely inspect the handful that seem most relevant or interesting. I wish that google scholar showed bidirectional links though. I suspect they might have concluded that the list of citations included in a particular paper is part of the content of the paper (not just metadata) so that showing it might infringe copyright.
stared 9y ago
Some time ago I wrote about academia vs industry in the context of data science: In academia, you are allowed to cherry-pick an artificial problem and work on it for 2 years. The result needs to be novel, and you need to research previous and similar solutions. The solution needs to be perfect, even if not on time. In industry, you should solve a given problem end-to-end. Things need to work, and there is little difference if it is based on an academic paper, usage of an existing library, your own code or an impromptu hack. The solution needs to be on time, even if just good enough and based on shady and poorly understood assumptions. So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science. (from http://p.migdal.pl/2016/03/15/data-science-intro-for-math-phys-background.html#priorities http://p.migdal.pl/2016/03/15/data-science-intro-for-math-ph...)
- denzil_correa 9y ago
  > So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science. Science in a nutshell is to take a hypothesis, test it rigorously and validate/invalidate the hypothesis based on the results. The results of this experiment could be extremely practical. Data science that follows the scientific experimental design, does science. The one that doesn't is equivalent to reproduction of past experimental results OR putting a blindfold on your eyes and throwing darts around until it hits the bull's eye.
  - stared 9y ago
    I disagree with this common, yet faulty, explanation. (IMHO coming from people who have never worked in science and for whom everything R&D looks like "science!".) Building engines or cars (involving a lot of tests!) is considered engineering. And as any serious engineering it's not purely random trails&errors. A/B tests are not a sufficient criterion to make something science. And data science (in business) is focused on solving practical problems (which may or may not be research) rather than open-ended research (which may or may not be practical). Source: I worked in academia. I now work in data science. You?
Radim 9y ago
Funny thing: do your eyes light up seeing a "scholarly academic paper", or a "deeply tinkering blog"? Which one have we trained our brains to react more strongly to, what is more exciting to you? What do you feel you learn more out of, have more confidence to choose as a starting point for further activities? OP's distinction between tinkering and research seems wishful thinking at best. There are many grievances people have with research (and it's easy to think of examples), but I find most tend to fall under: * (unaddressed) RISK: poor result analysis, cherry-picked examples; glossing over edge cases and failure modes; focus on novelty and "academic writing" at the cost of repeatability and conceptual clarity. The Mummy Effect of research that looks great but crumbles to dust when you actually try to touch it: https://rare-technologies.com/mummy-effect-bridging-gap-between-academia-industry/ https://rare-technologies.com/mummy-effect-bridging-gap-betw... * (missing) OWNERSHIP: cadence dictated by grant cycles rather than one's intrinsic motivation to solve a problem; slow or missing feedback cycles; no dog-fooding => poor code and documentation; few incentives to make it easier for others to pick up or validate the results In fact, both of these seem better addressed by tinkerers than by institutionalized research. No wonder people react more positively to "tinkerers" and "hackers"! Academia has a lot of positive inertia (and taxes) going for it, but all goodwill has its limits.
deepnet 9y ago
OP, Togelius, is quite wrong in 1 point at least. tl;dr Seth Bling directly references NEAT and his links[Show More [1]] link to 2 Togelius papers[13][14] among many others ! To wit, in OP, Togelius mentions Seth Bling's MarI/O[1] as lacking scholarship because he didn't mention Togelius: " seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. " Rather disdaignfully (sadly/ ironically/ poigniantly) Togelius refers to Seth not by name but only as " some guy ". Yet Seth Bling[2] in his short video, mentions and explains NEAT and directly links Ken Stanley's and Risto Miikkulainen's foundational NEAT paper[10] at U.texas which Togelius builds on [hint: click SHOW MORE under Seth's video] And much more, i.e.: ---- the innovative toolchain Seth uses: Bizhawk to script SNES emulators in Lua [12]! ---- Seth links the wikipedia page on Neuroevolution[11] which directly references 2 of Togelius's own papers[13][14] Seth's [Show More] Links on the MarI/O video[1] are very very comprehensive and (IMHO) indeed a great example of scholarship ! Togelius is a top flight scholar, and publishes in the field of AI game players and level generators[15] so I hope he will welcome some random tinkerer pointing out his error. [full disclosure: I am a Togelius fan] To give Togelius his due he does at least link to Seth's MarI/O video. Perhaps Togelius failed to click / read the Show More links ? The charitable benefit of doubt principle is accidental oversight. Seth Bling is primarily a Twitch Streamer[2] and Speed Runner[3] who often popularises complex programming[4], math and academic ideas[5] to a primarily young audience using Mario and Minecraft[6] and takes requests via his subreddit[7] Seth's very popular Youtube video MarI/O which uses NEAT (Neuro Evolution of Augmenting Topologies] to learn and play Super Mario World, has led to a huge number of streamers on twitch playing other MarI/O levels[16], using RNNs, LSTMs, and other SMW levels and SNES games, like RNN multiplayer x4 Mario Karts[17]. The OP article is great, academic rigor and bibliographic scholarship might sometimes differentiate the full time university scholar from the amateur researcher, enthusiast or tinkerer. Seth and his many followers may not be 'scholars' as in academic research but it is an amateur community of mostly youngsters doing their own research by having fun and sharing ideas, mostly undocumented outside their own transient twitch streams. Compare this to the well respected amateur astronomy community and perhaps a responsible academic approach is to try to reach out and tap in rather than adding to the distance. By his own OP standards of Togelius is lacking in that hasn't discovered/ mentioned the work done by the Twitch/ Bizhawk community - it is not entirely undocumented [16]. Togelius your AI Mario competition[8] is legendary in academic AI circles but not necessarily widely known by youngsters playing Mario ( and to be fair AFAIK you don't publish in their forums like twitch) although the code Infinite Mario[9] you base your work on is by Marcus Persson[10]. Though Persson is better known as Notch [10] ( author of Minecraft ) who is more famous than perhaps even Alan Turing , Yann le Cun, Jurgen Schmidhuber, or Geoff Hinton to the under 16 crowd. Perhaps Togelius could mashup some of his work and do a crossover episode with Seth or relaunch the Infinite Mario competition[8] / benchmarks to this new younger crowd evolving Neural Topologies to play Mario games - or even help document some of their fun. As public servants funded by tax dollars it is perhaps the responsibility of the academic to reach out rather than blaming fun loving amateurs popularising and doing research / 'tinkering' for not putting in the due diligence of scrupulously mentioning every source. [IMHO] I agree amateurs don't always do full and proper scholarship and links are food for hungry minds - but they aren't publicly funded researchers but amateurs working for love of the subject in their spare time, for free, or for streaming donations and youtube hits, using transitory, live media and whose audience must be constantly engaged and may have shorter attention spans than the average academic. [full disclosure I am an amatuer neural net nerd] [1]Seth Bling's MarI/O https://www.youtube.com/watch?v=qv6UVOQ0F44 https://www.youtube.com/watch?v=qv6UVOQ0F44 [2]https://www.twitch.tv/sethbling https://www.twitch.tv/sethbling [3]https://www.youtube.com/watch?v=-spFoon7klA https://www.youtube.com/watch?v=-spFoon7klA sub 1 min SMW via credits warp [4]Bling's arranging shells in SMW to human inject Flappy Bird code into SNES https://www.twitch.tv/videos/57032858 https://www.twitch.tv/videos/57032858 [4]Bling's Atari 2600 emulator built in Minecraft https://www.youtube.com/watch?v=5nViIUfDMJg https://www.youtube.com/watch?v=5nViIUfDMJg [5]Seth Bling's RNN Mario Cart race https://www.youtube.com/watch?v=Ipi40cb_RsI https://www.youtube.com/watch?v=Ipi40cb_RsI [6]Bling's Minecraft Redstone tutorials https://www.youtube.com/watch?v=DzSpuMDtyUU&list=PL2Qvl4gaBge1kABr3aBCrMSb94HFLIjbn https://www.youtube.com/watch?v=DzSpuMDtyUU&list=PL2Qvl4gaBg... [7]https://sethbling.reddit.com https://sethbling.reddit.com [8]Togelius 2009 Infinite Mario AI competition http://julian.togelius.com/mariocompetition2009/ http://julian.togelius.com/mariocompetition2009/ [9]https://web.archive.org/web/20080423023424/http://www.mojang.com/notch/mario/ https://web.archive.org/web/20080423023424/http://www.mojang... [10]Stanley Miikkulainen NEAT paper http://nn.cs.utexas.edu/?stanley:ec02 http://nn.cs.utexas.edu/?stanley:ec02 [11] https://en.wikipedia.org/wiki/Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution [12] Scripting Lua in Bizhawk Emulator http://tasvideos.org/Bizhawk/LuaFunctions.html http://tasvideos.org/Bizhawk/LuaFunctions.html [13] Risi, Sebastian; Togelius, Julian (2017). "Neuroevolution in Games: State of the Art and Open Challenges" (PDF). IEEE Transactions on Computational Intelligence and AI in Games.https://arxiv.org/pdf/1410.7326v3.pdf https://arxiv.org/pdf/1410.7326v3.pdf [14] Togelius, Julian; Schaul, Tom; Schmidhuber, Jurgen; Gomez, Faustino (2008), "Countering poisonous inputs with memetic neuroevolution" (PDF), Parallel Problem Solving from Nature https://www.academia.edu/download/30945872/poison.pdf https://www.academia.edu/download/30945872/poison.pdf [15] Togelius' Game AI book http://gameaibook.org/ http://gameaibook.org/ [16] https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat [16] https://thenextweb.com/artificial-intelligence/2018/01/03/this-live-stream-of-ai-learning-to-play-super-mario-bros-is-awesome/ https://thenextweb.com/artificial-intelligence/2018/01/03/th... [17] https://www.youtube.com/watch?v=S9Y_I9vY8Qw https://www.youtube.com/watch?v=S9Y_I9vY8Qw
isthatart 9y ago
I'm an academic researcher. I disagree with most of the article. The main distinction I see between "research" and "tinkering" is not what the author calls "scholarship". From personal experience the only serious distinction is that "researchers" tend to have a very long time attention span compared with "tinkerers". As for the rest, the goal of academic research, lately, seems to be to produce as many as possible units for the benefit of the publishing industry. In the case of very successful articles, you can fit all the readers in a (small) bus. Most of the articles are badly written, because they follow old habits from the age when printing was expensive. Citations are done in a very selective way. Not so long ago there was no need to give the source of an "unpublished" article, "preprint" was enough. Tinkerers work is almost never cited, even if it is read and used. Blogs? are you kidding me? that's not "scholarship". Heck, even arXiv still seems exotic in some fields, despite the fact that it is a good, standard communication mean for the researchers in other fields. Finally, peer review. How does it work? Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could.This passes for validation. Magic. In conclusion I believe the future of academic research comes from "tinkering" with a long time attention span. The dissemination is technically trivial. Peer review will be supplemented by more rigorous validation (although there is no absolute solution here). The same kind of validation will be applied to tinkerers results.
- throwawayjava 9y ago
  > The main distinction I see between "research" and "tinkering" is not what the author calls "scholarship". From personal experience the only serious distinction is that "researchers" tend to have a very long time attention span compared with "tinkerers". Many aspects of "scholarship" become inevitable when "tinkering over a long timeframe". You build up a knowledge base about the problem and/or methodology you're tinkering with, you eventually form a community around your tinkering, etc. Lots and lots of tinkerers gratify themselves without contributing any modicum of knowledge to the world. But the most successful tinkerers are almost always effectively scholars in the sense of this post. In the sense that they're familiar with a body of knowledge/the people producing that knowledge. And they contribute genuinely new ways of doing things, or new ideas of things to do, or excellent executions on existing ideas that achieve the aspirations of previously articulated ideas. E.g., Linus is certainly a successful OS scholar in the sense of this post. > In the case of very successful articles, you can fit all the readers in a (small) bus This is off by at least an order of magnitude for any even remotely reasonable definition of "successful". > Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could First, the author explicitly kinda-sorta agrees with you here (...Third, it's not really the publication...), so it's pretty clear that publication venue != scholarship. As an aside, peer review != validation!!! This is something that every young Ph.D. should explicitly learn early on.
sigi45 9y ago
Whenever i read a paper, i do have troubles understanding the fine details. That one math formular which is nowhere explained? Yeah good... I often enough, get the idea and details like results but thats it. Those papers are not tutorials and they are not written / done to be reproduced and easy understandable. They are, in my opinion, written for other science people who spend there work time doing something similiar or need to solve the problem what the paper solves and rebuild those results in time consuming work. Probably still better than having no paper but still way more work than just using it. i do read "the morning paper" (https://blog.acolyer.org/ https://blog.acolyer.org/) and he is really good in reevaluating papers. I'm very curious how much time he spends reading and analysing them.
cosmic_ape 9y ago
The term research became extremely diluted now. It attempts to put very different things under one roof, and so is not very useful. Near the end of that blog post you find: >>Reading the book, I felt that most of my research is not science, barely engineering and absolutely not mathematics. But I still think I do valuable and interesting research, so I set out to explain what I am doing. Of course, both the blog author and the book author are right in a way. But probably the useful thing to do here would be to come up with new names for these different kind of activities, rather than destroy the language by trying to just place one more thing into "research in general" category.
vinchuco 9y ago
Article author > The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Video >"I didn't come up with this on my own, it's based on an algorithm called NEAT based on ap a paper by..."

324 points by hardmaru 9y ago

I like this differentiation. Early on in the 3d printer craze I would often say "A lot of people are building 3D printers but only a few people are engineering them." which was much the same argument. When you "engineer" a thing you do so understanding the constraints and design it to reliably within those constraints. When you just build something, as soon as it works once you're done.

Xunxi 9y ago
This reminds me of the quote from one of my favorite shows, MythBusters. “Remember kids, the only difference between Science and screwing around is writing it down.” One is a crude and improvised or temporary solution to a problem, designed to be more functional and timely than precise. Something done with little forethought, organization, planning or precision. Whereas the other is meticulously documented to note the choreographed nuances.
Animats 9y ago
Engineering has a specific meaning in that area. It means you have a design which can be built using standard techniques and will then work.
- ChuckMcM 9y ago
  agreed as long as you also append 'to specifications under all acceptable conditions.' It was the latter bit that really challenged 3D printer makers. So many 3D printers would work one time and not the next because there were so many parameters that they didn't specifically account for or respond to.
  - Animats 9y ago
    Here's one that's very robust.[1] [1] https://www.youtube.com/watch?v=srVHzKsBguM https://www.youtube.com/watch?v=srVHzKsBguM

commandlinefan 9y ago

> It is easy to miss that someone tackled the same problem as you (or had the same idea as you) last year ... and hard to resist the temptation to go jump out the window when you find out that the thing that you've been trying to figure out for a year has already been figured out...

jacobolus 9y ago
There are independent benefits to struggling with something for a while by yourself before adopting someone else’s solution. For one thing, you will understand the problem and their solution a whole lot better.
- colmvp 9y ago
  Pretty much. There are for example studies that show that people who attempt a problem before finding a solution learn a lot more than people who just look at the solution.

joe_the_user 9y ago

A really excellent summary of the distinctions. One thing I'd note is that before that Internet era, the scholarship needed to discover who had done X previously took a really significant amount of time. The Internet has changed that a lot, at the same that it has tempted people into tinkering and overall wingnuttery. I remember in 80s, having professor who functioned as little more than a search-engine, tell him an idea and, after a couple of puffs of his pipe, he'd give you a list of people who'd worked on it. He was marvelous despite apparently never having done much more than providing this kind of information. Another thing to note that scholarship implies a "community of researchers". If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. The article kind of says this but also consider that the people who have worked on this stuff exist, they have publications, conferences, meetups, etc and you can go to these or look at these and find-out whether or not your idea was considered and rejected, think about what's different between your idea and the idea the "field" rejected and so-forth.

jamesrcole 9y ago
> If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly: - each of the earlier people were approaching that problem from a specific angle, and new people might be approaching it from an untried angle - the earlier people might have had limited time to really investigate the matter, so there may be unturned stones - advances since the earlier work may give newer researchers a leg up. Note that all of these apply even if (as is likely) the earlier researchers were very smart.
- joe_the_user 9y ago
  >>If you are working problem X and people have looked this problem before, you should think doubly or triply before you claim your step-forward is necessary because people as smart as you already tried to think of everything. > At the same time, such reasoning is used to dismiss or discourage new attempts, often unfairly Well, if you use such an approach of considering previous work, you will have the tools to show you why your work shouldn't be dismissed (whether you are believed or not is a different matter). If you aren't looking at earlier work, not only do people have good ammunition for dismissing your position but it really is hard to be sure you haven't just repeated something.

jacobolus 9y ago

There are also some unfortunate features of academic research. A big one is that the product the researcher cares about is papers and citations, not working software per se. Papers you only write once. You can cherry-pick examples, work around bugs, do some facile analysis, publish the paper and then move on never to look at the code again. This means that code and data is typically hacky, unpolished, poorly documented, unmaintained, and unsuitable for public use (or often not publicly inspectable at all). Often the folks doing the grunt work are graduate students, who leave after a few years, leaving a new batch of graduate students to deal with the code without much guidance, so it tends to accrete features but seldom get any refactoring love. Nobody is paying for bug fixes or software maintenance or documentation. Someone coming from industry (or even just a tinkerer) who wants to use the same idea in a production environment often realizes halfway through reimplementation that it isn’t actually better than existing alternatives, and that the analysis/benchmarks in the paper were limited or misleading for a general context. Sorting the wheat from the chaff can be pretty demanding, ideas can sometimes be obfuscated by abstruse mathematical formalism, and academic researchers don’t typically go out of their way to make their output usable for non-academics.

3pt14159 9y ago
True, but sometimes you commercialize something with some insights you bring from industry that never would have occurred to the original academic that made the piece in the first place. Basically what I and another guy did in 2012. We were stupid though, took the first real acquisition offer. That's the problem with being in your twenties you want money ASAP. "$1m today or $10m three years from now? Take the $1m, we'll figure out how to get the other millions later." I think it's another reason why the rich keep getting richer. Muuuuuch easier to turn down $1m when daddy is worth $50m.
joe_the_user 9y ago
Well, the thing with academic research is that it involves a "community of researcher" whose activities involve "evolving an idea". And in such a situation, the result is primarily expressed by its text/human-language representation while code is secondary to this. While this is problematic to verifying the results, it should be noted that code as a result has significant problems. Code doesn't have a long history of being read, interrogated and modified while human-language does. As the article effectively says, don't underestimate the significant value of putting a result in historical context. A human-language description can give a clue whether X has been done before, perfectly functioning code might not. This question touches on "programming literacy", which in the extreme asks, "how can computer programs also search as _documents_ in the fashion scientific papers have historically served as documents." So far, I don't think any answer has really appeared. Perhaps that problem itself is "AI complete", present either a terrible or green pastures for research.

snarfy 9y ago

"The difference between screwing around and science is writing it down." - tpai

ggm 9y ago

As a self-defined tinkerer, I have tried the transition to a peer-review recognised researcher. It is very hard. Science is a very harsh mistress, and what you want your work to be saying is quite distinct from strong tests into what it actually says. The game-like qualities of surviving peer-review are just that: a set of pretty arbitrary hurdles to cross. If you work in a narrow enough field, your set of blind review peers is tiny and their critique, even if masked, can seem very personal at times. I don't like that much. I don't overall judge my attempt a success. I think this is an endevour best started early. Late stage career, trying to demonstrate you understand the rules of science, is hard.

GuiA 9y ago
The best way to get your work accepted in a peer reviewed venue (journal, conference, etc) is to voraciously consume everything that venue has published before, and make your work match the style observed + copiously quote works from that venue. Obviously the intrinsic quality of your work matters. But you’ll more often see work of marginal value that properly follows the form than work of great value that does not follow the form. In fact, in my field (academic HCI), I have never seen the latter, yet see plenty of the former at every conference I attend.
- ThrustVectoring 9y ago
  >copiously quote works from that venue This is more important than you'd initially think. Incentives matter, and professors are rewarded for publishing papers that get cited by other papers. So if you cite your reviewer's paper, it's in your reviewer's best interest to get your paper through review. Hence, quote a ton of works from that venue and hope to get lucky like that.
- f1notformula1 9y ago
  Replying here because this matched my train of thought well. I'm actually really curious to know if anyone has first-hand knowledge of a tinkerer (by this blog's definition) getting their work published. As the GP said, understanding the rules is one thing, but demonstrating their understanding is another. And as you said, matching the style seems to be more important than the value of work that is demonstrated. I see the value in matching terms, jargon, style etc just so the reviewers can standardize their thought processes. But as someone who's been a tinkerer for ages now it's hard to change styles for no immediate benefit. Maybe some examples will inspire me to try :)

hmwhy 9y ago

I tend to disagree with those distinctions and I'm inclined to think the difference is simply that, to a "researcher", doing things is a job; and to a "tinkerer", doing things is a joy. ## Scholarship To begin with, there are many "tinkerers" out there who practise good scholarship and at acknowledge prior art, as well as provide references. Anyone who is courteous and has the awareness that the world doesn't revolve around her would honestly document these things even if she is a "tinkerer". And just as there are "tinkerers" who practise good scholarship, there are "researchers" who practise bad scholarship as has been pointed out by @jacobolus. ## Testing I'm not sure what makes the author thinks that rigorous testing is only a "researcher" quality. The article just insulted all the "tinkerers" out there by saying: > "Here's another big thing. A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens." A "tinkerer" doesn't just make something and puts it out there. A serious "tinkerer" would not want to put something that doesn't work out there—it's embarrassing and it's just, wrong. To make something (reasonably complex) that works usually involves a lot of testing, even if the goal isn't to publish a scientific paper it would still involve a lot of testing. ## Goals This bit really lost me. It sounds like a bunch of self-contradictory, "researcher"-glorifying statements one after another. > Usually, goals in research are not just goals, but ambitious goals. The reason we don't know what the results of a research project will be is that the project is ambitious; no-one (as far we know) has attempted what we do before so our best guesses at what will happen are just that: guesses. A "tinkerer" can have ambitious goals. A tinkerer may do something that nobody has attempted before and, naturally, in that case she would just have to make guesses. If, hypothetically, I set a goal to investigate the possibility of using CSS to make high-quality 3D games that can run at 60 FPS in web browsers, does that make me a "tinkerer" or a "researcher"? What if, in the process of doing so, I practise good scholarship, document everything carefully in a well-organised format, but just simply have no interest of publishing it in a scientific journal—does that make me a "tinkerer" or a "researcher"? On this: > However, if you read a scientific paper those are usually not the stated reasons for embarking on the research work presented in the paper. Usually, the work is said to be motivated by some scientific problem (e.g. optimizing real-value vectors in high-dimensional spaces, identifying faces in a crowd, generating fun game levels for Super Mario Bros). And that is often the truth, or at least part of the truth, from a certain angle. The stated reasons in scientific paper is usually after written after the research has been carried out and they are designed to tell a reader why the works is important. In a space where everyone is struggling to keep up with publishing so that they can keep their job, who in the right mind would begin a paper with "our research group embarked on [insert amazing thing] because it seemed fun to do so and has never been done before"? Actually, I would love to read papers like that, because most of the time you know it's the usual standard crap when academics start a paper by stating a problem—it probably means that they haven't solved the problem and are not even close. Also, I would like to draw your attention to the last, contradictory sentence. It seems that the author isn't quite sure either. ## Persistence If a someone spends 10 years modifying and perfecting something that has never been attempted before, but does not care for its scientific value, does that make her a "tinkerer" or a "researcher"? Bottom line, anyone who appreciates, and wants to do, good work will naturally do all of those things above—"tinkerer" or not. To throw away sentences like: > Probably the most importance difference between tinkering and research is scholarship. > A tinkerer makes a thing and puts it out there. A researcher also tests the thing in some way, and writes up what happens. > While tinkering can be (and often is) done for the hell of it, research is meant to have some kind of goal. > Tinkerers are content to release something and then forget about it. Researchers carry out sustained efforts over a long time, where individual experiments and papers are part of the puzzle. ... is just simply conceited.

Jedi72 9y ago

I wonder if the author would classify the Wright brothers as 'tinkerers'. Overall I find the tone pretty insulting, nobody believes they are the bees-knees quite like academics.

mmilano 9y ago

The word research is pretty vague in comparison to the context the author interprets its meaning. He over-reaches on his explanation of what a researcher is, and leaves too much of a gap between researcher and tinkerer.

hatmatrix 9y ago
Maybe the distinction he is trying to make is between academic work and non-academic work (hence the emphasis on scholarship). Non-academics can do research but what makes it academic is its placement in the broader context of civilized endeavors.

traverseda 9y ago

> Scholarship >Last year, some guy made an experiment with evolving neural networks for Super Mario Bros and made a YouTube video out of it. The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. Then publish your damn results in a damn open journal, dammit.

mathetic 9y ago
That's not always an option. Your career very much depends on consistently publishing in "A" venues (conference/journal). So it depends on your subcommunity's opinion on open access. For example, programming language people as a community are super keen on it, so most top venues are becoming open access. If that's not the case for your field, it's basically academic suicide unless you already have an amazing reputation.
- traverseda 9y ago
  You see, I read that clause in particular as "The difference between research and tinkering is whether you belong to an organisation that's willing to spend 30k+ a year on journal access". And fuck that. If that's the attitude that academia is going to take, clearly we need to kill it off, tinker for a while, and start fresh. Maybe we can refresh some standards for academic writing while we're at it.
  - 52-6F-62 9y ago
    I've heard some prominent academics speak to the same end, or farther, in the rare access I've had. (Yet with displaying no less passion for their chosen subjects. Who would have guessed!)
    - neuromantik8086 9y ago
      Freeman Dyson is actually quite vocal about his opposition to the current structure of academia, so I'm not surprised that you'd be able to find other academics who were equally self-critical.

1auralynn 9y ago

As someone who spends a fair amount of time with software developed by researchers (mostly biochemistry visualization software), that for all the "research into what has been done", I think there is a huge amount of effort being duplicated all over the place. In my experience this is due to the relatively short timelines of grants, the fact that you have to spend most of your grant money demonstrating that what you created has some kind of scientific impact vs actually implementing the thing, and, as others have pointed out, the fact that a lot of the core work is done by diamond-in-the-rough superstar grad students who will move on after 2-4 years.

brownbat 9y ago

> First of all, it's not about whether you work at a university and have a PhD. People can do excellent research without a PhD, and certainly not everything that a PhD-holder does deserves to called research. I appreciate this dismissed as a straw man, but there are definitely online communities (and individuals I know) who reflexively discount comments from people outside academia. I understand the instinct, it's a reasonable immune response when everyone thinks they're an expert because they read a blog post or a pop sci best seller. Sometimes, though, the antibodies kill perfectly healthy tissue too, like valid contributions from bright generalists.

the_cat_kittles 9y ago
another factor is that you are very defensive if you spent a bunch of time getting a phd and someone didnt and is still able to do research as well as you. and im sure its not very common, since getting good at something requires lots of time and energy. but i bet it does sometimes, and thats probably one of the reasons why some of the healthy tissue is killed
- DataWorker 9y ago
  And they have families to feed and their income depends on thier PhD being enough of a barrier to entry that they can pay thier bills. So much human progress has been lost because the economic model is broken. We seemed to make more progress when science was restricted to a smaller group of practitioners, many of whom were wealthy enough that they could be guided by the science rather than the funding.
  - newsbinator 9y ago
    Also in the early days there was more low-hanging fruit. You could make scientific discoveries by locking yourself in a room for a week and thinking deeply about something you read recently. Then come out and test it with items from your kitchen. Now it's harder to make breakthroughs without specialized equipment, and the area of interest has narrowed to a tiny sliver.

togelius 9y ago

I'm the author, and I'm somewhat amused by the negative tone of many of the comments. Seems there are plenty of people with resentments towards research here ;)

strken 9y ago
I think it's because it treats hackers as deficient academics who can't be bothered to cite their sources properly or do any research, while there's an equal and opposite argument that academics are deficient hackers who can't be bothered to put their code on github or write any documentation. In reality, neither is really deficient, they're just aiming at different audiences which need different things.
- togelius 9y ago
  Oh yes, I'm a terrible hacker, I rarely write code anymore. An unfortunate byproduct of academia is that you get "promoted out of the job", and all the hands-on work is done by your lab members. I didn't mean to imply that tinkering was inferior to research - the whole premise was just to tease out how they're different, with different audiences, as you say. Interestingly, the discussion here has been dominated by people who think that I look down on them. People who've discussed it in other fora have not read the post that way.
  - deepnet 9y ago
    Togelius I believe you are in error in Seth Bling's case. The Show More links under his youtube MarI/O video link directly to the Stanley Miikkulainen NEAT paper. and he links the wikipedia page on Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution which cites two [1][2] of your papers among others : yet you say : " it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. " a bit harsh perhaps ? At least mentioning the foundational NEAT paper is not a complete lack of scholarship. I feel you have overlooked that Seth does try and provide good citations to his audience and it would kind if you mentioned him by name in your article rather than " some guy " He did inspire other twitch streamers to experiment with neuroevolution and neural nets and benchmark many SNES games.[3] although downstream this follow up does goes uncited by yourself. [1] "Neuroevolution in Games: State of the Art and Open Challenges" https://arxiv.org/pdf/1410.7326v3.pdf https://arxiv.org/pdf/1410.7326v3.pdf [2] "Countering poisonous inputs with memetic neuroevolution" (PDF)https://www.academia.edu/download/30945872/poison.pdf https://www.academia.edu/download/30945872/poison.pdf [3] Mario Bros mari/o https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat https://www.youtube.com/watch?v=bRxUQNFxAWc https://www.youtube.com/watch?v=bRxUQNFxAWc Mari/o kart winterbunny https://clips.twitch.tv/HilariousPolishedZucchiniHassanChop https://clips.twitch.tv/HilariousPolishedZucchiniHassanChop - mario kart RNN mariflow

thethirdone 9y ago

> Seen as tinkering, that work and video is good work; seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. I think its important to note that that he does reference NEAT (and the paper it comes from) and the emulator he used. This is building on what others have already done. While the 2009 paper from the post author would definitely be worth note, I feel "atrocious" is a bit harsh.

cornelis 9y ago

Thank you OP for your very insightful blog post. You have been better than any other academic at articulating exactly what happens when I (or one of my colleagues) interact with the academic world and you made me realize that the academic world has a set of values that might be good but which I should not even attempt to incorporate in my own professional life. I have come to my personal conclusion that there is a difference between 'science' and 'the academic world' in much the same way as there is a difference between 'religion' and 'the church'. One can be very religious without going to the church and doing all the rites that are required by the church. In much the same way, a 'tinkerer' (or 'heathen') can be scientific without being academic. I believe that wiki summarizes it quite well for me: "Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe". There is no statement about giving proper credit (vanity) or not being allowed to reinvent the wheel without first making damn sure that you are indeed reinventing. I want to solve real world problems and the way I want to do that is in a structural and repeatable approach. Science helps me doing this. Of course I'm not living in a vacuum, so I will look up as much as I can about the subject at hand as is reasonable within the available amount of time. Interestingly enough, I almost never end up with a research paper, but almost always with blog posts, books, tutorials et cetera. Products don't sell themselves and neither do research papers apparently. I think there is something seriously wrong if that is not perceived as a crisis in the academic world.

iagovar 9y ago
Research papers are typically about very specific topics. It would be strange to end up reading one on the Internet unless you already know the field keywords and use google scholar or any other science-oriented browser.
- jacobolus 9y ago
  Indeed, google scholar and similar citation graphs are amazing. It makes tracking down vaguely relevant papers much easier than in the past. I can skim a few dozen papers one day, and then more closely inspect the handful that seem most relevant or interesting. I wish that google scholar showed bidirectional links though. I suspect they might have concluded that the list of citations included in a particular paper is part of the content of the paper (not just metadata) so that showing it might infringe copyright.

stared 9y ago

Some time ago I wrote about academia vs industry in the context of data science: In academia, you are allowed to cherry-pick an artificial problem and work on it for 2 years. The result needs to be novel, and you need to research previous and similar solutions. The solution needs to be perfect, even if not on time. In industry, you should solve a given problem end-to-end. Things need to work, and there is little difference if it is based on an academic paper, usage of an existing library, your own code or an impromptu hack. The solution needs to be on time, even if just good enough and based on shady and poorly understood assumptions. So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science. (from http://p.migdal.pl/2016/03/15/data-science-intro-for-math-phys-background.html#priorities http://p.migdal.pl/2016/03/15/data-science-intro-for-math-ph...)

denzil_correa 9y ago
> So, contrary to its name, data science is rarely science. That is, in data science the emphasis is on practical results (like in engineering) - not proofs, mathematical purity or rigour characteristic to academic science. Science in a nutshell is to take a hypothesis, test it rigorously and validate/invalidate the hypothesis based on the results. The results of this experiment could be extremely practical. Data science that follows the scientific experimental design, does science. The one that doesn't is equivalent to reproduction of past experimental results OR putting a blindfold on your eyes and throwing darts around until it hits the bull's eye.
- stared 9y ago
  I disagree with this common, yet faulty, explanation. (IMHO coming from people who have never worked in science and for whom everything R&D looks like "science!".) Building engines or cars (involving a lot of tests!) is considered engineering. And as any serious engineering it's not purely random trails&errors. A/B tests are not a sufficient criterion to make something science. And data science (in business) is focused on solving practical problems (which may or may not be research) rather than open-ended research (which may or may not be practical). Source: I worked in academia. I now work in data science. You?

Radim 9y ago

Funny thing: do your eyes light up seeing a "scholarly academic paper", or a "deeply tinkering blog"? Which one have we trained our brains to react more strongly to, what is more exciting to you? What do you feel you learn more out of, have more confidence to choose as a starting point for further activities? OP's distinction between tinkering and research seems wishful thinking at best. There are many grievances people have with research (and it's easy to think of examples), but I find most tend to fall under: * (unaddressed) RISK: poor result analysis, cherry-picked examples; glossing over edge cases and failure modes; focus on novelty and "academic writing" at the cost of repeatability and conceptual clarity. The Mummy Effect of research that looks great but crumbles to dust when you actually try to touch it: https://rare-technologies.com/mummy-effect-bridging-gap-between-academia-industry/ https://rare-technologies.com/mummy-effect-bridging-gap-betw... * (missing) OWNERSHIP: cadence dictated by grant cycles rather than one's intrinsic motivation to solve a problem; slow or missing feedback cycles; no dog-fooding => poor code and documentation; few incentives to make it easier for others to pick up or validate the results In fact, both of these seem better addressed by tinkerers than by institutionalized research. No wonder people react more positively to "tinkerers" and "hackers"! Academia has a lot of positive inertia (and taxes) going for it, but all goodwill has its limits.

deepnet 9y ago

OP, Togelius, is quite wrong in 1 point at least. tl;dr Seth Bling directly references NEAT and his links[Show More [1]] link to 2 Togelius papers[13][14] among many others ! To wit, in OP, Togelius mentions Seth Bling's MarI/O[1] as lacking scholarship because he didn't mention Togelius: " seen as research, it is atrocious because of the complete lack of scholarship. The guy didn't even know he was reinventing the wheel, and didn't care to look it up. " Rather disdaignfully (sadly/ ironically/ poigniantly) Togelius refers to Seth not by name but only as " some guy ". Yet Seth Bling[2] in his short video, mentions and explains NEAT and directly links Ken Stanley's and Risto Miikkulainen's foundational NEAT paper[10] at U.texas which Togelius builds on [hint: click SHOW MORE under Seth's video] And much more, i.e.: ---- the innovative toolchain Seth uses: Bizhawk to script SNES emulators in Lua [12]! ---- Seth links the wikipedia page on Neuroevolution[11] which directly references 2 of Togelius's own papers[13][14] Seth's [Show More] Links on the MarI/O video[1] are very very comprehensive and (IMHO) indeed a great example of scholarship ! Togelius is a top flight scholar, and publishes in the field of AI game players and level generators[15] so I hope he will welcome some random tinkerer pointing out his error. [full disclosure: I am a Togelius fan] To give Togelius his due he does at least link to Seth's MarI/O video. Perhaps Togelius failed to click / read the Show More links ? The charitable benefit of doubt principle is accidental oversight. Seth Bling is primarily a Twitch Streamer[2] and Speed Runner[3] who often popularises complex programming[4], math and academic ideas[5] to a primarily young audience using Mario and Minecraft[6] and takes requests via his subreddit[7] Seth's very popular Youtube video MarI/O which uses NEAT (Neuro Evolution of Augmenting Topologies] to learn and play Super Mario World, has led to a huge number of streamers on twitch playing other MarI/O levels[16], using RNNs, LSTMs, and other SMW levels and SNES games, like RNN multiplayer x4 Mario Karts[17]. The OP article is great, academic rigor and bibliographic scholarship might sometimes differentiate the full time university scholar from the amateur researcher, enthusiast or tinkerer. Seth and his many followers may not be 'scholars' as in academic research but it is an amateur community of mostly youngsters doing their own research by having fun and sharing ideas, mostly undocumented outside their own transient twitch streams. Compare this to the well respected amateur astronomy community and perhaps a responsible academic approach is to try to reach out and tap in rather than adding to the distance. By his own OP standards of Togelius is lacking in that hasn't discovered/ mentioned the work done by the Twitch/ Bizhawk community - it is not entirely undocumented [16]. Togelius your AI Mario competition[8] is legendary in academic AI circles but not necessarily widely known by youngsters playing Mario ( and to be fair AFAIK you don't publish in their forums like twitch) although the code Infinite Mario[9] you base your work on is by Marcus Persson[10]. Though Persson is better known as Notch [10] ( author of Minecraft ) who is more famous than perhaps even Alan Turing , Yann le Cun, Jurgen Schmidhuber, or Geoff Hinton to the under 16 crowd. Perhaps Togelius could mashup some of his work and do a crossover episode with Seth or relaunch the Infinite Mario competition[8] / benchmarks to this new younger crowd evolving Neural Topologies to play Mario games - or even help document some of their fun. As public servants funded by tax dollars it is perhaps the responsibility of the academic to reach out rather than blaming fun loving amateurs popularising and doing research / 'tinkering' for not putting in the due diligence of scrupulously mentioning every source. [IMHO] I agree amateurs don't always do full and proper scholarship and links are food for hungry minds - but they aren't publicly funded researchers but amateurs working for love of the subject in their spare time, for free, or for streaming donations and youtube hits, using transitory, live media and whose audience must be constantly engaged and may have shorter attention spans than the average academic. [full disclosure I am an amatuer neural net nerd] [1]Seth Bling's MarI/O https://www.youtube.com/watch?v=qv6UVOQ0F44 https://www.youtube.com/watch?v=qv6UVOQ0F44 [2]https://www.twitch.tv/sethbling https://www.twitch.tv/sethbling [3]https://www.youtube.com/watch?v=-spFoon7klA https://www.youtube.com/watch?v=-spFoon7klA sub 1 min SMW via credits warp [4]Bling's arranging shells in SMW to human inject Flappy Bird code into SNES https://www.twitch.tv/videos/57032858 https://www.twitch.tv/videos/57032858 [4]Bling's Atari 2600 emulator built in Minecraft https://www.youtube.com/watch?v=5nViIUfDMJg https://www.youtube.com/watch?v=5nViIUfDMJg [5]Seth Bling's RNN Mario Cart race https://www.youtube.com/watch?v=Ipi40cb_RsI https://www.youtube.com/watch?v=Ipi40cb_RsI [6]Bling's Minecraft Redstone tutorials https://www.youtube.com/watch?v=DzSpuMDtyUU&list=PL2Qvl4gaBge1kABr3aBCrMSb94HFLIjbn https://www.youtube.com/watch?v=DzSpuMDtyUU&list=PL2Qvl4gaBg... [7]https://sethbling.reddit.com https://sethbling.reddit.com [8]Togelius 2009 Infinite Mario AI competition http://julian.togelius.com/mariocompetition2009/ http://julian.togelius.com/mariocompetition2009/ [9]https://web.archive.org/web/20080423023424/http://www.mojang.com/notch/mario/ https://web.archive.org/web/20080423023424/http://www.mojang... [10]Stanley Miikkulainen NEAT paper http://nn.cs.utexas.edu/?stanley:ec02 http://nn.cs.utexas.edu/?stanley:ec02 [11] https://en.wikipedia.org/wiki/Neuroevolution https://en.wikipedia.org/wiki/Neuroevolution [12] Scripting Lua in Bizhawk Emulator http://tasvideos.org/Bizhawk/LuaFunctions.html http://tasvideos.org/Bizhawk/LuaFunctions.html [13] Risi, Sebastian; Togelius, Julian (2017). "Neuroevolution in Games: State of the Art and Open Challenges" (PDF). IEEE Transactions on Computational Intelligence and AI in Games.https://arxiv.org/pdf/1410.7326v3.pdf https://arxiv.org/pdf/1410.7326v3.pdf [14] Togelius, Julian; Schaul, Tom; Schmidhuber, Jurgen; Gomez, Faustino (2008), "Countering poisonous inputs with memetic neuroevolution" (PDF), Parallel Problem Solving from Nature https://www.academia.edu/download/30945872/poison.pdf https://www.academia.edu/download/30945872/poison.pdf [15] Togelius' Game AI book http://gameaibook.org/ http://gameaibook.org/ [16] https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat https://clips.twitch.tv/CourteousEmpathicTruffleCeilingCat [16] https://thenextweb.com/artificial-intelligence/2018/01/03/this-live-stream-of-ai-learning-to-play-super-mario-bros-is-awesome/ https://thenextweb.com/artificial-intelligence/2018/01/03/th... [17] https://www.youtube.com/watch?v=S9Y_I9vY8Qw https://www.youtube.com/watch?v=S9Y_I9vY8Qw

isthatart 9y ago

I'm an academic researcher. I disagree with most of the article. The main distinction I see between "research" and "tinkering" is not what the author calls "scholarship". From personal experience the only serious distinction is that "researchers" tend to have a very long time attention span compared with "tinkerers". As for the rest, the goal of academic research, lately, seems to be to produce as many as possible units for the benefit of the publishing industry. In the case of very successful articles, you can fit all the readers in a (small) bus. Most of the articles are badly written, because they follow old habits from the age when printing was expensive. Citations are done in a very selective way. Not so long ago there was no need to give the source of an "unpublished" article, "preprint" was enough. Tinkerers work is almost never cited, even if it is read and used. Blogs? are you kidding me? that's not "scholarship". Heck, even arXiv still seems exotic in some fields, despite the fact that it is a good, standard communication mean for the researchers in other fields. Finally, peer review. How does it work? Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could.This passes for validation. Magic. In conclusion I believe the future of academic research comes from "tinkering" with a long time attention span. The dissemination is technically trivial. Peer review will be supplemented by more rigorous validation (although there is no absolute solution here). The same kind of validation will be applied to tinkerers results.

throwawayjava 9y ago
> The main distinction I see between "research" and "tinkering" is not what the author calls "scholarship". From personal experience the only serious distinction is that "researchers" tend to have a very long time attention span compared with "tinkerers". Many aspects of "scholarship" become inevitable when "tinkering over a long timeframe". You build up a knowledge base about the problem and/or methodology you're tinkering with, you eventually form a community around your tinkering, etc. Lots and lots of tinkerers gratify themselves without contributing any modicum of knowledge to the world. But the most successful tinkerers are almost always effectively scholars in the sense of this post. In the sense that they're familiar with a body of knowledge/the people producing that knowledge. And they contribute genuinely new ways of doing things, or new ideas of things to do, or excellent executions on existing ideas that achieve the aspirations of previously articulated ideas. E.g., Linus is certainly a successful OS scholar in the sense of this post. > In the case of very successful articles, you can fit all the readers in a (small) bus This is off by at least an order of magnitude for any even remotely reasonable definition of "successful". > Researcher B reads what researcher A reports and expresses an opinion, just like any honest reader could First, the author explicitly kinda-sorta agrees with you here (...Third, it's not really the publication...), so it's pretty clear that publication venue != scholarship. As an aside, peer review != validation!!! This is something that every young Ph.D. should explicitly learn early on.

sigi45 9y ago

Whenever i read a paper, i do have troubles understanding the fine details. That one math formular which is nowhere explained? Yeah good... I often enough, get the idea and details like results but thats it. Those papers are not tutorials and they are not written / done to be reproduced and easy understandable. They are, in my opinion, written for other science people who spend there work time doing something similiar or need to solve the problem what the paper solves and rebuild those results in time consuming work. Probably still better than having no paper but still way more work than just using it. i do read "the morning paper" (https://blog.acolyer.org/ https://blog.acolyer.org/) and he is really good in reevaluating papers. I'm very curious how much time he spends reading and analysing them.

cosmic_ape 9y ago

The term research became extremely diluted now. It attempts to put very different things under one roof, and so is not very useful. Near the end of that blog post you find: >>Reading the book, I felt that most of my research is not science, barely engineering and absolutely not mathematics. But I still think I do valuable and interesting research, so I set out to explain what I am doing. Of course, both the blog author and the book author are right in a way. But probably the useful thing to do here would be to come up with new names for these different kind of activities, rather than destroy the language by trying to just place one more thing into "research in general" category.

vinchuco 9y ago

Article author > The video certainly reached more people on the internet than my work did; it makes no mention of any previous work. Video >"I didn't come up with this on my own, it's based on an algorithm called NEAT based on ap a paper by..."