19 ms·

My story as a self-taught AI researcher

364 points by emilwallner 7y ago

narenst 7y ago
This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. It requires a lot of trial and error and time/cost investment to bring the AI techniques to the masses. The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups). The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.
- woah 7y ago
  On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques
  - SQueeeeeL 7y ago
    Low data techniques are just another name for algorithms/equations. Dijstras algorithm required 0 training graphs to make. Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)
    - ssivark 7y ago
      Agree with your first statement and disagree with your second; I don’t think the former implies the latter. I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.
      - btrettel 7y ago
        These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables. As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.
  - TrainedMonkey 7y ago
    Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.
    - cygaril 7y ago
      Or our entire evolutionary history of data.
      - AlanSE 7y ago
        ...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.
        eanzenberg 7y ago
        I’m not sure our genetics encodes all the physics of being a person. A human brain is so complex we’re not even close to simulating it on silicon
      - K0SM0S 7y ago
        It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us. The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself. It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).
  - mendeza 7y ago
    I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation. Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reality_gap.pdf http://josh-tobin.com/assets/pdf/randomization_and_the_reali... http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_Tobin_RSS19.pdf http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T... And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText https://github.com/ankush-me/SynthText
    - tasogare 7y ago
      How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.
  - gdubs 7y ago
    This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.
- andreyk 7y ago
  wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.
  - K0SM0S 7y ago
    So if I understand correctly, to reformulate in my own words/views: while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal), fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough. Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
    - yorwba 7y ago
      It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone. On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.
    - andreyk 7y ago
      yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab (http://svl.stanford.edu/ http://svl.stanford.edu/) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.
- z3t4 7y ago
  > colorizing b&w photos You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.
qntty 7y ago
"Many are realizing that education is a zero-sum credential game." Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.
- Nasrudith 7y ago
  It is pretty strange even from an economist really - they of all people should be able to understand and articulate the difference between signaling value and direct utility value of a given good or service.
  - koube 7y ago
    Economists have been debating the skills vs signaling value of education, especially since Bryan Caplan released his book The Case Against Education. If you want to get a smattering of opinion on the issue the book's reviews and dicussionsn would be a good starting point. https://en.wikipedia.org/wiki/The_Case_Against_Education#Reviews https://en.wikipedia.org/wiki/The_Case_Against_Education#Rev... Bryan Caplan back and forth with Noah Smith on the book: https://www.econlib.org/archives/2015/04/educational_sig_1.html https://www.econlib.org/archives/2015/04/educational_sig_1.h... Bryan Caplan back and forth with Bill Dickens on the book: https://www.econlib.org/archives/2010/08/education_and_s.html https://www.econlib.org/archives/2010/08/education_and_s.htm...
  - Gimpei 7y ago
    It's not a majority view among economists. Caplan is the only person I can think of who holds this view.
    - jdminhbg 7y ago
      Caplan is definitely not the only economist who holds this view. Most place the signaling/human capital split around 50/50: https://www.econlib.org/archives/2011/11/kauffman_econ_b.html https://www.econlib.org/archives/2011/11/kauffman_econ_b.htm...
bluetwo 7y ago
The thing that disappoints me about the aspirations of being a researcher is that the goal is to get paid to study AI, not solve real-world problems. I would rather build a small company by solving a real problem than work for a big company spinning my wheels.
- currymj 7y ago
  for a lot of people who end up in research-type jobs, a sense of curiosity is one of their strongest motivators, and they want work that will let them pursue their curiosity. it sounds like you're motivated by something else.
  - bluetwo 7y ago
    Well yes, I am very curious, just more motivated to solve problems.
- hogFeast 7y ago
  Why I didn't go into academia but GL convincing other people of the value in that. I am sure there are cultural differences here but where I am, the goal of most people who study CS is: leave me alone while I mess about with X (evidence: the local college was doing speech processing/nlp in the 60s, they actively turned down paid work...unsurprisingly, they got left in the dust, professors are now being encouraged into doing commercial work but, of course, most of it is totally nonviable and is just more messing about with complex nonsense that doesn't work). I think if you look at history this is also evident: the inventions of the late 18th century were a function of necessity, the invention of semis (not just in the US but how Taiwan developed)...this isn't to say academia is pointless but there is just far more going on (I think if you look at some of the East Asian nations that get great academic results, their progress on actual R&D innovation is far less impressive).
  - ssivark 7y ago
    (American) Academia is a complicated matter, so I’ll elide commenting on that. For a thoughtful counterpoint to the necessity argument, see: https://jnd.org/technology_first_needs_last/ https://jnd.org/technology_first_needs_last/ (previously discussed on HN)
exdsq 7y ago
Survivorship bias or reality: 3 months learning FastAI, 3-12 months personal projects and consulting, 2 months flashcards of ~100 papers, 6 months to publish a paper What does he mean by ‘paper’? A Medium post? NeurIPS?
- TrackerFF 7y ago
  It could be that he's an exceptionally smart and driven guy, that just happens to pick up things really fast; I've seen those IRL myself, but not in only a year or two. But yeah, going from 6 months of programming experience with C, to a Deep Learning internship - that sounds a bit far stretched.
- itsmefaz 7y ago
  Submitting papers in conferences rather than journals.
  - exdsq 7y ago
    Without submitting from a known university or research group? Seems unlikely to me. I have no doubt it can and has been done, but for it to be a regular thing such that one can recommend it in a 'Couch to 5k' style method? No way. I have some friends in Oxford who are DPhil/Postdocs in highly reputable research departments specializing in ML and if they sometimes struggle to get more than a poster session at the leading conferences, with the addition of well known professors names attached, then there's just no way I can believe Joe Bloggs who just learnt python 12 months ago is able to do the same. I almost want to follow his guide just to check.
    - itsmefaz 7y ago
      The idea is to choose conferences like InfoQ where application type research is accepted. Like Build-X-using-TensorFlow, it doesn't have to align with standard research which requires formal education. One also needs to be skeptical while reading such a PR post and not get swayed away by the hero's journey in it.
K0SM0S 7y ago
This was a great read (and great nuggets, like that paper on Intelligence by Chollet). I wonder: — Is math a problem for non-academic researchers? Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but math nonetheless. Not that you can't learn these on your own, but how big is the gap in practice, on the job, compared with actual PhDs in ML/math? (how much of a hinderance, a problem it is for the self-taught researcher) — "Contracting" in the field of AI sounds great but, how exactly? Especially solo: what type of clients and how/where to find them, what type of 'business proposition' as a freelancer do you offer, what's the pricing structure of such gigs? I mean, I can sell you websites and visuals and stuff, but AI? I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny and demand scripting skills to sort out (extract business value), not AI, so the value proposition is low for both parties; it's still early adoption so 90% don't even consider spending 1 cent on "AI" unless as a SaaS (they actually don't need to know if it's AI or programming). I can imagine tons of fantastic research to do with SMBs, as partners or 'interested sponsors' (should they reap benefits on a low investment), but really not much yet in the way of "freelancer products" to market and sell for a living. I'm eagerly anticipating those days, but it's more like 2025-2030 as I see it. I would love to hear first hand takes on this.
- JamesBarney 7y ago
  It's my understanding that dirty datasets that "demand scripting skills to sort out" is pretty common and most data scientists spend 80% of their time "sorting this out".
- deepnotderp 7y ago
  To be honest, linear algebra is not that difficult to learn on your own, and plenty of people do. Gilbert Strang's course on OCW has made introductory linear algebra quite accessible. Things like topology (e.g. TDA, persistent homology, etc.) aren't really mainstream yet, but even then most of it isn't really "hardcore" math in the sense that you can get away with a basic understanding, e.g. what a Vietoris-Rips complex is and why we use it instead of a Cech complex in TDA. Plus most DL research nowadays is pretty (advanced) math-light. That being said, taking the time to understand the math is absolutely worthwhile in my experience. It should also be noted that a lot of real world ML/AI projects in industry aren't really about brand new algorithms using advanced math, but rather more about applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
LemonAndroid 7y ago
I don't see how this is self-taught, as the person got picked up for an internship and could learn from experts first-handly. FAKE.
- curiousgal 7y ago
  He also studied at 42, which is most likely why he got picked for the internship to begin with. I don't get this self-congratulating BS, guy says he toured the world (good luck doing that with a shitty passport) and was named king of a village in Ghana (right..). I guess people who get lucky have to always go to great lengths to justify and spin that. They're free to do so of course, but they should not give advice based on it. Regardless of that, I suppose the bar for being a "researcher" has been stooped so low. According to this guy publishing an ML paper is equivalent to writing a blog post or making a video about "AI".
rmah 7y ago
Is this guy actually a researcher in the way most people would think of it? That is, someone who pushes the boundaries of science; who develops new AI techniques or finds the hard boundaries of existing AI techniques; who finds new ways compose multiple AI techniques cohesively; who explores the theoretical foundations of AI. Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)? I can't help but wonder a bit.
- ssivark 7y ago
  For better or worse, the definition of researcher has morphed into a combination of 1. Solves previously unsolved problems 2. Publishes papers sharing those solutions without regard to the kind/spirit/scope of problems solved. Since conference publications don’t have the same number constraints as journal papers, and are accepting of application-specific results, this explosion of what is considered “research” is somewhat inevitable. Also, there are a lot of people chasing this given the prestige associated with the title.
  - deleted 7y ago
    [deleted]
- rlayton2 7y ago
  Research needs people at the entire spread of the spectrum - from those making fundamental improvements to underlying theory, all the way to people running the thing to see if it works on actual problems people have (obviously in a robust and verifiable way).
- anjc 7y ago
  Most people would be wrong if they think that this is what all researchers do
- throwawayjava 7y ago
  Yes. From his GH profile looks like he's a competitive applicant for ML engineering positions or perhaps a fellowship/residency/PhD program. So, a junior researcher at the level of a decent second or third year PhD student. A researcher, maybe someone you'd trust to build a prototype or product, lots of potential, but probably not someone you'd trust to run a research program.
wigl 7y ago
This reeks of survivorship bias to me. I much prefer Andreas Madsen's more sober and self-conscious take on independent research [0]. > I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to self-fund your research. Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020? How many are going to place in a Kaggle competition with just half a year of experience? More importantly, just how many people are privileged/secure enough to put their all into learning, with no sense of security or peer support? > I started working with Google because I reproduced an ML paper, wrote a blog post about it, and promoted it. Google’s brand department was looking for case studies of their products, TensorFlow in this case. They made a video about my project. Someone at Google saw the video, though my skill set could be useful, and pinged me on Twitter. So what really mattered was self-promotion, good timing, and luck. > Tl;dr, I spent a few years planning and embarking on personal development adventures. They were loosely modeled after the Jungian hero’s journey with the influences of Buddhism and Stoicism. Why does the author have to present his life like one would in a fucking college essay? [0] https://medium.com/@andreas_madsen/becoming-an-independent-researcher-and-getting-published-in-iclr-with-spotlight-c93ef0b39b8b https://medium.com/@andreas_madsen/becoming-an-independent-r...
- drongoking 7y ago
  > So what really mattered was self-promotion, good timing, and luck. Yes. He seems like someone who is good at self-promotion and networking. Well, good for him, but I think he underplays the role these have in his success. > Why does the author have to present his life like one would in a fucking college essay? I guess that's the self-promotion. And humble-bragging. Like this bit: "I started working as a teacher in the countryside, but after invoking the spirit of their dead chief, they later annotated me the king of their village."
  - wigl 7y ago
    > Well, good for him, but I think he underplays the role these have in his success. Exactly. Good for Emil, but it's always frustrating to hear survivorship bias preaching. Even the interviewer starts off by saying: "By the way, I really love your CV - the quirks section was especially fun to read." It's even more frustrating when I hear non-POC's talk about their journey to some non-western country (and subsequent conquering of fantastical goals like gaining the approval of locals) or pursuit of some sense of foreign culture. It's almost a given that they have internalized and appropriated the ideas (i.e. Buddhism or even worse post-retreat Buddhism). Good for the author to receive such positive feedback for such signaling, but it makes me sad to know that I might not receive the same.
ineedasername 7y ago
I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1. What might be termed "basic" or "pure" research. I'm not placing one of these above the other in terms of importance. They are both necessary, and they form a virtuous feedback loop between the two that, one without the other, would see the other wither on the vine. In the example of this particular person, Emil Wallner, he appears to be doing #1, and perhaps doing so in a way that might help inform more of #2.
- JamesBarney 7y ago
  I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
  - rumanator 7y ago
    > Mind providing some concrete examples? It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications. To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
    - JamesBarney 7y ago
      I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical). But I was asking because I was specifically looking for concrete examples in deep learning.
octokatt 7y ago
Was anyone else really put off by the congratulatory tone of the article, and the #Quirks list on the resume?: https://github.com/emilwallner/Emil-Wallner-LinkedIn-Resume#quirks https://github.com/emilwallner/Emil-Wallner-LinkedIn-Resume#...
- z3rgl1ng 7y ago
  Yes. The writing style is unkind to the author.
- TrackerFF 7y ago
  You know, they (read: recruiters) say that if you don't have a normal background, you should have an interesting one. For some reason, if you don't have the same cookie-cutter background as everyone else, you need to have some amazing and convincing story to tell. I think it's good that companies are willing to look into non-trad candidates, that may not have found their "calling", so to speak, until their late 20's / 30's or whatever. But it does start to sound contrived when a bunch of 'em have the same type of alternate-route stories, which involves traveling to Africa / India / SE Asia to help out kids, create some startup aimed at climate / poverty / equality / etc. I guess it makes you sound passionate and legit - no-one can say that you wasted your time on chasing those things.
  - octokatt 7y ago
    I guess it can help sound passionate, but the list together doesn't sound interesting -- it sounds like an AI read a bunch of minimalism lifestyle blogs and output "interesting_backstory.txt". Nothing on the quirks list is actually a quirk. They're interesting things he's done that other people wrote books about, received praise for, and then he followed their newer, well-traveled path. It's not a non-traditional background. He's not a refugee who managed to learn coding. He's not volunteering at a needle exchange clinic. I think that's what's bothering me; he's pretending to be interesting, and taking the room which could be going to someone else. Thank you for helping me get to why something felt off. Appreciated, internet stranger.
- floodyberry- 7y ago
NWM123 7y ago
I personally found this article to be very interesting. I don't know much about AI, but I was fascinated by the discussion of peer to peer educational system. I believe that they will become more prevalent as student loan payments cause debt to so much of our population in order to get an education .
jshowa3 7y ago
I don't know why people think getting a credential does nothing or that people "copy and paste" the assignments. Sure it may be possible, but what prevents people from copying and pasting public git repos? Either way, this whole focus on "portfolios are everything and credentials are meaningless" spits in the face of all the work I did to get my university education. And it didn't involve "copying assignments". And you come out with one hell of a portfolio if you take your education seriously. I mean I don't think self-educated people are without merit. I happen to think they're really important. But I only ever see them rag on higher education, despite them having "never been there". Just another example of wunderkin super genius knows all because he was able to follow a non-standard path and make it. Glad he was smart enough to become a Google employee. But I question whether he should be giving advice on paths to get there when there's always many paths to a position. And especially after reading his brief comments on how credentials imply you're a liar.
- ynx 7y ago
  yeah, as someone who had a partial education, I totally get the value of a degree, so whenever someone says "higher education is useless" I read it as "However successful I am now, I wasn't the kind of person who would succeed in school then"
  - randomdata 7y ago
    Whenever I hear someone say "higher education is useless" I hear it in reference to the "common wisdom" that higher education increases incomes. In which case they are absolutely correct. Higher education is useless in achieving that. Incomes have held stagnant for many decades, even as more and more of the population attain higher levels of scholastic achievement. Mathematically, incomes cannot not remain stagnant if more money is earned as a result of attaining higher education. I'm not sure I have heard anyone claim that "higher education is useless" in general. Education is never useless in general.
    - jshowa3 7y ago
      Statistically, higher education does increase incomes when factoring in multiple disciplines. The data is pretty clear on this.
DoctorOetker 7y ago
This is a great example of how we "collectively" [1] conflate phenomena, skillsets, ... into one topic: machine learning, AI. 1) There is the general phenomena or collective project, where hardware, algorithms and human insights are improved to approach the situation of man-made intelligent machines. 2) There are the people who are designing algorithms, using mathematical intuition and knowledge, analogies with physics, etc... Most people would agree these people are doing optimization / machine learning "proper". 3) There are the people working on improving hardware for machine learning / optimization purpouses, by looking at the most performant algorithms, breaking them down into primitive operations and requirements for hardware, there are also people working on the algorithms themselves and finding computational shortcuts (which can end up in software or hardware, can end up as proprietary knowledge or common knowledge, ...). The distinction between hard and software is somewhat blurry, since hardware designers can optimize or implement a section of software into hardware. A lot of this can still be considered ML "proper". 4) Then there are the people who apply the ML frameworks and their exposed choices and settings to a specific problem domain. Many of them don't need to understand the internals if they don't need state of the art results. Many would nevertheless benefit from understanding the internals, and the requisite math. What I propose is to stop calling their activity as Machine Learning, and instead call it Machine Teaching. They are teachers, and just like elite schools they can choose which specific type of available student they will teach, and they can tweak (or filter from a large family of students) which student they select to teach the task at hand. There are bound to be many advantages of having actual human teachers get involved in machine teaching. These people will not be proficient in designing novel families of students unless they also know the requisite math, and identify those ML papers that are ML "proper" instead of ML "teacher". When trying to find important foundational insights in ML "proper" one is typically overwhelmed by a large surplus of ML "teacher" type papers. These are important datapoints, and necessary to advance human insight into ML "proper", but they are data, not knowledge. There are actual ML "proper" knowledge papers out there that explain why a certain phenomena is such and so, and they get very little attention because they necessarily lag the breakthrough ML datapoint paper, and most ML "teachers" don't have the math background to understand them. So the probability that a given ML "proper" researcher fundamentally improves the state of the art is much higher than the probability that a given ML "teacher" will fundamentally improve the state of the art. At the same time the probability that a given fundamental breakthrough was achieved by an ML "teacher" is higher than the probability that a given fundamental breakthrough was achieved by an ML "proper" researcher: P( Breakthrough | Proper ) > P ( Breakthrough | teacher) while P ( Teacher | Breakthrough ) > P ( Proper | Breakthrough ) Since most people don't have the broad math / physics / ... knowledge to draw on, the number of ML "teachers" is much higher than ML "proper" researchers. [1] well, really, some actors have vested interests in conflating those together... EDIT: just to be clear, I am not complaining about ML Teachers, we need the ML Teachers, and their breakthrough datapoints. What I am complaining about, is conflating both activities of ML Proper and ML Teaching. This makes it harder for the few ML Proper researchers to find each other's insights.
vector_spaces 7y ago
> Creating value with your knowledge is evidence of learning. I see learning as a by-product of trying to achieve an intrinsic goal, rather than an isolated activity to become educated. > Early evidence of practical knowledge often comes from usage metrics on GitHub, or reader metrics from your work blog. Progress in theoretical work starts by having researchers you consider interesting engage with your work. > Taste has more to do about character development than knowledge. You need taste to form an independent opinion of a field, having the courage to pursue unconventional areas and to not get caught up in self-admiration. When I study abstract interpretation or lattices, I'm doing so because I find those subjects interesting and beautiful, and studying math relaxes me. I can lie to myself and say that it's improving my problem solving ability and that it's like doing mental yoga and will make me better at my job or some baloney, but that's not why I do it. I can spend time with a plant in my garden, take a cutting, root it and replant it, and watch it grow, learn the ebbs and flows of its watering needs through the seasons, learn what its seed pods look like, and eventually watch it die through some misstep of my own or otherwise. And in doing so, I am learning, and building a mental model for this plant and an intuition for it, but I'm not "creating value" in some weird capitalist sense, which I feel always underlies these sorts of opinions about learning and education, and people who self-identify as "makers" in general. It rubs me the wrong way because it encourages a very narrow view of the human experience and what it means to learn and why we should learn.
- itsmefaz 7y ago
  An underrated comment!
newswasboring 7y ago
I'm not going to lie, his life story made me jealous. Extremely jealous. He did all the things I wanted to do (speaking in categories, not exact things) and is free to do more. It seems like in some cultures (mine is South Asian) there is a threshold on exploration time. Usually around the age of 28-30 years old (for some even lower than that, I consider myself one of the most fortunate ones). As I approach that number I feel the invisible hand of expectations and responsibilities crushing my spirit. But I must also remember comparisons on life scales don't really work and nobody can win neither the happiness Olympics nor the misery Olympics.
- Insanity 7y ago
  I think the limit is mostly set by having kids, and not necessarily age. My wife and I travel frequently (if you consider this exploration time, but I might have misunderstood). And in addition she is now going back to university for a third master after having worked for some time. We're about the age you describe, late-20s, early-30s. But our friends of around the same age with kids seem to have a lot less of this 'freedom'. Responsibilities take over. Not saying you can't do those things with kids - but it does seem harder.
- TrackerFF 7y ago
  What do you put into exploration? Here in Northern Europe you're expected to do most of your traveling in your 20's, but we also have pretty decent vacations - so the solo / friend / backpacking type of traveling gets replaced with more family friendly stuff when you start getting kids. I have lots of friends in their late 20's / early 30's that still travel the world, many times a year. But they don't have kids, and their travels (outside summer vacation) tend to be shorter, as in long-weekends etc.
  - newswasboring 7y ago
    It's not just about traveling. It's just one of the components. This guy was a teacher, musician, maker and now DL expert. And it doesn't seem like he is settled on that yet. He is still exploring what he wants to do. Changes his life to match his interests. I lived some of my life like that. Took some non standard paths. But it is getting increasingly hard for me to do this. Not because there aren't opportunities for me, but because it's getting unjustifiable to "society". And I know the usual line about "who cares what others think" in the west. Although it's logical in the west its not so in South Asia. Even if you don't care, your parents will and you will care for your parents. So it's inescapable. By Western standards I should be a totally free person, I am not married, don't have any debt, a high earning job which I'm bored by only 50% of the time. But these same things trap me. Single status leads to public derision in social gatherings (also friends get married and it becomes increasingly hard to be the only single guy in the group), my insistence on not taking on any debt means I live in a small rented apartment which it hard to be accepted in the society around me, a high earning job means I can never just quit because then I will have no social support due to it being a bad decision. As I write this, my (admittedly limited) understanding of how Western society works makes me think these would not be problems but my biggest assets.
itsmefaz 7y ago
The problem with Emil's approach to learning is that it restricts his ability to learn anything that he has no intrinsic goal off. That includes areas like pure mathematics, theoretical computer science, finance, economics, literature, etc. Those subjects require a different sort of motivation than a motivation to just achieve a set goal i.e capitalistic motivation Also, Emil's approach to learning will create a flawed sense of expertise. Look at how the article presents him as if he has a deep-domain expertise which might not be true. One important thing to consider is to look at the article more like content marketing tactic, that FloydHub is using promote its brand which might not serve well for engineers as it lacks some aspect of truth.
- ramblerman 7y ago
  It's not a zero sum situation. Him taking this approach does not restrict someone else from studying what they want. Also finance... really? I'm not sure I would agree that is a subject with a less capitalistic motivation. I don't get a lot of the bitterness here. I mean not everyone here creates their own programming language, or would know how to write a database, yet we are fine using them as tools. Why must we understand all the code and concepts in a neural network to apply it?
  - itsmefaz 7y ago
    Not necessarily bitterness! I'm very happy for what he is doing. I'm looking at it very objectively, the problem is that the way the article presents his story is very flawed and incorrect. It sounds very much like a PR post. To address the Finance point, there are few areas of finance that is purely research driven and requires a lot of expertise of lateral domain like economics, mathematics. And once cannot grok that expertise in a year or so.. That was the point I was making! > Why must we understand all the code and concepts to apply it? That is one of the reason why our engineers are so sub-par because we were told to just shut up and write shit. We could have become a force to be reckoned with because of our expertise, because of our ability to solve complex problems. Yet we are living in an industry were there is huge disparity in salary, structure, and principles. I don't necessarily agree with why one shouldn't understand the concepts, I'm more a guy who says why not? Because, if you are standing on the shoulders of pioneers and claiming to be improving their work at least do it with compassion and honesty. Sorry about the rant.. would love to hear alternative opinions!
ptah 7y ago
> deep learning internship at FloydHub. nice to be able to work for free and not starve
anentropic 7y ago
I think most of us here on this site could do the same in terms of learning and research Seems to me the difficult part is how to support yourself financially while spending your time doing interesting learning and research, or how to get paid to do it Maybe the most important detail in the story is "He co-founded a seed investment firm that focuses on education technology" but it is not discussed further

364 points by emilwallner 7y ago

This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. It requires a lot of trial and error and time/cost investment to bring the AI techniques to the masses. The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups). The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.

woah 7y ago
On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques
- SQueeeeeL 7y ago
  Low data techniques are just another name for algorithms/equations. Dijstras algorithm required 0 training graphs to make. Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)
  - ssivark 7y ago
    Agree with your first statement and disagree with your second; I don’t think the former implies the latter. I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.
    - btrettel 7y ago
      These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables. As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.
- TrainedMonkey 7y ago
  Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.
  - cygaril 7y ago
    Or our entire evolutionary history of data.
    - AlanSE 7y ago
      ...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.
      - eanzenberg 7y ago
        I’m not sure our genetics encodes all the physics of being a person. A human brain is so complex we’re not even close to simulating it on silicon
    - K0SM0S 7y ago
      It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us. The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself. It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).
- mendeza 7y ago
  I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation. Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reality_gap.pdf http://josh-tobin.com/assets/pdf/randomization_and_the_reali... http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_Tobin_RSS19.pdf http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T... And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText https://github.com/ankush-me/SynthText
  - tasogare 7y ago
    How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.
- gdubs 7y ago
  This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.
andreyk 7y ago
wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.
- K0SM0S 7y ago
  So if I understand correctly, to reformulate in my own words/views: while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal), fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough. Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
  - yorwba 7y ago
    It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone. On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.
  - andreyk 7y ago
    yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab (http://svl.stanford.edu/ http://svl.stanford.edu/) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.
z3t4 7y ago
> colorizing b&w photos You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.

qntty 7y ago

"Many are realizing that education is a zero-sum credential game." Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.

Nasrudith 7y ago
It is pretty strange even from an economist really - they of all people should be able to understand and articulate the difference between signaling value and direct utility value of a given good or service.
- koube 7y ago
  Economists have been debating the skills vs signaling value of education, especially since Bryan Caplan released his book The Case Against Education. If you want to get a smattering of opinion on the issue the book's reviews and dicussionsn would be a good starting point. https://en.wikipedia.org/wiki/The_Case_Against_Education#Reviews https://en.wikipedia.org/wiki/The_Case_Against_Education#Rev... Bryan Caplan back and forth with Noah Smith on the book: https://www.econlib.org/archives/2015/04/educational_sig_1.html https://www.econlib.org/archives/2015/04/educational_sig_1.h... Bryan Caplan back and forth with Bill Dickens on the book: https://www.econlib.org/archives/2010/08/education_and_s.html https://www.econlib.org/archives/2010/08/education_and_s.htm...
- Gimpei 7y ago
  It's not a majority view among economists. Caplan is the only person I can think of who holds this view.
  - jdminhbg 7y ago
    Caplan is definitely not the only economist who holds this view. Most place the signaling/human capital split around 50/50: https://www.econlib.org/archives/2011/11/kauffman_econ_b.html https://www.econlib.org/archives/2011/11/kauffman_econ_b.htm...

bluetwo 7y ago

The thing that disappoints me about the aspirations of being a researcher is that the goal is to get paid to study AI, not solve real-world problems. I would rather build a small company by solving a real problem than work for a big company spinning my wheels.

currymj 7y ago
for a lot of people who end up in research-type jobs, a sense of curiosity is one of their strongest motivators, and they want work that will let them pursue their curiosity. it sounds like you're motivated by something else.
- bluetwo 7y ago
  Well yes, I am very curious, just more motivated to solve problems.
hogFeast 7y ago
Why I didn't go into academia but GL convincing other people of the value in that. I am sure there are cultural differences here but where I am, the goal of most people who study CS is: leave me alone while I mess about with X (evidence: the local college was doing speech processing/nlp in the 60s, they actively turned down paid work...unsurprisingly, they got left in the dust, professors are now being encouraged into doing commercial work but, of course, most of it is totally nonviable and is just more messing about with complex nonsense that doesn't work). I think if you look at history this is also evident: the inventions of the late 18th century were a function of necessity, the invention of semis (not just in the US but how Taiwan developed)...this isn't to say academia is pointless but there is just far more going on (I think if you look at some of the East Asian nations that get great academic results, their progress on actual R&D innovation is far less impressive).
- ssivark 7y ago
  (American) Academia is a complicated matter, so I’ll elide commenting on that. For a thoughtful counterpoint to the necessity argument, see: https://jnd.org/technology_first_needs_last/ https://jnd.org/technology_first_needs_last/ (previously discussed on HN)

exdsq 7y ago

Survivorship bias or reality: 3 months learning FastAI, 3-12 months personal projects and consulting, 2 months flashcards of ~100 papers, 6 months to publish a paper What does he mean by ‘paper’? A Medium post? NeurIPS?

TrackerFF 7y ago
It could be that he's an exceptionally smart and driven guy, that just happens to pick up things really fast; I've seen those IRL myself, but not in only a year or two. But yeah, going from 6 months of programming experience with C, to a Deep Learning internship - that sounds a bit far stretched.
itsmefaz 7y ago
Submitting papers in conferences rather than journals.
- exdsq 7y ago
  Without submitting from a known university or research group? Seems unlikely to me. I have no doubt it can and has been done, but for it to be a regular thing such that one can recommend it in a 'Couch to 5k' style method? No way. I have some friends in Oxford who are DPhil/Postdocs in highly reputable research departments specializing in ML and if they sometimes struggle to get more than a poster session at the leading conferences, with the addition of well known professors names attached, then there's just no way I can believe Joe Bloggs who just learnt python 12 months ago is able to do the same. I almost want to follow his guide just to check.
  - itsmefaz 7y ago
    The idea is to choose conferences like InfoQ where application type research is accepted. Like Build-X-using-TensorFlow, it doesn't have to align with standard research which requires formal education. One also needs to be skeptical while reading such a PR post and not get swayed away by the hero's journey in it.

K0SM0S 7y ago

This was a great read (and great nuggets, like that paper on Intelligence by Chollet). I wonder: — Is math a problem for non-academic researchers? Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but math nonetheless. Not that you can't learn these on your own, but how big is the gap in practice, on the job, compared with actual PhDs in ML/math? (how much of a hinderance, a problem it is for the self-taught researcher) — "Contracting" in the field of AI sounds great but, how exactly? Especially solo: what type of clients and how/where to find them, what type of 'business proposition' as a freelancer do you offer, what's the pricing structure of such gigs? I mean, I can sell you websites and visuals and stuff, but AI? I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny and demand scripting skills to sort out (extract business value), not AI, so the value proposition is low for both parties; it's still early adoption so 90% don't even consider spending 1 cent on "AI" unless as a SaaS (they actually don't need to know if it's AI or programming). I can imagine tons of fantastic research to do with SMBs, as partners or 'interested sponsors' (should they reap benefits on a low investment), but really not much yet in the way of "freelancer products" to market and sell for a living. I'm eagerly anticipating those days, but it's more like 2025-2030 as I see it. I would love to hear first hand takes on this.

JamesBarney 7y ago
It's my understanding that dirty datasets that "demand scripting skills to sort out" is pretty common and most data scientists spend 80% of their time "sorting this out".
deepnotderp 7y ago
To be honest, linear algebra is not that difficult to learn on your own, and plenty of people do. Gilbert Strang's course on OCW has made introductory linear algebra quite accessible. Things like topology (e.g. TDA, persistent homology, etc.) aren't really mainstream yet, but even then most of it isn't really "hardcore" math in the sense that you can get away with a basic understanding, e.g. what a Vietoris-Rips complex is and why we use it instead of a Cech complex in TDA. Plus most DL research nowadays is pretty (advanced) math-light. That being said, taking the time to understand the math is absolutely worthwhile in my experience. It should also be noted that a lot of real world ML/AI projects in industry aren't really about brand new algorithms using advanced math, but rather more about applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.

LemonAndroid 7y ago

I don't see how this is self-taught, as the person got picked up for an internship and could learn from experts first-handly. FAKE.

curiousgal 7y ago
He also studied at 42, which is most likely why he got picked for the internship to begin with. I don't get this self-congratulating BS, guy says he toured the world (good luck doing that with a shitty passport) and was named king of a village in Ghana (right..). I guess people who get lucky have to always go to great lengths to justify and spin that. They're free to do so of course, but they should not give advice based on it. Regardless of that, I suppose the bar for being a "researcher" has been stooped so low. According to this guy publishing an ML paper is equivalent to writing a blog post or making a video about "AI".

rmah 7y ago

Is this guy actually a researcher in the way most people would think of it? That is, someone who pushes the boundaries of science; who develops new AI techniques or finds the hard boundaries of existing AI techniques; who finds new ways compose multiple AI techniques cohesively; who explores the theoretical foundations of AI. Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)? I can't help but wonder a bit.

ssivark 7y ago
For better or worse, the definition of researcher has morphed into a combination of 1. Solves previously unsolved problems 2. Publishes papers sharing those solutions without regard to the kind/spirit/scope of problems solved. Since conference publications don’t have the same number constraints as journal papers, and are accepting of application-specific results, this explosion of what is considered “research” is somewhat inevitable. Also, there are a lot of people chasing this given the prestige associated with the title.
- deleted 7y ago
  [deleted]
rlayton2 7y ago
Research needs people at the entire spread of the spectrum - from those making fundamental improvements to underlying theory, all the way to people running the thing to see if it works on actual problems people have (obviously in a robust and verifiable way).
anjc 7y ago
Most people would be wrong if they think that this is what all researchers do
throwawayjava 7y ago
Yes. From his GH profile looks like he's a competitive applicant for ML engineering positions or perhaps a fellowship/residency/PhD program. So, a junior researcher at the level of a decent second or third year PhD student. A researcher, maybe someone you'd trust to build a prototype or product, lots of potential, but probably not someone you'd trust to run a research program.

wigl 7y ago

This reeks of survivorship bias to me. I much prefer Andreas Madsen's more sober and self-conscious take on independent research [0]. > I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to self-fund your research. Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020? How many are going to place in a Kaggle competition with just half a year of experience? More importantly, just how many people are privileged/secure enough to put their all into learning, with no sense of security or peer support? > I started working with Google because I reproduced an ML paper, wrote a blog post about it, and promoted it. Google’s brand department was looking for case studies of their products, TensorFlow in this case. They made a video about my project. Someone at Google saw the video, though my skill set could be useful, and pinged me on Twitter. So what really mattered was self-promotion, good timing, and luck. > Tl;dr, I spent a few years planning and embarking on personal development adventures. They were loosely modeled after the Jungian hero’s journey with the influences of Buddhism and Stoicism. Why does the author have to present his life like one would in a fucking college essay? [0] https://medium.com/@andreas_madsen/becoming-an-independent-researcher-and-getting-published-in-iclr-with-spotlight-c93ef0b39b8b https://medium.com/@andreas_madsen/becoming-an-independent-r...

drongoking 7y ago
> So what really mattered was self-promotion, good timing, and luck. Yes. He seems like someone who is good at self-promotion and networking. Well, good for him, but I think he underplays the role these have in his success. > Why does the author have to present his life like one would in a fucking college essay? I guess that's the self-promotion. And humble-bragging. Like this bit: "I started working as a teacher in the countryside, but after invoking the spirit of their dead chief, they later annotated me the king of their village."
- wigl 7y ago
  > Well, good for him, but I think he underplays the role these have in his success. Exactly. Good for Emil, but it's always frustrating to hear survivorship bias preaching. Even the interviewer starts off by saying: "By the way, I really love your CV - the quirks section was especially fun to read." It's even more frustrating when I hear non-POC's talk about their journey to some non-western country (and subsequent conquering of fantastical goals like gaining the approval of locals) or pursuit of some sense of foreign culture. It's almost a given that they have internalized and appropriated the ideas (i.e. Buddhism or even worse post-retreat Buddhism). Good for the author to receive such positive feedback for such signaling, but it makes me sad to know that I might not receive the same.

ineedasername 7y ago

I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between: 1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research" 2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1. What might be termed "basic" or "pure" research. I'm not placing one of these above the other in terms of importance. They are both necessary, and they form a virtuous feedback loop between the two that, one without the other, would see the other wither on the vine. In the example of this particular person, Emil Wallner, he appears to be doing #1, and perhaps doing so in a way that might help inform more of #2.

JamesBarney 7y ago
I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
- rumanator 7y ago
  > Mind providing some concrete examples? It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications. To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
  - JamesBarney 7y ago
    I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical). But I was asking because I was specifically looking for concrete examples in deep learning.

octokatt 7y ago

Was anyone else really put off by the congratulatory tone of the article, and the #Quirks list on the resume?: https://github.com/emilwallner/Emil-Wallner-LinkedIn-Resume#quirks https://github.com/emilwallner/Emil-Wallner-LinkedIn-Resume#...

z3rgl1ng 7y ago
Yes. The writing style is unkind to the author.
TrackerFF 7y ago
You know, they (read: recruiters) say that if you don't have a normal background, you should have an interesting one. For some reason, if you don't have the same cookie-cutter background as everyone else, you need to have some amazing and convincing story to tell. I think it's good that companies are willing to look into non-trad candidates, that may not have found their "calling", so to speak, until their late 20's / 30's or whatever. But it does start to sound contrived when a bunch of 'em have the same type of alternate-route stories, which involves traveling to Africa / India / SE Asia to help out kids, create some startup aimed at climate / poverty / equality / etc. I guess it makes you sound passionate and legit - no-one can say that you wasted your time on chasing those things.
- octokatt 7y ago
  I guess it can help sound passionate, but the list together doesn't sound interesting -- it sounds like an AI read a bunch of minimalism lifestyle blogs and output "interesting_backstory.txt". Nothing on the quirks list is actually a quirk. They're interesting things he's done that other people wrote books about, received praise for, and then he followed their newer, well-traveled path. It's not a non-traditional background. He's not a refugee who managed to learn coding. He's not volunteering at a needle exchange clinic. I think that's what's bothering me; he's pretending to be interesting, and taking the room which could be going to someone else. Thank you for helping me get to why something felt off. Appreciated, internet stranger.
floodyberry- 7y ago

NWM123 7y ago

I personally found this article to be very interesting. I don't know much about AI, but I was fascinated by the discussion of peer to peer educational system. I believe that they will become more prevalent as student loan payments cause debt to so much of our population in order to get an education .

jshowa3 7y ago

I don't know why people think getting a credential does nothing or that people "copy and paste" the assignments. Sure it may be possible, but what prevents people from copying and pasting public git repos? Either way, this whole focus on "portfolios are everything and credentials are meaningless" spits in the face of all the work I did to get my university education. And it didn't involve "copying assignments". And you come out with one hell of a portfolio if you take your education seriously. I mean I don't think self-educated people are without merit. I happen to think they're really important. But I only ever see them rag on higher education, despite them having "never been there". Just another example of wunderkin super genius knows all because he was able to follow a non-standard path and make it. Glad he was smart enough to become a Google employee. But I question whether he should be giving advice on paths to get there when there's always many paths to a position. And especially after reading his brief comments on how credentials imply you're a liar.

ynx 7y ago
yeah, as someone who had a partial education, I totally get the value of a degree, so whenever someone says "higher education is useless" I read it as "However successful I am now, I wasn't the kind of person who would succeed in school then"
- randomdata 7y ago
  Whenever I hear someone say "higher education is useless" I hear it in reference to the "common wisdom" that higher education increases incomes. In which case they are absolutely correct. Higher education is useless in achieving that. Incomes have held stagnant for many decades, even as more and more of the population attain higher levels of scholastic achievement. Mathematically, incomes cannot not remain stagnant if more money is earned as a result of attaining higher education. I'm not sure I have heard anyone claim that "higher education is useless" in general. Education is never useless in general.
  - jshowa3 7y ago
    Statistically, higher education does increase incomes when factoring in multiple disciplines. The data is pretty clear on this.

DoctorOetker 7y ago

This is a great example of how we "collectively" [1] conflate phenomena, skillsets, ... into one topic: machine learning, AI. 1) There is the general phenomena or collective project, where hardware, algorithms and human insights are improved to approach the situation of man-made intelligent machines. 2) There are the people who are designing algorithms, using mathematical intuition and knowledge, analogies with physics, etc... Most people would agree these people are doing optimization / machine learning "proper". 3) There are the people working on improving hardware for machine learning / optimization purpouses, by looking at the most performant algorithms, breaking them down into primitive operations and requirements for hardware, there are also people working on the algorithms themselves and finding computational shortcuts (which can end up in software or hardware, can end up as proprietary knowledge or common knowledge, ...). The distinction between hard and software is somewhat blurry, since hardware designers can optimize or implement a section of software into hardware. A lot of this can still be considered ML "proper". 4) Then there are the people who apply the ML frameworks and their exposed choices and settings to a specific problem domain. Many of them don't need to understand the internals if they don't need state of the art results. Many would nevertheless benefit from understanding the internals, and the requisite math. What I propose is to stop calling their activity as Machine Learning, and instead call it Machine Teaching. They are teachers, and just like elite schools they can choose which specific type of available student they will teach, and they can tweak (or filter from a large family of students) which student they select to teach the task at hand. There are bound to be many advantages of having actual human teachers get involved in machine teaching. These people will not be proficient in designing novel families of students unless they also know the requisite math, and identify those ML papers that are ML "proper" instead of ML "teacher". When trying to find important foundational insights in ML "proper" one is typically overwhelmed by a large surplus of ML "teacher" type papers. These are important datapoints, and necessary to advance human insight into ML "proper", but they are data, not knowledge. There are actual ML "proper" knowledge papers out there that explain why a certain phenomena is such and so, and they get very little attention because they necessarily lag the breakthrough ML datapoint paper, and most ML "teachers" don't have the math background to understand them. So the probability that a given ML "proper" researcher fundamentally improves the state of the art is much higher than the probability that a given ML "teacher" will fundamentally improve the state of the art. At the same time the probability that a given fundamental breakthrough was achieved by an ML "teacher" is higher than the probability that a given fundamental breakthrough was achieved by an ML "proper" researcher: P( Breakthrough | Proper ) > P ( Breakthrough | teacher) while P ( Teacher | Breakthrough ) > P ( Proper | Breakthrough ) Since most people don't have the broad math / physics / ... knowledge to draw on, the number of ML "teachers" is much higher than ML "proper" researchers. [1] well, really, some actors have vested interests in conflating those together... EDIT: just to be clear, I am not complaining about ML Teachers, we need the ML Teachers, and their breakthrough datapoints. What I am complaining about, is conflating both activities of ML Proper and ML Teaching. This makes it harder for the few ML Proper researchers to find each other's insights.

vector_spaces 7y ago

> Creating value with your knowledge is evidence of learning. I see learning as a by-product of trying to achieve an intrinsic goal, rather than an isolated activity to become educated. > Early evidence of practical knowledge often comes from usage metrics on GitHub, or reader metrics from your work blog. Progress in theoretical work starts by having researchers you consider interesting engage with your work. > Taste has more to do about character development than knowledge. You need taste to form an independent opinion of a field, having the courage to pursue unconventional areas and to not get caught up in self-admiration. When I study abstract interpretation or lattices, I'm doing so because I find those subjects interesting and beautiful, and studying math relaxes me. I can lie to myself and say that it's improving my problem solving ability and that it's like doing mental yoga and will make me better at my job or some baloney, but that's not why I do it. I can spend time with a plant in my garden, take a cutting, root it and replant it, and watch it grow, learn the ebbs and flows of its watering needs through the seasons, learn what its seed pods look like, and eventually watch it die through some misstep of my own or otherwise. And in doing so, I am learning, and building a mental model for this plant and an intuition for it, but I'm not "creating value" in some weird capitalist sense, which I feel always underlies these sorts of opinions about learning and education, and people who self-identify as "makers" in general. It rubs me the wrong way because it encourages a very narrow view of the human experience and what it means to learn and why we should learn.

itsmefaz 7y ago
An underrated comment!

newswasboring 7y ago

I'm not going to lie, his life story made me jealous. Extremely jealous. He did all the things I wanted to do (speaking in categories, not exact things) and is free to do more. It seems like in some cultures (mine is South Asian) there is a threshold on exploration time. Usually around the age of 28-30 years old (for some even lower than that, I consider myself one of the most fortunate ones). As I approach that number I feel the invisible hand of expectations and responsibilities crushing my spirit. But I must also remember comparisons on life scales don't really work and nobody can win neither the happiness Olympics nor the misery Olympics.

Insanity 7y ago
I think the limit is mostly set by having kids, and not necessarily age. My wife and I travel frequently (if you consider this exploration time, but I might have misunderstood). And in addition she is now going back to university for a third master after having worked for some time. We're about the age you describe, late-20s, early-30s. But our friends of around the same age with kids seem to have a lot less of this 'freedom'. Responsibilities take over. Not saying you can't do those things with kids - but it does seem harder.
TrackerFF 7y ago
What do you put into exploration? Here in Northern Europe you're expected to do most of your traveling in your 20's, but we also have pretty decent vacations - so the solo / friend / backpacking type of traveling gets replaced with more family friendly stuff when you start getting kids. I have lots of friends in their late 20's / early 30's that still travel the world, many times a year. But they don't have kids, and their travels (outside summer vacation) tend to be shorter, as in long-weekends etc.
- newswasboring 7y ago
  It's not just about traveling. It's just one of the components. This guy was a teacher, musician, maker and now DL expert. And it doesn't seem like he is settled on that yet. He is still exploring what he wants to do. Changes his life to match his interests. I lived some of my life like that. Took some non standard paths. But it is getting increasingly hard for me to do this. Not because there aren't opportunities for me, but because it's getting unjustifiable to "society". And I know the usual line about "who cares what others think" in the west. Although it's logical in the west its not so in South Asia. Even if you don't care, your parents will and you will care for your parents. So it's inescapable. By Western standards I should be a totally free person, I am not married, don't have any debt, a high earning job which I'm bored by only 50% of the time. But these same things trap me. Single status leads to public derision in social gatherings (also friends get married and it becomes increasingly hard to be the only single guy in the group), my insistence on not taking on any debt means I live in a small rented apartment which it hard to be accepted in the society around me, a high earning job means I can never just quit because then I will have no social support due to it being a bad decision. As I write this, my (admittedly limited) understanding of how Western society works makes me think these would not be problems but my biggest assets.

itsmefaz 7y ago

The problem with Emil's approach to learning is that it restricts his ability to learn anything that he has no intrinsic goal off. That includes areas like pure mathematics, theoretical computer science, finance, economics, literature, etc. Those subjects require a different sort of motivation than a motivation to just achieve a set goal i.e capitalistic motivation Also, Emil's approach to learning will create a flawed sense of expertise. Look at how the article presents him as if he has a deep-domain expertise which might not be true. One important thing to consider is to look at the article more like content marketing tactic, that FloydHub is using promote its brand which might not serve well for engineers as it lacks some aspect of truth.

ramblerman 7y ago
It's not a zero sum situation. Him taking this approach does not restrict someone else from studying what they want. Also finance... really? I'm not sure I would agree that is a subject with a less capitalistic motivation. I don't get a lot of the bitterness here. I mean not everyone here creates their own programming language, or would know how to write a database, yet we are fine using them as tools. Why must we understand all the code and concepts in a neural network to apply it?
- itsmefaz 7y ago
  Not necessarily bitterness! I'm very happy for what he is doing. I'm looking at it very objectively, the problem is that the way the article presents his story is very flawed and incorrect. It sounds very much like a PR post. To address the Finance point, there are few areas of finance that is purely research driven and requires a lot of expertise of lateral domain like economics, mathematics. And once cannot grok that expertise in a year or so.. That was the point I was making! > Why must we understand all the code and concepts to apply it? That is one of the reason why our engineers are so sub-par because we were told to just shut up and write shit. We could have become a force to be reckoned with because of our expertise, because of our ability to solve complex problems. Yet we are living in an industry were there is huge disparity in salary, structure, and principles. I don't necessarily agree with why one shouldn't understand the concepts, I'm more a guy who says why not? Because, if you are standing on the shoulders of pioneers and claiming to be improving their work at least do it with compassion and honesty. Sorry about the rant.. would love to hear alternative opinions!

ptah 7y ago

> deep learning internship at FloydHub. nice to be able to work for free and not starve

anentropic 7y ago

I think most of us here on this site could do the same in terms of learning and research Seems to me the difficult part is how to support yourself financially while spending your time doing interesting learning and research, or how to get paid to do it Maybe the most important detail in the story is "He co-founded a seed investment firm that focuses on education technology" but it is not discussed further