15 ms·

Ask HN: How to get back into AI?

266 points by quibono 4y ago

I was involved in machine learning and AI a few years ago, mainly before the onset of the new diffusion models, large transformers (GPT*), Graph NNs and Neural ODE stuff. I am comfortable with autograd/computation graphs, PyTorch, "classic" neural nets and ones used for vision-type applications, as well as the basics of Transformer networks (I've trained a few smaller ones myself) and RNNs. Do you know of any good resources to slowly get back into the loop? So far I plan on reading through the original Diffusion/GPT papers and start going from there but I'd love to see what you think are some good sources. I would especially love to see some Jupyter notebooks to fiddle with as I find I learn best when I get to play around with the code. Thank you

greggarious 4y ago
Find the paper that got you into that (or anything else) then see who's cited it since and catch up that way.
optbuild 4y ago
> I am comfortable with autograd/computation graphs, PyTorch, "classic" neural nets and ones used for vision-type applications, as well as the basics of Transformer networks (I've trained a few smaller ones myself) and RNNs. Maybe I am a bit off the track. But how do someone reach this state?
- newsoul 4y ago
  You can try out fast.ai's courses. Also there is a youtube series by Andrej Karpathy that goes into much more detail and builds a library from scratch.
yonz 4y ago
In the same spirit, what are the three learning algorithms to implement for learning? My to-do list has: 1) Refresher: RNNs, deep net classifier 2) LSTMS 3) Self Attention anything Any other suggestions for getting back in the loop?
jstx1 4y ago
> I am comfortable with autograd/computation graphs, PyTorch, "classic" neural nets and ones used for vision-type applications, as well as the basics of Transformer networks (I've trained a few smaller ones myself) and RNNs. It sounds like you're into it already. And you already know which new papers are interesting to you.
sigmoid10 4y ago
To be honest, for transformers just go to huggingface.co and see what interests you. They have tons of examples to run and they also link to all the papers in the documentation. It doesn't get much easier to get into it. Even for the more recent stuff like vision transformers and diffusion models.
- jerpint 4y ago
  The “hard” part is taking an example and adapting it to new tasks. This is the best way to learn imo
anonreeeeplor 4y ago
I think you should be careful about dropping whatever you are doing and running back to this new iteration of AI. Quite honestly, the opportunity all seems to be on the front end. The idea that you are going to airdrop yourself as a hands on AI programmer into this market doesn’t make a huge amount of sense to me from a career perspective. The opportunity is with the tools and how they are applied. Building front end experiences on ChatGPT and integrations and applied scenarios. You actually doing the AI yourself means competing with PHDs and elite academics immersed in the field. I think knowledge of AI is far less valuable than knowledge of the emerging landscape combined with a broad understanding of different tools and how they are applied. The new trend here is very strongly Large Language Models (LLM). You should be far more specific with what your goal is and where to spend your time. A lot of the “AI” you are referring to seems to be no longer relevent or interesting to the market. If you are spending time with Jupiter notebooks I would say you are probably completely wasting your time and heading in the wrong direction. LLM is the major trend. Focus entirely on that and the tools landscape and how to integrate it and apply it. It feels like you are navigating using an out of date map.
- cbsudux 4y ago
  This is a good reply, but the OP might be generally interested in the fundamentals of ML
- steve_adams_86 4y ago
  Is LLM even applicable to many things though? If we want more nuanced and contextual vision, does LLM help?
  - jerpint 4y ago
    Fine tuning models is where the future will be, it doesn’t necessarily mean LLMs but I suspect that LLMs will shortly become multi modal and at that point they will be the ultimate models to fine tune for a given task
    - saliagato 4y ago
f0e4c2f7 4y ago
This should jog your memory and catch you up with the latest and greatest. https://course.fast.ai https://course.fast.ai
- nestorD 4y ago
  This is the course I used last time I wanted to get up to date, I would recommend.
turkeygizzard 4y ago
Similar situation as you. I stopped keeping up around 2015ish. I just read Francois Chollet's Deep Learning with Python and found it to be a fantastic high level overview of all the recent progress. There's some code, but not a lot. I mostly just appreciated it as a very straightforward plain-language treatment of RNNs, CNNs, and transformers. Now I'm going through Stanford's CS224 lectures. I'm sort of planning to read papers but as some other comments have pointed out, I'm less sure of the ROI on that since I'm not sure how feasible a future in AI is for me
macrolime 4y ago
I'm in the same boat kinda, but even more outdated on some parts. I had some AI specialization back in college, but that was before deep learning was even a thing, so we did stuff like self-organizing maps and evolutionary algorithms, but it wasn't really all that useful for much back then. Been following deep learning from the sidelines, but for work my AI work has been restricted to GOFAI until recently. Some of the stuff I'm currently reading/watching or have recently Practical Deep Learning, though it sounds like you may know this stuff already (https://course.fast.ai/ https://course.fast.ai/) Practical Deep learning part 2, more about diffusion models. Full course coming early next year (https://www.fast.ai/posts/part2-2022-preview.html https://www.fast.ai/posts/part2-2022-preview.html) Hugging Face course (https://huggingface.co/course/chapter1/1 https://huggingface.co/course/chapter1/1) Diffusion models from hugging face https://huggingface.co/blog/annotated-diffusion https://huggingface.co/blog/annotated-diffusion https://huggingface.co/docs/diffusers/index https://huggingface.co/docs/diffusers/index Andrej Karpathy's Neural Networks: Zero to Hero. He goes from the basics up to how GPT etc, so you can start wherever suits you (https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...) 3blue1brown's videos. I've found all his videos on neural networks and math worth watching, even for stuff that I already know, he sometimes has some new perspectives and nice animations. brilliant.org. Nice math refresher and the courses there are almost like fun little games.
- Railsify 4y ago
  Have you had a look at https://nnfs.io/ https://nnfs.io/ ? I bought the book and am gearing up to start working through it, I would be interested to know your thoughts. Generally I want to chart a personal curriculum from data engineer to practical application of modern AI to real business problems.
  - macrolime 4y ago
    I hadn't seen that book before. Looks like it's a companion to this YouTube series that looks pretty interesting https://youtube.com/playlist?list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3 https://youtube.com/playlist?list=PLQVvvaa0QuDcjD5BAw2DxE6OF... I will check it out for sure
CoastalCoder 4y ago
What's your goal? For example, do you want to develop models as a hobby? Make models or software for a living? Use AI in some particular problem domain?
- quibono 4y ago
  I meant more as a hobby, though I would love to practically apply some of these models in a little side project.
  - CoastalCoder 4y ago
    Everyone is different, but I'd have trouble making time for a project like this unless it served some goal I truly cared about. If you're the same, is there something more specific to help you focus and put in the time?
mythhouse 4y ago
How are you software engineering skills. Thats the biggest gap i currently see at my current employer. Way too many data scientists not able to make an impact because they are not able to put their notebooks into a product and run it in prodction.
- grepfru_it 4y ago
  I see this too. So many well intended projects that were scrapped because management didn't see a business case from jupyter notebook results...
- civilized 4y ago
  I partly agree with this but I think it's a bit overrated. There's no massive barrier to getting an actually good model into production. If the model seems promising but no one can figure out how to productionize it, it probably has fatal flaws as a model. There's no guarantee that a mess of Python code that somehow produces a nice AUC curve is actually doing anything valuable. As in science in general, there are many ways to fool yourself in data science.
  - mythhouse 4y ago
    I am not saying its a hard skill or anything. But asking your manager everytime to go find someone to put your notebooks in productions is not good strategy to succeed at work. If you can get your notebook pretty far into production and have software engineering skills it goes a long way.
    - civilized 4y ago
      I completely agree, I think we're saying the same thing. I guess I'm just emphasizing that it's hard to believe that such a person is even a good data scientist. Throwing everything over the wall is denying yourself a lot of valuable lessons about data science.
      - oarabbus_ 4y ago
stephc_int13 4y ago
Read all the leading papers, many times, to get a deep understanding, the writing quality is usually pretty low, but the information density can be very high, you'll probably miss the important details the first time. Most medium and low-quality papers are full of errors and noise, but you can still learn from them. Get your hands dirty with real code. I would take a look at those: https://github.com/geohot/tinygrad https://github.com/geohot/tinygrad https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
- zone411 4y ago
  Disagree. The information density of 99% of ML research papers is not high. It's usually just one idea that's often just an evolutionary modification of something that already exists, or something that worked in a different ML area applied here, along with an intro, maybe some supporting math, and some experiments. The information density might seem high if you're not familiar with previous papers that are referenced. I also don't agree that medium quality (how to decide what's medium quality anyway? includes main conferences?) papers are full of errors.
  - stephc_int13 4y ago
    This comes from my experience with Computer Vision papers about ten years ago. I've done some work in the field, and I've seen errors in almost all papers. Sometimes they are not really errors, but they give a solution that only works in the very specific context they tested it and completely falls apart anywhere else. I've lost countless hours and days studying and replicating papers. Of course, ML could be an entirely different experience.
    - zone411 4y ago
      I do expect that it was different back then. The replication situation has improved somewhat lately but it's still not great.
axpy906 4y ago
Following. I got off the train about two years ago to work more in engineering. The way I see it, if you’re not a research scientist - this field is best addressed as an ML engineer as there are more challenges in systems. Would love to be proved wrong.
- curious-guy 4y ago
  Hey, I am a research engineer/applied ai person building vision models in healthcare domain. I am currently preparing to transition to engineering roles like you did. For that, I am currently going through web dev - both frontend and backend. Would love to get some pointers from you on my approach and any recommendations from your side. Thanks!
  - axpy906 4y ago
    The best advice I could give: Build out your portfolio so you have projects that you can speak to and show your tech depth. FWIW, I do AI backend at FAANG and mentor as tech lead.
bobleeswagger 4y ago
Maybe it's just me, but it sounds like you're only interested in getting back into AI because it is currently in the limelight. "Working in AI" a few years ago doesn't mean much if you have nothing to show for it. What I'm getting at, is your motivations don't seem genuine; there's nothing that tells me you care about the technology advancing more than a paycheck. As someone with 10 years of professional experience in software, I find every AI "trend" that has come up in that time to be incredibly odd. It is certainly remarkable what chatGPT, StableDiffusion, and other examples are doing today... Ultimately people are giving waaaaaaaay too much credit without understanding the technical details. These are pidgeon-holed examples that still aren't solving any real problems. AI is still just statistics with marketing.
- spyder 4y ago
  Huh, WTF? What's wrong in getting interested in something because there is new progress in it? And how can you assume he is only interested in it because of the money when he only asked about how to learn, play around with code and didn't ask anything about finding a job or projects to work on?
  - bobleeswagger 4y ago
    It's disingenuous. There's a difference between getting into a topic because it interests someone, and getting into a topic because it's incredibly hyped right now. Do you really believe it's the former...? Considering OP mentioned a lot of recent hype trains relating to AI, it's perfectly reasonable to assume they are chasing dollars, rather than contributing to the underlying technology without further expectations. So, OP comes off as very disingenuous when they lead with the hype and provide nothing of substance.
- joenot443 4y ago
  On the side, I make cheap Wordpress sites for small businesses (mostly as a favor to folks I know personally) and always had a hard time writing good marketing copy. I’ve found that what ChatGPT comes up with is often a great start and has already saved me hours of time. Is it as good as paying a professional? Probably not. But I think it’s fair to say these models are already solving real world problems, even if they still need a bit of a helping hand. Just my thoughts, I’m not an expert on the models themselves.
marban 4y ago
Related question, what's the current state of paraphrasing text with off-the-shelf Python libs — PyTorch/Transformers?
- rg111 4y ago
  Very easy to do. Just use Hugging Face.
  - marban 4y ago
    Yeah, last time I played around with it was something like FB/Bart-CNN but I was under the impression that things have evolved/changed a lot since then.
  - Der_Einzige 4y ago
    Please give an actual answer next time. The OP wants specifically sentence-transformers, which are hosted on huggingface but it's a separate library. You'll want to go with whatever models that the sentence transformers documentation recommends.
deleted 4y ago
[deleted]
marban 4y ago
For Python, here's a nice compilation: https://github.com/ml-tooling/best-of-ml-python/blob/main/README.md https://github.com/ml-tooling/best-of-ml-python/blob/main/RE...
mathgladiator 4y ago
I'm curious what you discover as I did some AI decades ago, and now I have a new AI problem. I'm trying to research how to build a generic self learning board game agent for my platform ( https://www.adama-platform.com/ https://www.adama-platform.com/ ) as I've reduced the game flow to a decision problem. How I intend to start is to experiment with simple stuff at a low level, and then use that experience to find out what to buy.
quibono 4y ago
OP here. For some context, something I should have mentioned in the original post but failed to do: I was not intending to do a professional pivot to an AI role; it is more of a personal interest. I used to be really excited about this stuff and am looking forward to getting involved in it again just because I find it interesting. Thank you, I really appreciate everyone's responses.
- godelski 4y ago
  I'm not sure why people are telling you professional stuff. You do you. I am an ML researcher working on generative modeling. I think you have enough experience that you'll catch up quickly. But the question is most what you're interested in? With that I can give better advice. Don't let anyone stop you from learning just to learn. Not everything has to be a career. A lot of us got here because it's fun. I do think you'll pick up diffusion models quickly. I like the explicit density side of things more and density estimation. So I like Song's works and similar with Kingma. Also check out Lilian Wang's blogs. They are a wealth of material and sources. Can't go wrong there. You'll find that diffusion and VAEs are kinda similar. The difficulty you'll have in understanding something like stable diffusion is actually the programming (at least this was the hardest part for me). Good luck and let me know if I can help.
- maurits 4y ago
  Im an ML researcher (reinforcement learning). I learned by implementing papers from scratch endlessly. Any specific subfield you are interested in?
  - optbuild 4y ago
    What kind of programming and math proficiency is good to have before someone starts to implement papers?
Simon_O_Rourke 4y ago
If you're something of a snakeoil salesman with some, but not deep, technical knowledge, then AI Ethics is for you. There are (or were) companies out there who would pay big bucks for folks to tell them some models potentially could be biased or otherwise discriminatory because of poorly selected data. Wash, rinse, repeat and see your paychecks roll in.
29athrowaway 4y ago
I recommend this video from Andrew Ng: https://www.youtube.com/watch?v=avoijDORAlc https://www.youtube.com/watch?v=avoijDORAlc
usgroup 4y ago
Do you guys expect DL to have longevity generally speaking, or is the usurper already on the horizon? There are a few non equivalent universal approximation approaches. I’m not sure I fully understand why this is will end up being the one even on a 10 year horizon.
- qorrect 4y ago
  By Deep Learning you mean neural networks with more than one hidden layer ? I've seen some results from massive random forests that are on par, but I don't know of anything that is set to replace the NN architecture, which at this point has been around in one form or another since 1957.
alexmolas 4y ago
For diffusion models I recommend you this blog post https://eugeneyan.com/writing/text-to-image/ https://eugeneyan.com/writing/text-to-image/
matchagaucho 4y ago
Similarly, my linear regression and decision tree ML skills are feeling really outdated :) But understanding AI fundamentals gives me a fresh perspective on how to build applications that leverage ChatGPT (for example). Crafting the inputs to achieve desired outputs. Training the models with a corpus of data relevant to a niche industry, etc...
ttul 4y ago
I signed up for Jeremy Howard’s second AI course at the University of Queensland. The course lectures were streamed live to participants around the world in October and November. An online forum organizes everything. When this course becomes public next year, I think it will be a great way to get caught up. In the meantime, you might still be able to pay the AU $500 fee and watch the course content, which was all recorded, if you are anxious to get going.
zone411 4y ago
Start with survey papers. Starting with GPT is jumping in the middle. It's better to know the big picture first.
- wbsun 4y ago
  +1 to this. This is the way to start a PhD research (assuming OP is trying to stay up-to-date in this area by "get back").
ineptech 4y ago
Possibly too basic for you, but I was hugely impressed with https://www.nlpdemystified.org/course https://www.nlpdemystified.org/course (found on HN a week or two back). Each chapter has a large jupyter notebook with lots of annotated sample code. Also there's no cost, no book to buy, no email signup, it's just a guy sharing knowledge like the old days. Great course.
sjkoelle 4y ago
Personally I think that contributing to an open source community is the move. Join the Eleuther Discord. Futz around on Hugging Face. Play with notebooks on Uberduck. Have fun!!! Gatekeeping is dumb.
- Der_Einzige 4y ago
  Eleuther has many members who are very difficult to work with. Also, if they ever make another dataset like the pile, don't expect to get your own data into it (even if they say that they will add it). I'd work with other groups if I were you.
deleted 4y ago
[deleted]
naruvimama 4y ago
nektro 4y ago
don't
uoaei 4y ago
AI is so much more than the eye-catching generative art experiments.
jdeaton 4y ago
I am an ML researcher working in the industry: by far the most effective way to maintain/advance my understanding of ML methods is implement the core of an interesting paper and reproduce (some) of their results. Completing a working implementation really forces your understanding to be on another level than if you just read the paper and think "I get it". It can be easy to read (for example) a diffusion/neural ode paper and come away thinking that you "get it" while still having a wildly inadequate understanding of how to actually get it to work yourself. You can view this approach in the same way that a beginner learns to program. The best way to learn is by attempting to implement (as much on your own as possible) something that solves a problem you're interested in. This has been my approach from the start (for both programming and ML), and is also what I would recommend for a beginner. I've found that continuing this practice, even while working on AI systems professionally, has been critical to maintaining a robust understanding of the evolving field of ML. The key is finding a good method/paper that meets all of the following 0) is inherently very interesting to you 1) you don't already have a robust understanding of the method 2) isn't so far above your head that you can't begin to grasp it 3) doesn't require access to datasets/compute resources you don't have of course, finding such a method isn't always easy and often takes some searching. I want to contrast this with other types of approaches to learning AI with include - downloading and running other people's ML code (in a jupyter notebook or otherwise) - watching lecture series / talks giving overviews of AI methods - reading (without putting into action) the latest ML papers all of which I have found to be significantly less impactful on my learning.
- Throwaway23459 4y ago
  Most recent papers, in NLP at least, are so sparse on detail that it is impossible to reproduce their models. And then there's the compute cost, as at least one other poster has mentioned.
- janalsncm 4y ago
  Can you recommend some papers which fit those criteria?
  - HybridCurve 4y ago
whiplash451 4y ago
The latest development in AI are very cool, but a big lure in my opinion. You have two options: 1. Work full-time for companies doing this state-of-the-art stuff (OpenAI, Meta, etc.) 2. Work full-time for a (good) AI company that is doing interesting AI work, but most likely not based on GPT/SD/etc. In both cases, you will learn a lot. Anything else seems like a costly and dangerous distraction to me.

joxel 4y ago

Applied AI/ML is still a great career field, especially if you have a knowledge of physics.

drtm 4y ago
Hi, would you mind elaborating, or throwing some links my way regarding the physics connection in particular? I am a physics postdoc that has been interested in the field for some time (this becoming less of a unique characteristic as the technology develops and hype cycles peak). I was motivated by Stuart Russell's Reith Lectures in 2021 to pursue the field outright and had been working towards this, but am becoming increasingly aware that seeking direct research involvement in the field is a bit quixotic starting from where I am.
- joxel 4y ago
  check my reply to the other comment
  - drtm 4y ago
    Great! Thank you for taking the time. With that information I can look into a more local equivalent.
- mellosouls 4y ago
  I was motivated by Stuart Russell's Reith Lectures in 2021 to pursue the field outright For anybody intrigued, here are those Reith Lectures: https://www.bbc.co.uk/programmes/m001216k/episodes/player https://www.bbc.co.uk/programmes/m001216k/episodes/player https://wikipedia.org/wiki/Reith_Lectures https://wikipedia.org/wiki/Reith_Lectures
  - drtm 4y ago
    Thanks! The key point for me was that Russell put forward a strong case that the moral approach to the problems of artificial intelligence was to involve oneself with the evolution of these technologies where you can (8000hours style) and seek (in whatever limited capacity one can) to guide them away from the excesses and damages of the invisible hand (and the human condition it is a proxy for). This would have been influential to me when I was a teenager and nihilistic about accelerating technology in the hands of distinctly-not-developing humankind (reading all about the Manhattan-project forebears to AI), but after devoting a decade to science and being fascinated yet boycotting of AI research and it's implications, I think it has ended up a bit too late for a change of heart and direction to have as much positive impact. For those fortunate enough to be in a position of even trivial influence over the cutting-edge, and whose moral sophistication can therefore matter more than the next person, I hope you don't take the implications of whatever agency you may have lightly!
    - Der_Einzige 4y ago
      Anyone with even that kind of "trivial influence" can be absolutely sure than an "agency" is constantly tracking them. I wouldn't be surprised if AI conferences have a minimum of 10 spooks from every major western intelligence agency in attendence.
- zone411 4y ago
  One connection I've seen is energy-based models. I suppose you could try applying ML to things like fluid dynamics? In general, it is thought that physics knowledge transfers better to ML than math because it's less abstract and physicists are more likely to be used to dealing with large datasets and software.
  - drtm 4y ago
    Thanks for the insight! I have some friends that do computational fluid dynamics, with particle physics being similarly numerical, and was looking at physics-informed ML for my own particular area in quantum physics in a recent grant application in the hopes for funding to close the gap a bit myself. What is so powerful about ML and related statistical techniques is their versatility and genericity, so a project that can be benefited by that region of statistics tends not to be too far away. I will look into energy-based models, too.
    - joxel 4y ago
      His comment is quite good, I do work with physics informed ML for cfd and other dynamical systems (temperature, hydrology etc.), there is just a ton of opportunity and funding for this type of work in research. Coming from a typical physics education, where you’re learning quantum and Astro, and realizing that 90% of the physics funding from government is in the earth sciences and the related physics was eye opening. I felt shortchanged by my physics education not even including fluids etc.
      - drtm 4y ago
        It was the same here - fluid dynamics was an elective at my university as well (one I took, but still not core syllabus). I guess amount of funding for a domain depends strongly on impact, and in the earth sciences output is much more immediately tangible than uncovering another supremely true but at-the-time inapplicable pattern of physical behaviour in the quantum, or context to humanity in the astronomical or cosmological domains.
barefeg 4y ago
Could you elaborate? I’m in the AI field myself but as a software engineer. But have a PhD in physics
- joxel 4y ago
  https://www.usajobs.gov/Search/Results?k=physical%20scientist&p=1 https://www.usajobs.gov/Search/Results?k=physical%20scientis... If you're already in the field you'd likely have to take a pay cut, but the federal government is always interested in research physical scientists and will snatch up anyone who knows the physics and also knows ML.

synapticpaint 4y ago

I wouldn't say that "the opportunity all seems to be on the front end". Specifically for stable diffusion, there are a lot of different ways to use the model. I think we're just starting to scratch the surface of what SD can do, so there is some value in tinkering with different ways to use and apply the model. Example 1: have a look through here: http://synapticpaint.com/dreambooth/browse/ http://synapticpaint.com/dreambooth/browse/ for some examples of dreambooth models people have created Example 2: you can merge different dreambooth models together to varying degrees of success (the idea being, you train model A on subject A, model B on subject B, and now you want to generate pictures of A and B together). My understanding is that this doesn't work too well at the moment, but it's possible that a different interpolation algorithm can yield better results. I do agree with the general sentiment that you wouldn't necessarily be training your own models or creating your own architecture, just want to provide the perspective that understanding the AI side is valuable because it can lead to different capabilities and products.

akhmatova 4y ago
http://synapticpaint.com/dreambooth/browse/ http://synapticpaint.com/dreambooth/browse/ Can you explain the value of these models, please? This is a serious question. My eyes just see one horrendously ugly (and dystopian) eyesore after another.
- nuancebydefault 4y ago
  I agree that the current produced images by say SD, are more of a curiosity than true art. Give it a year or so, I would say, and we will change our mind. Remember the early VR models? Not comparable with the quality you have now in real time. Only AI seems to increase in quality of output at a much faster pace.
bsenftner 4y ago
I'd say the "opportunity" is primarily in creating business models with these new AIs. The AI field is going to be innovating on them regardless what any individual chooses to do. Discussions about viable applications of these new capabilities are scant, beyond reducing existing technical artist head counts I see little to no discussion about new capabilities and new applications not possible before. Sure, we're developers, but we're also supposed to be entrepreneurs and this lack of creative discussion about what can be done that was not possible before is curious in itself.
- fsloth 4y ago
  Guess based on historical anecdata: I think that is because it looks like the current gen AI:s will help automating lot of stuff, but wont enable anything new. I would say we are at the analogous point where ’computer’ no longer meant a chain of humans with calculators. First the automation needs to become entrenched, then new innovations can emerge.
  - bsenftner 4y ago
    Can I future quote your "current gen AI's will help automating lot of stuff, but won't enable anything new" when that "something new" tears a new economic hole in our global economy? Personally, I see this tech capable of destroying and recreating the advertising industry completely via inserting everyday consumers into ad media, depicting them as happy consumers of a product they've not used yet - while celebrity spokespeople, appearing as their personal friend, inform them how much the celebrity idolizes them for using said product. This is an obvious non-subtle application. There will be many, many more.
    - fsloth 4y ago
      Current gen is not ready for that. It's ready soon enough for sure. When I said "gen" I meant "not currently" but with this speed of development I'm not ready to bet if the scenario you described is 3 months of 5 years away.

samvher 4y ago

There is a lot of transferable knowledge to gain from learning this stuff properly, even if you don’t expect to do core AI work in a commercial setting. Optimization, function fitting, probability/statistics, GPU programming… My impression is that the field is more disciplined in terms of knowledge now than it was ~8 years ago - the fundamentals are better understood and more clearly expressed in literature. Also there are still plenty of topics on which the new techniques can probably be fruitfully applied, especially if you have some domain knowledge that the math/CS PhDs don’t have. For OP - I’m in a similar situation and have been going through Kevin Murphy’s “Probabilistic Machine Learning”, which is pretty massive and dense but also very lucid.

FiberBundle 4y ago
> My impression is that the field is more disciplined in terms of knowledge now than it was ~8 years ago - the fundamentals are better understood and more clearly expressed in literature. Is that really true? That's not my impression at all (though to be fair I haven't been keeping up with current research as much as I used to). My understanding is that there is still hardly any knowledge on what deep learning models (and large language models in particular) actually learn. The loss surfaces are opaque, one still doesn't know why local minima reached by gradient descent tend to generalize fairly well for these models. The latent representations that generative language models learn is, with the exception of the occasional paper that finds some superficial correlations, hardly investigated at all and overall extremely poorly understood. Very much interested in any references that contradict that sentiment.
- samvher 4y ago
  Maybe I'm biased specifically because of the book I mentioned. For me it's providing a theoretical basis for many things that earlier I learned in a hand-wavy way (e.g. way back I took Hinton's NN course and Ng's ML course, and learned about gradient descent, momentum, regularization, training/validation loss etc, and now with this book for the first time I feel like I get the bigger picture in the context of optimization/stats). The previous version of this book was from 2012 though and I'm not 100% sure how much of the material in the current edition is new (there is definitely a _lot_ more deep learning stuff in it). So yeah it could be that my impression is wrong, or that I made the scope at which it applies sound bigger than it is.
  - FiberBundle 4y ago
    I've only read (parts of) the Murphy book from 2012, I assume you're reading this one: https://probml.github.io/pml-book/toc1.pdf https://probml.github.io/pml-book/toc1.pdf? Almost all of the content that the new book covers, with the exception of the third part on deep learning, is about theory that was almost exclusively invented before 2012. Classical ML (non deep learning) is actually very rigorous compared to modern ML. There exist good theorems (statistical learning theory) for most of the classical models I'm aware of.
    - samvher 4y ago
      Yes that’s the one. Good to know, thanks for making me aware!

tomp 4y ago

Sorry if this is a stupid question, but from a non-practitioner's perspective, how or why is this sensible? Most of the cutting edge papers are trained on several $100k worth of GPU time, so does it even make sense to implement the algorithm without the available data & compute? How can you be sure that your implementation is correct, if you can't train it (hence you can't run proper inference with a good model)? Compare that to e.g. reimplementing a pure CS paper, almost anything can be reimplemented in a simple way - even something like "distributed database over 1000 nodes", well you don't technically need 1000 servers, you can just, you know, simulate them quite cheaply. Of course there might be similar techniques for ML but I'm just not aware of them.

madisonmay 4y ago
Often it might be viable to implement prediction w/o necessarily implementing training (especially if there are published weights or a reference implementation). Not viable for papers where the key contribution is a change to the pre-training objective / training methodology / optimizer, but useful for papers where the key contribution is architectural.
jdeaton 4y ago
> how or why is this sensible? The whole objective here is personal learning and this advice would be wildly different for how to practice ML professionally. The approach is directly analogous to advising a beginner programmer to get better at programming by actually writing computer programs. > Most of the cutting edge papers are trained on several $100k worth of GPU time Its besides the point, but I said nothing about a requirement that the methods that you choose to implement and learn from having to be cutting edge. More to the point, unless we have a different definition for what "cutting edge" means, you're wrong that "most of the cutting edge papers" require high computational resources. If that were true it would be nearly impossible for the field to make progress at the pace it does. There is a plethora of research in purely algorithmic approaches which do not require massive compute resources, and in fact this is the most productive portion of research to learn from because there the focus is on theory and progress in how to conceptualize / frame ML problems. Works which amount to "we took method X and massively scaled it up" are (in my opinion) less intellectually interesting to someone seeking to grow their knowledge in ML (though the results may might be extremely impressive and impactful, and it may be intellectually very interesting for the working directly on that project). > How can you be sure that your implementation is correct, if you can't train it (hence you can't run proper inference with a good model)? This is like asking how you can be sure that you've correctly implemented a B tree if you haven't used it to serve a distributed database to 1 million users. The answer is small isolated tests. One of the best ways to really test your knowledge of an ML algorithm is to design and write unit tests to assert it behaves correctly on trivial cases. You'll find bugs in your implementation, but you'll also be forced to think carefully about what the core characteristics of the algorithm are that must be asserted in order to convince yourself that its correct. Its a common beginner mistake in ML to just run/train your model and have that be the only test of its correctness. Its like deploying a web service with zero tests and letting "do I get X number of users" be the only test of your code's correctness. It sounds insane but its basically equivalent to what most beginners do in ML (my former self included).
- tomp 4y ago
  Thanks for extensive answer. Do you have a few of these cutting-edge algorithmic advances papers in mind, could you list them? I guess I got too pessimistic because of things like "emergent features" [1] / "grokking" [2] that seems to happen only with a lot of compute, and also the fact that the original (vanilla) transformer architecture remains (one of) the best, despite many additional ideas and "advances" (but that is only evident at large scale) [3]. Because of the points above, it's really hard for me, as a non-expert, to assess which papers are true advancements, and which were only published in pursuit of vanity metrics (e.g. publication counts) but actually represent overfit/cherry-picked results rather than robust progress. [1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... [2] https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanis... [3] https://twitter.com/YiTayML/status/1551657355036676096 https://twitter.com/YiTayML/status/1551657355036676096
  - jdeaton 4y ago
    I posted in another comment on this thread a list of papers which met these criteria for me at the time and which I learned a lot by implementing. > it's really hard for me, as a non-expert, to assess which papers are true advancements Its hard for me too, though I wouldn't consider myself an expert, just someone with a moderate amount of experience. Learning to discriminate important from less-important papers is another skill which takes effort to develop.
meken 4y ago
> How can you be sure that your implementation is correct, if you can't train it (hence you can't run proper inference with a good model)? Seems impossible to know for sure. Best I can think is to train your model to overfit a small amount of data
mxkopy 4y ago
> Most of the cutting edge papers are trained on several $100k worth of GPU time You can scale some things down. VGG 16 is basically a stack of CNNs, there's no reason you need 16 of them with an input size of 224x224x3; you can just as easily watch a 4 layer CNN learn filters on inputs of size 64x64x1. Obviously if the paper's result is achieved from sheer compute this won't work, but plenty of results come purely from the architecture. You could also implement and run networks that are designed to be really cheap to compute. ResNet/InceptionNet, for example. I think this is a pretty important part of the space right now, considering how performant, general, and therefore inefficient Transformer architectures are.
- mudrockbestgirl 4y ago
  But these are "old" models from 5+ years ago. Implementing them is not going to help you get up to speed with more recent AI research. From the OPs post, it seems like he already knows these basics.