6 ms·

The open source learning curve for AI researchers

155 points by bobvanluijt 3y ago

ShamelessC 3y ago
tl;dr - AI researchers have never had to undergo code review, deal with continuous integration, or manage expectations of inexperienced users of their software. Which is just far too true, and remains true outside of VC-backed scenarios like huggingface, but only because their playbook explicitly optimizes for this. On the other hand, it’s a great time to be a seasoned software engineer who can help with these sorts of issues. If you’re trying to level up your machine learning experience - consider a strategy like this: Pick a random lucidrains repository. Try to get it running on your setup. It probably won’t. (No offense Phil! You’re doing God’s work and can’t cover every possible configuration). Fix the issue. Write a test for it. Setup basic CI for the test. Try to get the PR merged!
- charcircuit 3y ago
  That doesn't sound like gaining machine learning experience. It sounds like you are just doing free infrastructure related work for someone else.
  - ShamelessC 3y ago
    That’s just a first step. The next step is to fix issues with, for instance, training instability. It is definitely free work though. There are absolutely ways to get paid to do this sort of thing. But if you believe in learning by volunteering it’s a good option. Having contributions to popular ML repos looks good to tech recruiters as well, so there’s that.
    - sdesol 3y ago
      Here's some insights for what I believe to be the most popular ML repo out there right now, which is ggerganov/llama.cpp https://devboard.gitsense.com/ggerganov/llama.cpp https://devboard.gitsense.com/ggerganov/llama.cpp The popularity of this project is kind of insane (which probably explains why it was able to raise fairly easily). If you take a look at https://reviews.gitsense.com/insights/github?q=merged%3Atrue%2Bpull-age%3A%3C%3D120&t=crc-insights&tb=closed&v=ggerganov%2Fllama.cpp%3A%3Agithub%3Aggerganov%2Fllama.cpp%3A%3A%3A%3A https://reviews.gitsense.com/insights/github?q=merged%3Atrue... you can see all the merged pull requests (for PRs created within the last 120 days). And if you look at the "gift icon" you can see how long the contributors have been contributing to the project for. For many, it's less than 4 months, which means many new contributors are successfully getting their changes merged. Edit: Rejected PRs https://reviews.gitsense.com/insights/github?q=pull-age%3A%3C%3D120%2Brejected%3Atrue&t=crc-insights&tb=closed&v=ggerganov%2Fllama.cpp%3A%3Agithub%3Aggerganov%2Fllama.cpp%3A%3A%3A%3A https://reviews.gitsense.com/insights/github?q=pull-age%3A%3... Full Disclosure: This is my tool.
      - itchy_spider44 3y ago
        What the eff is this? This is awesome, I've always hated GitHub limited ways to compare repos/forks by activity. Thank you for making this
        sdesol 3y ago
        Hey thanks! And hopefully your green username doesn't make people think we are one of the same as your comment, while greatly appreciated, is not some smart (or stupid) marketing strategy on my part. I honestly believe we can work smarter and I talk about the paradigm shift here https://devboard.gitsense.com/ggerganov/llama.cpp?board=gitsense_examples.intro_to_widgets https://devboard.gitsense.com/ggerganov/llama.cpp?board=gits... Note, the repo that is talked about in the intro DevBoard hasn't been pushed yet. DevBoard will be open sourced and I'm currently working on a browser extension (will also be open sourced) that will convert GitHub's insights tab into a hackable dashboard, which should make it easy for people to address GitHub limitations.
        omeze 3y ago
        gitsense is def a sick tool, ty for sharing
        lhl 3y ago
        old user here, just want to confirm that this tool is super neat, can see it being very useful for comparing project health when picking a lib to use. hope it can be sustainable/stick around in the future.
- anotherhue 3y ago
  Reproducible builds ala Nix and others can really shine here, they remove entire classes of 'it doesn't work for me' issues. Alas, it's another learning curve to add.
- ibizaman 3y ago
  Step 0 should be open an issue to know if they’re at least interested, before starting implementing anything.
- asimeqi 3y ago
  As somebody who was paid for many years to turn AI researchers' code into usable products, I have one piece of advice. If you want to become an AI researcher, don't fix their code. I did this in the hope that I eventually would get to work directly in an AI project. When I finally got to participate in such a project part-time, it turned out that I could finish in a matter of days a task that would take the AI researchers weeks. That didn't please them at all so soon enough I was switched full-time to my non AI project since that one "needed me more". If you want to become an AI researcher, do AI researcher projects, simple as that.
  - brizii 3y ago
    I'm not sure I follow. They weren't pleased with the speed at which you completed tasks? I feel this might be specific to your company. I find this to be a very desirable trait, considering most professors at my university won't take undergraduate researchers on the basis that they simply don't complete tasks as quickly as a PhD would. Different contexts but I feel this example still holds some merit. Also, we're entering a paradigm in which a lot of research is constrained by compute, and it can be very difficult to just "do AI research projects" when such projects involved training policies in virtual environments or designing the next transformer, for instance.
    - asimeqi 3y ago
      Professors, of course, don't feel threatened by students so want them to be as good as possible. But in a company where I had the same pay and title as the AI researchers, just different fields of work, me applications, they research, they realized real quick that if I kept doing 2-3 times more tasks than they were doing I would get experience with everything they knew. Everybody wants to have their own little field where they are the boss. A newcomer that keeps putting their nose everywhere might not be very desirable. I knew what I was doing and I knew the risk I was running. I was hoping that the project manager would support me. The manager did not support me, I was out.
      - coyotespike 3y ago
- Der_Einzige 3y ago
  Honestly, keeping the best ML repos hard to run unironically gives strong job security. I’m happy when something requires a bit of work to get running, means I get to stay employed longer and the normies who will ruin it (I.e. the idiots writing shit like “wormGPT”) will be delayed. We are also in a rapidly ending golden age of the AI research community treating copyright like it doesn’t exist (which is good and Aaron Swartz was and is a saint) Every PR that makes it easier to run their code makes us that much closer to going back to the hell of strong IP regulations and laws. Please don’t make their stuff easier to run.
- techwizrd 3y ago
  As a researcher, I've even had difficulty making sure the code matches the algorithm as described in the paper. Getting old code running and validating that it's correct so you can compare is really non-trivial.
  - bravura 3y ago
    Go to arXiV and download the latex source. Plug it into ChatGPT and ask it to implement in pytorch. I've really enjoyed this approach.
    - Calamity 3y ago
      how well does this actually pan out for you?
      - bravura 3y ago
        I've used it with great success.
        techwizrd 3y ago
        Can you provide an example of where this has been successful? I've spoken with many researchers and grad students only to find that there was a critical typo in the algorithm or undescribed setting (e.g., only converges when a learning rate scheduler is applied) in the code. I'll see the same algorithm implemented differently in different repositories. This is even the case for papers I've found with thousands of citations. It can be tremendously difficult to reproduce the results of papers, especially when they may require large amounts of compute that small researchers don't have. And I don't know whether it's a bug in my code or the paper or the algorithm.
        bravura 3y ago
        I mean, what you describe is an unfortunate and unavoidable issue in academia (and in the world in general). GPT4 doesn't work magic here, of course. You still have to: 1) Understand the work and the motivation. (GPT4 can help by playing the role of junior PhD if you can play the role of astute advisor.) 2) Sniff out things that are underspecified or seem wrong. (GPT4 also can help here, see above.) 3) Email the authors with questions, compare against shitty published codebases, etc. depending upon how gnarly/rushed the prose is. With that said, it's also "research smell" (compare "code smell") if a paper is so hastily written and undercited that you're the first person replicating it. And maybe instead of going for "my new bleeding edge approach that got 0.1% score better than boring old model with 50 cites", you probably should just implement boring old model. So, where this has been successful for me is in implementing denoising diffusion for different problem domains. Given that there is broad literature on denoising diffusion, when some things are underspecified you can start looking at best practices for other researchers. Alternately, other things like specific transformers etc. Basically what I'm saying is that if you are trying to reimplement something that is so niche, it's like catching butterflies. A better research agenda involves working within a particular field of study where there is supporting evidence and approaches to compare and constrast to. This goes without saying, regardless of whether an AI is involved or not.
- petemir 3y ago
  lol, this hits close to home. As a PhD student with several years of past experience in Dev work, I have become the main reference point of the lab to deal with this stuff. Just now I have spent 2 days debugging a Dockerfile shared in a repository published in a paper to build the environment to run the experiments. One interesting thing about this is that since the moment the experiments have been done, and the paper is actually published, you are probably several (even major!) versions behind on every possible software. If the shared information does not take this into account (i.e., versions are not fixed when trying to create reproducible environments), you're in for some painful debugging :).
twobitshifter 3y ago
The midjourney rockband example here is another case of the two many fingers problem. I had heard that was solved? When you start looking closely, the faces are all mutated in odd ways, but somehow it’s convincing when quickly scrolling by.
- omnicognate 3y ago
  Zooming in, it's terrifying.
revskill 3y ago
[flagged]
jbkkd 3y ago
Where is a good gathering place for the AI community? Any publicly open Discord/IRC?
- throwawayadvsec 3y ago
  https://www.eleuther.ai/ https://www.eleuther.ai/ ?

155 points by bobvanluijt 3y ago

tl;dr - AI researchers have never had to undergo code review, deal with continuous integration, or manage expectations of inexperienced users of their software. Which is just far too true, and remains true outside of VC-backed scenarios like huggingface, but only because their playbook explicitly optimizes for this. On the other hand, it’s a great time to be a seasoned software engineer who can help with these sorts of issues. If you’re trying to level up your machine learning experience - consider a strategy like this: Pick a random lucidrains repository. Try to get it running on your setup. It probably won’t. (No offense Phil! You’re doing God’s work and can’t cover every possible configuration). Fix the issue. Write a test for it. Setup basic CI for the test. Try to get the PR merged!

charcircuit 3y ago
That doesn't sound like gaining machine learning experience. It sounds like you are just doing free infrastructure related work for someone else.
- ShamelessC 3y ago
  That’s just a first step. The next step is to fix issues with, for instance, training instability. It is definitely free work though. There are absolutely ways to get paid to do this sort of thing. But if you believe in learning by volunteering it’s a good option. Having contributions to popular ML repos looks good to tech recruiters as well, so there’s that.
  - sdesol 3y ago
    Here's some insights for what I believe to be the most popular ML repo out there right now, which is ggerganov/llama.cpp https://devboard.gitsense.com/ggerganov/llama.cpp https://devboard.gitsense.com/ggerganov/llama.cpp The popularity of this project is kind of insane (which probably explains why it was able to raise fairly easily). If you take a look at https://reviews.gitsense.com/insights/github?q=merged%3Atrue%2Bpull-age%3A%3C%3D120&t=crc-insights&tb=closed&v=ggerganov%2Fllama.cpp%3A%3Agithub%3Aggerganov%2Fllama.cpp%3A%3A%3A%3A https://reviews.gitsense.com/insights/github?q=merged%3Atrue... you can see all the merged pull requests (for PRs created within the last 120 days). And if you look at the "gift icon" you can see how long the contributors have been contributing to the project for. For many, it's less than 4 months, which means many new contributors are successfully getting their changes merged. Edit: Rejected PRs https://reviews.gitsense.com/insights/github?q=pull-age%3A%3C%3D120%2Brejected%3Atrue&t=crc-insights&tb=closed&v=ggerganov%2Fllama.cpp%3A%3Agithub%3Aggerganov%2Fllama.cpp%3A%3A%3A%3A https://reviews.gitsense.com/insights/github?q=pull-age%3A%3... Full Disclosure: This is my tool.
    - itchy_spider44 3y ago
      What the eff is this? This is awesome, I've always hated GitHub limited ways to compare repos/forks by activity. Thank you for making this
      - sdesol 3y ago
        Hey thanks! And hopefully your green username doesn't make people think we are one of the same as your comment, while greatly appreciated, is not some smart (or stupid) marketing strategy on my part. I honestly believe we can work smarter and I talk about the paradigm shift here https://devboard.gitsense.com/ggerganov/llama.cpp?board=gitsense_examples.intro_to_widgets https://devboard.gitsense.com/ggerganov/llama.cpp?board=gits... Note, the repo that is talked about in the intro DevBoard hasn't been pushed yet. DevBoard will be open sourced and I'm currently working on a browser extension (will also be open sourced) that will convert GitHub's insights tab into a hackable dashboard, which should make it easy for people to address GitHub limitations.
        omeze 3y ago
        gitsense is def a sick tool, ty for sharing
        lhl 3y ago
        old user here, just want to confirm that this tool is super neat, can see it being very useful for comparing project health when picking a lib to use. hope it can be sustainable/stick around in the future.
anotherhue 3y ago
Reproducible builds ala Nix and others can really shine here, they remove entire classes of 'it doesn't work for me' issues. Alas, it's another learning curve to add.
ibizaman 3y ago
Step 0 should be open an issue to know if they’re at least interested, before starting implementing anything.
asimeqi 3y ago
As somebody who was paid for many years to turn AI researchers' code into usable products, I have one piece of advice. If you want to become an AI researcher, don't fix their code. I did this in the hope that I eventually would get to work directly in an AI project. When I finally got to participate in such a project part-time, it turned out that I could finish in a matter of days a task that would take the AI researchers weeks. That didn't please them at all so soon enough I was switched full-time to my non AI project since that one "needed me more". If you want to become an AI researcher, do AI researcher projects, simple as that.
- brizii 3y ago
  I'm not sure I follow. They weren't pleased with the speed at which you completed tasks? I feel this might be specific to your company. I find this to be a very desirable trait, considering most professors at my university won't take undergraduate researchers on the basis that they simply don't complete tasks as quickly as a PhD would. Different contexts but I feel this example still holds some merit. Also, we're entering a paradigm in which a lot of research is constrained by compute, and it can be very difficult to just "do AI research projects" when such projects involved training policies in virtual environments or designing the next transformer, for instance.
  - asimeqi 3y ago
    Professors, of course, don't feel threatened by students so want them to be as good as possible. But in a company where I had the same pay and title as the AI researchers, just different fields of work, me applications, they research, they realized real quick that if I kept doing 2-3 times more tasks than they were doing I would get experience with everything they knew. Everybody wants to have their own little field where they are the boss. A newcomer that keeps putting their nose everywhere might not be very desirable. I knew what I was doing and I knew the risk I was running. I was hoping that the project manager would support me. The manager did not support me, I was out.
    - coyotespike 3y ago
Der_Einzige 3y ago
Honestly, keeping the best ML repos hard to run unironically gives strong job security. I’m happy when something requires a bit of work to get running, means I get to stay employed longer and the normies who will ruin it (I.e. the idiots writing shit like “wormGPT”) will be delayed. We are also in a rapidly ending golden age of the AI research community treating copyright like it doesn’t exist (which is good and Aaron Swartz was and is a saint) Every PR that makes it easier to run their code makes us that much closer to going back to the hell of strong IP regulations and laws. Please don’t make their stuff easier to run.
techwizrd 3y ago
As a researcher, I've even had difficulty making sure the code matches the algorithm as described in the paper. Getting old code running and validating that it's correct so you can compare is really non-trivial.
- bravura 3y ago
  Go to arXiV and download the latex source. Plug it into ChatGPT and ask it to implement in pytorch. I've really enjoyed this approach.
  - Calamity 3y ago
    how well does this actually pan out for you?
    - bravura 3y ago
      I've used it with great success.
      - techwizrd 3y ago
        Can you provide an example of where this has been successful? I've spoken with many researchers and grad students only to find that there was a critical typo in the algorithm or undescribed setting (e.g., only converges when a learning rate scheduler is applied) in the code. I'll see the same algorithm implemented differently in different repositories. This is even the case for papers I've found with thousands of citations. It can be tremendously difficult to reproduce the results of papers, especially when they may require large amounts of compute that small researchers don't have. And I don't know whether it's a bug in my code or the paper or the algorithm.
        bravura 3y ago
        I mean, what you describe is an unfortunate and unavoidable issue in academia (and in the world in general). GPT4 doesn't work magic here, of course. You still have to: 1) Understand the work and the motivation. (GPT4 can help by playing the role of junior PhD if you can play the role of astute advisor.) 2) Sniff out things that are underspecified or seem wrong. (GPT4 also can help here, see above.) 3) Email the authors with questions, compare against shitty published codebases, etc. depending upon how gnarly/rushed the prose is. With that said, it's also "research smell" (compare "code smell") if a paper is so hastily written and undercited that you're the first person replicating it. And maybe instead of going for "my new bleeding edge approach that got 0.1% score better than boring old model with 50 cites", you probably should just implement boring old model. So, where this has been successful for me is in implementing denoising diffusion for different problem domains. Given that there is broad literature on denoising diffusion, when some things are underspecified you can start looking at best practices for other researchers. Alternately, other things like specific transformers etc. Basically what I'm saying is that if you are trying to reimplement something that is so niche, it's like catching butterflies. A better research agenda involves working within a particular field of study where there is supporting evidence and approaches to compare and constrast to. This goes without saying, regardless of whether an AI is involved or not.
petemir 3y ago
lol, this hits close to home. As a PhD student with several years of past experience in Dev work, I have become the main reference point of the lab to deal with this stuff. Just now I have spent 2 days debugging a Dockerfile shared in a repository published in a paper to build the environment to run the experiments. One interesting thing about this is that since the moment the experiments have been done, and the paper is actually published, you are probably several (even major!) versions behind on every possible software. If the shared information does not take this into account (i.e., versions are not fixed when trying to create reproducible environments), you're in for some painful debugging :).

twobitshifter 3y ago

The midjourney rockband example here is another case of the two many fingers problem. I had heard that was solved? When you start looking closely, the faces are all mutated in odd ways, but somehow it’s convincing when quickly scrolling by.

omnicognate 3y ago
Zooming in, it's terrifying.

revskill 3y ago

[flagged]

jbkkd 3y ago

Where is a good gathering place for the AI community? Any publicly open Discord/IRC?

throwawayadvsec 3y ago
https://www.eleuther.ai/ https://www.eleuther.ai/ ?