5 ms·

How to Read Deep Learning Paper as a Software Engineer

74 points by research_pie 2y ago

apwell23 2y ago
great thing about collapse of LLM hype is that I no longer have FOMO as a software engineer.
- ibash 2y ago
  Which means now’s the time to learn and build.
- danielmarkbruce 2y ago
  Media collapse != collapse. Most people working on these things are about as bullish as ever.
  - wpietri 2y ago
    Most people working on these things are still riding a wave of investor funding. The investor hype wave lags the media hype wave. We'll see how bullish they are once that wave crests and we hit AI Winter #3 (or however many waves you want to count [1]). [1] https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
    - danielmarkbruce 2y ago
      People building things and who understand LLMs actually believe in it. The people who believe because of investors and media aren't in that bucket. Re winter - there is a difference this time - a hugely useful product (ChatGPT + gpt4 apis) doing billions of revenue. And it's almost all inference revenue.
      - lispisok 2y ago
        >People building things and who understand LLMs actually believe in it. You can say the same thing about blockchain
        danielmarkbruce 2y ago
        touche.
      - wpietri 2y ago
        I would suggest that the Venn diagram for "people building things" and "people who understand LLMs" is worth contemplating here. ChatGPT is certainly interesting, and some people are paying for it, but I think a) it's not clear how much of that use is also driven by hype, b) of the non-hype use, it's not clear how much will be sustained over the long term, and c) of that long-term use, it's not clear how much is actually of net benefit to society. Regarding A, an interesting parallel is blockchain hype. There were a zillion blockchain projects, from startups to enterprise efforts. I can't quickly find a reliable number for how much money product vendors, service vendors, and consulting companies took in, but I wouldn't be surprised if that was in the low billions at peak, to say nothing of all the investor and in-house money spent on staff and whatnot. And as far as I know, no non-cryptocurrency success was ever demonstrated. So we can't just suggest that revenue means something will last. Regarding B, it's a very volatile landscape both in terms of technology and in terms of market. E.g., for a hot minute, everybody loved lively AI generated images as stock photo replacement, but that fad is already past. People are getting a better sense for machine-generated text, too. I know of one company that fired their contract development shop because they kept getting machine-generated communications from the people they were paying to do actual work. Or we could look at Alexa and her kin as an example of something where despite initial excitement, it turns out people just don't care much. Same for VR; since the 1990s the technology keeps getting better and it keeps being a small niche. And we haven't even gotten to how competition and technological improvement will change (or perhaps eliminate) the margins here. And regarding C, a number of the uses people are paying for are things that are not making the world better. Academic cheating, for example, may be to the advantage of a student who just wants a credential, but it burns a lot of money and makes the world worse. The same applies for people in companies who want to give the appearance of work. Then we have things like spam and influence operations. Over the long term, parasites tend to get squished, however much they can flourish temporarily when they learn a new trick. So that's another slice of AI revenue that can't be counted on. And I should add that belief among the technically literate has so far been negatively correlated with there not being a winter. In this talk, iRobot founder and noted robot scientist takes a look at that: https://www.youtube.com/watch?v=pgrzEHJTPPM#t=36m55s https://www.youtube.com/watch?v=pgrzEHJTPPM#t=36m55s In particular, I set the timestamp to his list of AI hype cycles he had seen, 25 of them. Those all had people who understood the technology and believed in it. Maybe it's different this time, but that's what people say in every hype cycle.
  - apwell23 2y ago
    not sure what media collapse is.
ibash 2y ago
This video is great. Especially the comment that it takes a week to a month to deeply understand a research paper.
- simonw 2y ago
  The idea that it takes a week to a month to deeply understand a research paper feels to me like a massive failure in our expectations of academic writing. The reason it takes so long is that academic culture deliberately encourages creating documents that are extremely difficult for people to learn from. I am confident that in the vast majority of cases the increased understanding that somebody gets from spending 1-4 weeks of effort on a single paper was not actually worth that effort - the same result could have been had in an afternoon of direct conversation with the author of the paper. Academia really needs to break away from the culture of obfuscating these discoveries! At the very least, any paper worth its salt should be accompanied by an informal blog post that helps explain the discovery and maybe video or audio of the researcher communicating it as effectively as possible. EDIT: I just spotted that comment, it's a YouTube comment not a note in the video itself https://www.youtube.com/watch?v=nL7lAo95D-o&lc=UgxHeJrOv1g_tdQxYYh4AaABAg.A7virQ3xFm3A7vnI6iATMn https://www.youtube.com/watch?v=nL7lAo95D-o&lc=UgxHeJrOv1g_t... > If I'm trying to understand it deeply enough to mechanistically understand how the methodology work, this can take a few days to a week. > If I'm trying to reproduce the paper result to incorporate it within my own research project, I might wrestle with the thing for a full month. That comment feels very reasonable to me - for actually implementing a new deep learning strategy from a paper a full month feels fine to me. (I committed the cardinal Hacker News sin here of jumping on an opportunity to share one of my pet peeves, rather than engaging with the information directly!)
  - sigmoid10 2y ago
    >academic culture deliberately encourages creating documents that are extremely difficult for people to learn from. That's because they are not meant to be used as a learning resource. They're meant to communicate results to other experts who will understand them much faster than you ever could. If you want to learn a new field, start with textbooks and then look for review papers or things like that. Understanding research papers may be your ultimate goal, but don't expect to get there just by consuming these types of papers.
dboreham 2y ago
Applies to reading a paper in any field fwiw.
matt1 2y ago
[flagged]
- deleted 2y ago
  [deleted]
deleted 2y ago
[deleted]

74 points by research_pie 2y ago

apwell23 2y ago

great thing about collapse of LLM hype is that I no longer have FOMO as a software engineer.

ibash 2y ago
Which means now’s the time to learn and build.
danielmarkbruce 2y ago
Media collapse != collapse. Most people working on these things are about as bullish as ever.
- wpietri 2y ago
  Most people working on these things are still riding a wave of investor funding. The investor hype wave lags the media hype wave. We'll see how bullish they are once that wave crests and we hit AI Winter #3 (or however many waves you want to count [1]). [1] https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
  - danielmarkbruce 2y ago
    People building things and who understand LLMs actually believe in it. The people who believe because of investors and media aren't in that bucket. Re winter - there is a difference this time - a hugely useful product (ChatGPT + gpt4 apis) doing billions of revenue. And it's almost all inference revenue.
    - lispisok 2y ago
      >People building things and who understand LLMs actually believe in it. You can say the same thing about blockchain
      - danielmarkbruce 2y ago
        touche.
    - wpietri 2y ago
      I would suggest that the Venn diagram for "people building things" and "people who understand LLMs" is worth contemplating here. ChatGPT is certainly interesting, and some people are paying for it, but I think a) it's not clear how much of that use is also driven by hype, b) of the non-hype use, it's not clear how much will be sustained over the long term, and c) of that long-term use, it's not clear how much is actually of net benefit to society. Regarding A, an interesting parallel is blockchain hype. There were a zillion blockchain projects, from startups to enterprise efforts. I can't quickly find a reliable number for how much money product vendors, service vendors, and consulting companies took in, but I wouldn't be surprised if that was in the low billions at peak, to say nothing of all the investor and in-house money spent on staff and whatnot. And as far as I know, no non-cryptocurrency success was ever demonstrated. So we can't just suggest that revenue means something will last. Regarding B, it's a very volatile landscape both in terms of technology and in terms of market. E.g., for a hot minute, everybody loved lively AI generated images as stock photo replacement, but that fad is already past. People are getting a better sense for machine-generated text, too. I know of one company that fired their contract development shop because they kept getting machine-generated communications from the people they were paying to do actual work. Or we could look at Alexa and her kin as an example of something where despite initial excitement, it turns out people just don't care much. Same for VR; since the 1990s the technology keeps getting better and it keeps being a small niche. And we haven't even gotten to how competition and technological improvement will change (or perhaps eliminate) the margins here. And regarding C, a number of the uses people are paying for are things that are not making the world better. Academic cheating, for example, may be to the advantage of a student who just wants a credential, but it burns a lot of money and makes the world worse. The same applies for people in companies who want to give the appearance of work. Then we have things like spam and influence operations. Over the long term, parasites tend to get squished, however much they can flourish temporarily when they learn a new trick. So that's another slice of AI revenue that can't be counted on. And I should add that belief among the technically literate has so far been negatively correlated with there not being a winter. In this talk, iRobot founder and noted robot scientist takes a look at that: https://www.youtube.com/watch?v=pgrzEHJTPPM#t=36m55s https://www.youtube.com/watch?v=pgrzEHJTPPM#t=36m55s In particular, I set the timestamp to his list of AI hype cycles he had seen, 25 of them. Those all had people who understood the technology and believed in it. Maybe it's different this time, but that's what people say in every hype cycle.
- apwell23 2y ago
  not sure what media collapse is.

ibash 2y ago

This video is great. Especially the comment that it takes a week to a month to deeply understand a research paper.

simonw 2y ago
The idea that it takes a week to a month to deeply understand a research paper feels to me like a massive failure in our expectations of academic writing. The reason it takes so long is that academic culture deliberately encourages creating documents that are extremely difficult for people to learn from. I am confident that in the vast majority of cases the increased understanding that somebody gets from spending 1-4 weeks of effort on a single paper was not actually worth that effort - the same result could have been had in an afternoon of direct conversation with the author of the paper. Academia really needs to break away from the culture of obfuscating these discoveries! At the very least, any paper worth its salt should be accompanied by an informal blog post that helps explain the discovery and maybe video or audio of the researcher communicating it as effectively as possible. EDIT: I just spotted that comment, it's a YouTube comment not a note in the video itself https://www.youtube.com/watch?v=nL7lAo95D-o&lc=UgxHeJrOv1g_tdQxYYh4AaABAg.A7virQ3xFm3A7vnI6iATMn https://www.youtube.com/watch?v=nL7lAo95D-o&lc=UgxHeJrOv1g_t... > If I'm trying to understand it deeply enough to mechanistically understand how the methodology work, this can take a few days to a week. > If I'm trying to reproduce the paper result to incorporate it within my own research project, I might wrestle with the thing for a full month. That comment feels very reasonable to me - for actually implementing a new deep learning strategy from a paper a full month feels fine to me. (I committed the cardinal Hacker News sin here of jumping on an opportunity to share one of my pet peeves, rather than engaging with the information directly!)
- sigmoid10 2y ago
  >academic culture deliberately encourages creating documents that are extremely difficult for people to learn from. That's because they are not meant to be used as a learning resource. They're meant to communicate results to other experts who will understand them much faster than you ever could. If you want to learn a new field, start with textbooks and then look for review papers or things like that. Understanding research papers may be your ultimate goal, but don't expect to get there just by consuming these types of papers.

dboreham 2y ago

Applies to reading a paper in any field fwiw.

matt1 2y ago

[flagged]

deleted 2y ago
[deleted]

deleted 2y ago

[deleted]