7 ms·

Arxiv Sanity Preserver

129 points by stared 11y ago

zenlikethat 11y ago
I have been checking this site out lately and using it to download PDFs to read on my phone later. I really like it! The options to see most popular papers and to search by field is really nice. Thank you Andrej for putting this together and maintaining it.
gaur 11y ago
> because things were seriously getting out of hand. Whatever that means.
- clishem 11y ago
  I thought it meant a filter for nonsense articles, but apparently arXiv is not very searchable anymore.
- karpathy 11y ago
  Imagine waking up every morning with 50 new arxiv papers uploaded that night. You panic and quickly scan through the papers - any of them could be very related to your research, or scoop your latest idea, or have good ideas you can use in your own work. Arxiv makes no attempts to filter these for you, so it's up to you to carefully scan through this unlabeled list of paper titles. You eventually find 3 papers that you have to read and put them on your list. You manage to read 1 that day. Next day you wake up and 50 new papers are up. You iterate for a few weeks and suddenly you have a toread list of 20 papers and 100 new arxiv papers just came in that evening. That's what's currently happening in research at least in deep learning (but I imagine more widely too), especially around big conference deadlines, and that's what I label "things seriously getting out of hand". That's a first use case. The second way things are out of hand is that you remember this paper from 3 years ago that was very related to this one, but can't remember it's name anymore. Here you can sort by similarity to any paper, and usually these papers come up on top of the sorted list. This is also useful for finding related work. Another use case is a peace of mind that you somehow did not miss some papers that you definitely should know about. Google Scholar is supposed to have similar features: it emails you papers it thinks you would be interested in and can in principle show similar papers. I don't know what they do internally but these features are quite terrible and low quality in my own experience compared to what I get here. More generally the amount of innovation in Google Scholar over the last few years is sadly either zero or negative (but overall I still get nightmares about what would happen to academia if Google pulled a Google Reader with Scholar). For arxiv-sanity it's tfidf vectors of bigrams from full text of each paper and I do L2 lookups for similarity ranking and train personalized SVMs for people for recommendations. The results are, at least for me, significantly better.
  - jakub_h 11y ago
    What about using deep learning to properly classify arxiv papers about deep learning (and other things, perhaps)? ;)
    - karpathy 11y ago
      the right tool for the job :) In this case I'm perfectly happy with SVMs over tfidf bigrams and where that places you in the tradeoffs space.
    - ajford 11y ago
      Do you want Skynet? This sounds like the start of Skynet...
  - dalke 11y ago
    I have no real point here, only historical commentary. I've been reading papers from the 1960s, which is when the term "information explosion" was coined. People then were struggling to stay current with the literature, and thought 'things were seriously getting out of hand.' This was the start of abstracting services, like ISI, where you could even arrange the results of a keyword search of all the new papers to be sent to you each week - a clear predecessor to personalized RSS feeds. Going back even further to the immediate post-war era, the library systems of the time, which were structured around books and journals and organized by topic, couldn't keep up with the deluge of research reports which cut across multiple topics. The field of information retrieval, using first punched cards and then computers, started because the publication flow was 'seriously getting out of hand'. Or for a specific example, after high T_c superconductors were discovered in 1986, there was a mad rush of interest as solid state physicists from around the world explored the new territory. A Google Scholar search for "high temperature superconductor" finds: 1986 - 846 publications 1987 - 2 600 1988 - 3 900 1989 - 4 780 1990 - 4 870 1991 - 5 250 That's 14 papers per day, any one of which might be "very related to your research, or scoop your latest idea, or have good ideas you can use in your own work." Granted, 14 << 50, but that doesn't include some of the papers about "high Tc" which don't use the whole phrase. Also, those are 14 peer-reviewed papers per day, so there has been some filter, and experimental research in high Tc research requires more equipment than deep learning. Think of my comment as a reminder that things have been out of hand for most of a century, and dealing with that deluge emotionally connects you to the headache that generations of researchers before you have had to suffer with. :)
georgeoliver 11y ago
Not having anything to do with publishing papers myself I thought the 100,000+ papers submitted in 2015 sounded like a lot, until I looked into it [1] and saw there's likely more than a million academic papers published every year. What are all those papers like? I wonder if a large proportion are the equivalent of peer-reviewed blog posts. [1] https://www.quora.com/How-many-academic-papers-are-published-each-year https://www.quora.com/How-many-academic-papers-are-published...
- xamuel 11y ago
  I wonder how many research professors there are in the world. Basic googling suggests there are order-of-magnitude 1000000 university faculty in the world, so a million papers is at most 1 paper per faculty member per year, which doesn't seem unreasonable. Of course, how many of those faculty are serious researchers, and how many are just instructors, is a harder question.
  - dalke 11y ago
    It's not only research professors who publish. Corporate research scientists also publish, including those at Microsoft, Google, and Intel.
    - jrowley 11y ago
      Grad students publish too!
      - dalke 11y ago
        True! And undergrads. And private scholars. And .. The easy way to answer this would be to use something like Web of Knowledge to get a rough sense of the number of distinct authors.
  - ajford 11y ago
    Part of it comes from the fact that a lot of institutions require continuous publication by their faculty/research staff. If you don't publish because you're focusing on a long term paper, it can look bad on you during annual review. To combat this a fair number of people will publish many times on the same developing experiment (not that this is necessarily a bad thing). Don't forget grad students and dedicated undergrads. That adds a fair bit to your existing numbers.
smhx 11y ago
Being in the deep learning community, the number of papers appearing has been getting out of control. Most of the papers appear on arxiv and are of low quality. This is particularly problematic right before conference deadlines. Karpathy's Arxiv-sanity helps a lot to keep in touch with the latest and greatest deep learning without having to spend all my time reading papers.
- deleted 11y ago
  [deleted]
stared 11y ago
To get some sense of the arXiv submission growth: http://arxiv.org/help/stats/2015_by_area/index http://arxiv.org/help/stats/2015_by_area/index
strahil 11y ago
Ironically it says "The connection was reset".
jeffjose 11y ago
Looks like the site is down.
- karpathy 11y ago
  Yeah sorry about that - I see cryptic errors in the server logs popping up at random. I think it must be something to do with the scale of requests coming in and breaking the site in some way I don't currently understand. I have near zero experience with scaling web sites, if anyone who does is passionate about meta research you're very welcome to look through serve.py and help me out. One of the problems I caught: error: [Errno 24] Too many open files from tornado. Trying to fix (edit ok made ulimit -n larger and I don't see this error anymore at least)
  - drostie 11y ago
    I'd offer but I have other commitments this afternoon; I hope you find someone who can help!
  - raverbashing 11y ago
    It is a funny error Does this happen always with the same traceback (that is, with the same area of your code in the stack trace)?
  - kuprel 11y ago
    Would be cool if you could upvote/downvote papers
    - karpathy 11y ago
      I thought about this quite a bit. I don't think I want downvoting. And I don't want effectless upvoting. The way it is right now is that you can add a paper to your library, and it will then feed into your personal SVM as a positive example of papers you like to see more of. Adding paper to your library as a type of paper you like to see more of effectively counts as an "up vote", and is what is sorted by when you go to the "top" tab.
stared 11y ago
For talking about arXiv papers and recommending them to others, there is also: https://scirate.com/ https://scirate.com/
- da-bacon 11y ago
  I wrote the first version of Scirate exactly because the number of papers I had to eyeball each day was so high. Now that I've left academia I find it even more useful (so huge thanks to those who rewrote it from scratch)! If you are in quantum computing it definitely helps your sanity.
conceit 11y ago
Who curates the top recent papers seen on first visit and is personal preference only accounted for in recommended?
lucidrains 11y ago
Hey, if you need any help expanding the site, I'd be game to help out!
jchung 11y ago
As someone who uses arxiv only very rarely, can someone please explain how this preserves sanity?
0x54MUR41 11y ago
That's an awesome website. I hope it will support TLS for registration or login mechanisms.

129 points by stared 11y ago

I have been checking this site out lately and using it to download PDFs to read on my phone later. I really like it! The options to see most popular papers and to search by field is really nice. Thank you Andrej for putting this together and maintaining it.

gaur 11y ago

> because things were seriously getting out of hand. Whatever that means.

clishem 11y ago
I thought it meant a filter for nonsense articles, but apparently arXiv is not very searchable anymore.
karpathy 11y ago
Imagine waking up every morning with 50 new arxiv papers uploaded that night. You panic and quickly scan through the papers - any of them could be very related to your research, or scoop your latest idea, or have good ideas you can use in your own work. Arxiv makes no attempts to filter these for you, so it's up to you to carefully scan through this unlabeled list of paper titles. You eventually find 3 papers that you have to read and put them on your list. You manage to read 1 that day. Next day you wake up and 50 new papers are up. You iterate for a few weeks and suddenly you have a toread list of 20 papers and 100 new arxiv papers just came in that evening. That's what's currently happening in research at least in deep learning (but I imagine more widely too), especially around big conference deadlines, and that's what I label "things seriously getting out of hand". That's a first use case. The second way things are out of hand is that you remember this paper from 3 years ago that was very related to this one, but can't remember it's name anymore. Here you can sort by similarity to any paper, and usually these papers come up on top of the sorted list. This is also useful for finding related work. Another use case is a peace of mind that you somehow did not miss some papers that you definitely should know about. Google Scholar is supposed to have similar features: it emails you papers it thinks you would be interested in and can in principle show similar papers. I don't know what they do internally but these features are quite terrible and low quality in my own experience compared to what I get here. More generally the amount of innovation in Google Scholar over the last few years is sadly either zero or negative (but overall I still get nightmares about what would happen to academia if Google pulled a Google Reader with Scholar). For arxiv-sanity it's tfidf vectors of bigrams from full text of each paper and I do L2 lookups for similarity ranking and train personalized SVMs for people for recommendations. The results are, at least for me, significantly better.
- jakub_h 11y ago
  What about using deep learning to properly classify arxiv papers about deep learning (and other things, perhaps)? ;)
  - karpathy 11y ago
    the right tool for the job :) In this case I'm perfectly happy with SVMs over tfidf bigrams and where that places you in the tradeoffs space.
  - ajford 11y ago
    Do you want Skynet? This sounds like the start of Skynet...
- dalke 11y ago
  I have no real point here, only historical commentary. I've been reading papers from the 1960s, which is when the term "information explosion" was coined. People then were struggling to stay current with the literature, and thought 'things were seriously getting out of hand.' This was the start of abstracting services, like ISI, where you could even arrange the results of a keyword search of all the new papers to be sent to you each week - a clear predecessor to personalized RSS feeds. Going back even further to the immediate post-war era, the library systems of the time, which were structured around books and journals and organized by topic, couldn't keep up with the deluge of research reports which cut across multiple topics. The field of information retrieval, using first punched cards and then computers, started because the publication flow was 'seriously getting out of hand'. Or for a specific example, after high T_c superconductors were discovered in 1986, there was a mad rush of interest as solid state physicists from around the world explored the new territory. A Google Scholar search for "high temperature superconductor" finds: 1986 - 846 publications 1987 - 2 600 1988 - 3 900 1989 - 4 780 1990 - 4 870 1991 - 5 250 That's 14 papers per day, any one of which might be "very related to your research, or scoop your latest idea, or have good ideas you can use in your own work." Granted, 14 << 50, but that doesn't include some of the papers about "high Tc" which don't use the whole phrase. Also, those are 14 peer-reviewed papers per day, so there has been some filter, and experimental research in high Tc research requires more equipment than deep learning. Think of my comment as a reminder that things have been out of hand for most of a century, and dealing with that deluge emotionally connects you to the headache that generations of researchers before you have had to suffer with. :)

georgeoliver 11y ago

Not having anything to do with publishing papers myself I thought the 100,000+ papers submitted in 2015 sounded like a lot, until I looked into it [1] and saw there's likely more than a million academic papers published every year. What are all those papers like? I wonder if a large proportion are the equivalent of peer-reviewed blog posts. [1] https://www.quora.com/How-many-academic-papers-are-published-each-year https://www.quora.com/How-many-academic-papers-are-published...

xamuel 11y ago
I wonder how many research professors there are in the world. Basic googling suggests there are order-of-magnitude 1000000 university faculty in the world, so a million papers is at most 1 paper per faculty member per year, which doesn't seem unreasonable. Of course, how many of those faculty are serious researchers, and how many are just instructors, is a harder question.
- dalke 11y ago
  It's not only research professors who publish. Corporate research scientists also publish, including those at Microsoft, Google, and Intel.
  - jrowley 11y ago
    Grad students publish too!
    - dalke 11y ago
      True! And undergrads. And private scholars. And .. The easy way to answer this would be to use something like Web of Knowledge to get a rough sense of the number of distinct authors.
- ajford 11y ago
  Part of it comes from the fact that a lot of institutions require continuous publication by their faculty/research staff. If you don't publish because you're focusing on a long term paper, it can look bad on you during annual review. To combat this a fair number of people will publish many times on the same developing experiment (not that this is necessarily a bad thing). Don't forget grad students and dedicated undergrads. That adds a fair bit to your existing numbers.

smhx 11y ago

Being in the deep learning community, the number of papers appearing has been getting out of control. Most of the papers appear on arxiv and are of low quality. This is particularly problematic right before conference deadlines. Karpathy's Arxiv-sanity helps a lot to keep in touch with the latest and greatest deep learning without having to spend all my time reading papers.

deleted 11y ago
[deleted]

stared 11y ago

To get some sense of the arXiv submission growth: http://arxiv.org/help/stats/2015_by_area/index http://arxiv.org/help/stats/2015_by_area/index

strahil 11y ago

Ironically it says "The connection was reset".

jeffjose 11y ago

Looks like the site is down.

karpathy 11y ago
Yeah sorry about that - I see cryptic errors in the server logs popping up at random. I think it must be something to do with the scale of requests coming in and breaking the site in some way I don't currently understand. I have near zero experience with scaling web sites, if anyone who does is passionate about meta research you're very welcome to look through serve.py and help me out. One of the problems I caught: error: [Errno 24] Too many open files from tornado. Trying to fix (edit ok made ulimit -n larger and I don't see this error anymore at least)
- drostie 11y ago
  I'd offer but I have other commitments this afternoon; I hope you find someone who can help!
- raverbashing 11y ago
  It is a funny error Does this happen always with the same traceback (that is, with the same area of your code in the stack trace)?
- kuprel 11y ago
  Would be cool if you could upvote/downvote papers
  - karpathy 11y ago
    I thought about this quite a bit. I don't think I want downvoting. And I don't want effectless upvoting. The way it is right now is that you can add a paper to your library, and it will then feed into your personal SVM as a positive example of papers you like to see more of. Adding paper to your library as a type of paper you like to see more of effectively counts as an "up vote", and is what is sorted by when you go to the "top" tab.

stared 11y ago

For talking about arXiv papers and recommending them to others, there is also: https://scirate.com/ https://scirate.com/

da-bacon 11y ago
I wrote the first version of Scirate exactly because the number of papers I had to eyeball each day was so high. Now that I've left academia I find it even more useful (so huge thanks to those who rewrote it from scratch)! If you are in quantum computing it definitely helps your sanity.

conceit 11y ago

Who curates the top recent papers seen on first visit and is personal preference only accounted for in recommended?

lucidrains 11y ago

Hey, if you need any help expanding the site, I'd be game to help out!

jchung 11y ago

As someone who uses arxiv only very rarely, can someone please explain how this preserves sanity?

0x54MUR41 11y ago

That's an awesome website. I hope it will support TLS for registration or login mechanisms.