6 ms·

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

optimalsolver 3y ago
I'd like to imagine quirky titles make them harder for other researchers to find, leading to lower citation counts, but that may be wishful thinking.
- bundie 3y ago
  I think this trend is happening because of the entire buzz around OPENAI LLMs and that 2017 Google Research Paper
  - rsfern 3y ago
    Ironically, attention is what you need to get high citation counts Whether quirky titles help or hurt with that i think depends on name recognition and the publication venue
- skybrian 3y ago
  I don’t think it makes much difference. If anything, it might help? It’s easier to search for a paper if you remember its name. If not, you can search on author or words in the abstract. The problem isn’t quirky titles, it’s with websites like Hacker News that only display a headline and not the abstract.
hikingsimulator 3y ago
There is a lot of jargon in ML, an example is found in the object detection literature where you will often find sentences like such: "we present a novel simple non-hierarchical feature network backbone with window-shifted attention based learning accommodating neck+regression head fine-tuning or masked cascade RCNN second stage." I'm half joking. Surveys are often a godsend.
- alephxyz 3y ago
  You forgot to toss in a few "state-of-the-art"
- chas 3y ago
  The major ML conferences all have pretty tight page limits, so more expository sentences usually get cut. This also means that papers usually only explain how their work is different from previous work, so they assume you are familiar with the papers they cite or are willing to read the cited papers. This means that people who have an up-to-date knowledge of a given subfield can quickly get a lot out of a new papers. Unfortunately, it also means that it usually takes a pretty decent stack of papers to get up to speed on a new subfield since you have to read the important segments of the commonly cited papers in order to gain the common knowledge that papers are being diffed against. Traditionally, this issue is solved by textbooks, since the base set of ideas in a given field or subfield is pretty stable. ML has been moving pretty fast in recent years, so there is still a sizable gap between the base knowledge required for productive paper reading and what you can get out of a textbook. For example, Goodfellow et al [1] is a great intro to the core ideas of deep learning, but it was published before transformers were invented, so it doesn’t mention them at all. [1] https://www.deeplearningbook.org/ https://www.deeplearningbook.org/
visarga 3y ago
> "Understanding Limitations and Chaos-Based Failures in Gradient-Based Optimization Methods." (gpt4) Fixed with ML.
- xpe 3y ago
  I’ve provided the PDF URL to ChatGPT 4.0 and asked it to summarize the article with alternative titles, but for some reason it keeps getting the original title, authors, abstract, and body wrong. What prompt did you use?
  - visarga 3y ago
    I asked a bunch of things. So initially I posted the abstract and title, then prompted just: "Explain". Then I asked "Give me background knowledge on spectrum of the Jacobian" and "Explain the title of the paper" and in the end "Reformulate the title in a more explicit manner". Maybe you can skip directly to the last prompt.
  - wcoenen 3y ago
    That's because chatgpt doesn't have the ability to retrieve PDFs from the internet. (Unless maybe if you have early access to the version with plugins?) Bing chat does have the ability to read opened PDFs when used from the Edge side bar.
satvikpendem 3y ago
The clever titles are more brandable. See the citations for "Attention is all you need" or "Chinchilla limit" versus more mundane titles.
- rolisz 3y ago
  I'm pretty sure Attention is all you need would have gotten a lot of citations even if it had a "boring" title. It was a groundbreaking paper with lots of good ideas.
- cubefox 3y ago
  The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.
  - dragonwriter 3y ago
    I mean, it would be slightly more accurate if it had been “Compute-Optimal Training of Large Language Models”, since the models so-trained aren’t the thing that is compute-optimal, the training is compute-optimal. But… yeah, its hardly a title chosen for marketing rather than description.
    - sroussey 3y ago
      But we all know what you mean when you say “the Chinchilla paper”
    - moritzwarhier 3y ago
      Stupid question, is training considered a part of the model? Or is this only common parlance for GPT (the P).
    - cubefox 3y ago
      Interestingly, the models themselves also have compute cost associated with them, namely the cost of running them (inference). The smaller a model is, the less compute it needs for inference. Which is an interesting trade-off, because you can overtrain (rather than undertrain, as in the past) a model, i.e. use more tokens, fewer parameters, and ultimately more compute than optimal during training, to get lower inference cost. https://www.harmdevries.com/post/model-size-vs-compute-overhead/ https://www.harmdevries.com/post/model-size-vs-compute-overh... This website has an interesting graph which visualizes this: > For example, the compute overhead for 75% of the optimal model size is only 2.8%, whereas for half of the optimal model size, the overhead rises to 20%. As we move towards smaller models, we observe an asymptotic trend, and at 25% of the compute-optimal model size, the compute overhead increases rapidly to 188%. So if you train your model you shouldn't just look at compute-optimal training but try to anticipate how much the model will probably be used for inference, to minimize total compute cost. Basically, the less inference you expect to do with your model, the closer the training should be to Chinchilla-optimality.
sho_hn 3y ago
Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.
- preommr 3y ago
  > Branding is eating the world. Not realizing that it always has is why it's still effective. History is written by the victors and victors tend to have the best marketting.
  - deleted 3y ago
    [deleted]
- mumblemumble 3y ago
  It's inescapable. 40 years ago, when articles went into print publications, you'd just get your paper into a key print journal and then trust that everyone who gets it would at least look through the article headlines and read the abstracts of articles that seemed relevant to them. And it was manageable because you'd only have a few new issues rolling in per month. But arXiv had an average of 167 CS papers being submitted per day in 2021. An academic who wants to keep their career alive needs to resort to every trick in the book to be heard above that din.
  - V__ 3y ago
    Isn't this more a problem of not good enough curation? There are more papers now than ever, but the signal is probably getting worse and worse.
    - mumblemumble 3y ago
      How does one, practically speaking, curate article publication in a world with arXiv? Curation kind of requires gatekeepers, and good gatekeeping is a full-time job that needs to be paid for. In the days of print journals, there was both a motivation, because you can only print so many articles per month due to materials costs, and a steady revenue stream to pay for it, because everyone has to pay for the journals to get any access to articles at all. The Internet's been eroding both of those factors for a good quarter century now.
dauertewigkeit 3y ago
At some point you run into the problem where titles become useless because there are 100 papers on the same exact topic with very slight variation in the title. At this point people use surnames and date to cite papers. But then the title can become something catchy that will give you more visibility.
scrubs 3y ago
Agree -- owl talk of Winnie-the-pooh childhood fame --- sucks.
xpe 3y ago
Getting cited is probably the largest individual incentive. To various degrees, authors also want to get their ideas and reputation “out there”. To that end, authors want their work to be (i) skimmed (the abstract at least); (ii) shared; (iii) read. A catchy title almost always helps (right?); there doesn’t seem to be significant downside to catchy titles. So how do we get more desirable system behavior? It seems we have a collective action problem.
tough 3y ago
I asked GPT > https://arxiv.org/abs/2111.05803 https://arxiv.org/abs/2111.05803 make a good useful title that's not silly for this paper and wht's about I dunno if it even read the article lols ==== Title: “Deep Reinforcement Learning for Multi-Agent Navigation in Complex Environments” This paper proposes a deep reinforcement learning approach for multi-agent navigation in complex environments. The proposed method uses a centralized critic and decentralized actor architecture, where each agent has its own policy network and learns from its local observations, while the critic network takes in the global state information to provide a centralized value estimate. The method is evaluated on a variety of benchmark environments and achieves state-of-the-art performance compared to existing methods.
- Bootvis 3y ago
  ChatGPT doesn’t read URL’s on the fly but it will pretend it does.
  - tough 3y ago
    lol
ajb 3y ago
There's quite a long history of these titles, even before ML: "Sometime’ is Sometimes ‘Not Never’" - Lamport "Reflections on trusting trust" - Thompson "On the cruelty of really teaching computer science" - Dijkstra I'm sure there are more
abhgh 3y ago
Your comment reminded me this talk [1] by David Duvenaud where he discusses his work on Neural ODEs, introduced in the paper [2] titled "Neural Ordinary Differential Equations" (this was one of the Best Papers in Neurips 2018). Apparently the original name of the paper was "Training Infinitesimally-Layered Neural Networks by Backpropagating through Black-box ODE Solvers". I found that to be much better description for the paper. It seems he thinks so too, but eventually the catch-phrase-y title won out. In the video [1], this bit is at 00:48-01:22. PS: [1] is a good talk in general where he discusses some of the limitations of the paper and things that could have been done better. [1] https://www.youtube.com/watch?v=YZ-_E7A3V2w https://www.youtube.com/watch?v=YZ-_E7A3V2w [2] https://papers.nips.cc/paper_files/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf https://papers.nips.cc/paper_files/paper/2018/file/69386f6bb...
nextaccountic 3y ago
This also happens with security vulnerabilities as well (Heartbleed, POODLE, GHOST, Shellshock, FREAK, CRIME, SMACK)

optimalsolver 3y ago

I'd like to imagine quirky titles make them harder for other researchers to find, leading to lower citation counts, but that may be wishful thinking.

bundie 3y ago
I think this trend is happening because of the entire buzz around OPENAI LLMs and that 2017 Google Research Paper
- rsfern 3y ago
  Ironically, attention is what you need to get high citation counts Whether quirky titles help or hurt with that i think depends on name recognition and the publication venue
skybrian 3y ago
I don’t think it makes much difference. If anything, it might help? It’s easier to search for a paper if you remember its name. If not, you can search on author or words in the abstract. The problem isn’t quirky titles, it’s with websites like Hacker News that only display a headline and not the abstract.

hikingsimulator 3y ago

There is a lot of jargon in ML, an example is found in the object detection literature where you will often find sentences like such: "we present a novel simple non-hierarchical feature network backbone with window-shifted attention based learning accommodating neck+regression head fine-tuning or masked cascade RCNN second stage." I'm half joking. Surveys are often a godsend.

alephxyz 3y ago
You forgot to toss in a few "state-of-the-art"
chas 3y ago
The major ML conferences all have pretty tight page limits, so more expository sentences usually get cut. This also means that papers usually only explain how their work is different from previous work, so they assume you are familiar with the papers they cite or are willing to read the cited papers. This means that people who have an up-to-date knowledge of a given subfield can quickly get a lot out of a new papers. Unfortunately, it also means that it usually takes a pretty decent stack of papers to get up to speed on a new subfield since you have to read the important segments of the commonly cited papers in order to gain the common knowledge that papers are being diffed against. Traditionally, this issue is solved by textbooks, since the base set of ideas in a given field or subfield is pretty stable. ML has been moving pretty fast in recent years, so there is still a sizable gap between the base knowledge required for productive paper reading and what you can get out of a textbook. For example, Goodfellow et al [1] is a great intro to the core ideas of deep learning, but it was published before transformers were invented, so it doesn’t mention them at all. [1] https://www.deeplearningbook.org/ https://www.deeplearningbook.org/

visarga 3y ago

> "Understanding Limitations and Chaos-Based Failures in Gradient-Based Optimization Methods." (gpt4) Fixed with ML.

xpe 3y ago
I’ve provided the PDF URL to ChatGPT 4.0 and asked it to summarize the article with alternative titles, but for some reason it keeps getting the original title, authors, abstract, and body wrong. What prompt did you use?
- visarga 3y ago
  I asked a bunch of things. So initially I posted the abstract and title, then prompted just: "Explain". Then I asked "Give me background knowledge on spectrum of the Jacobian" and "Explain the title of the paper" and in the end "Reformulate the title in a more explicit manner". Maybe you can skip directly to the last prompt.
- wcoenen 3y ago
  That's because chatgpt doesn't have the ability to retrieve PDFs from the internet. (Unless maybe if you have early access to the version with plugins?) Bing chat does have the ability to read opened PDFs when used from the Edge side bar.

satvikpendem 3y ago

The clever titles are more brandable. See the citations for "Attention is all you need" or "Chinchilla limit" versus more mundane titles.

rolisz 3y ago
I'm pretty sure Attention is all you need would have gotten a lot of citations even if it had a "boring" title. It was a groundbreaking paper with lots of good ideas.
cubefox 3y ago
The Chinchilla paper was called "Training Compute-Optimal Large Language Models", which is exactly on point.
- dragonwriter 3y ago
  I mean, it would be slightly more accurate if it had been “Compute-Optimal Training of Large Language Models”, since the models so-trained aren’t the thing that is compute-optimal, the training is compute-optimal. But… yeah, its hardly a title chosen for marketing rather than description.
  - sroussey 3y ago
    But we all know what you mean when you say “the Chinchilla paper”
  - moritzwarhier 3y ago
    Stupid question, is training considered a part of the model? Or is this only common parlance for GPT (the P).
  - cubefox 3y ago
    Interestingly, the models themselves also have compute cost associated with them, namely the cost of running them (inference). The smaller a model is, the less compute it needs for inference. Which is an interesting trade-off, because you can overtrain (rather than undertrain, as in the past) a model, i.e. use more tokens, fewer parameters, and ultimately more compute than optimal during training, to get lower inference cost. https://www.harmdevries.com/post/model-size-vs-compute-overhead/ https://www.harmdevries.com/post/model-size-vs-compute-overh... This website has an interesting graph which visualizes this: > For example, the compute overhead for 75% of the optimal model size is only 2.8%, whereas for half of the optimal model size, the overhead rises to 20%. As we move towards smaller models, we observe an asymptotic trend, and at 25% of the compute-optimal model size, the compute overhead increases rapidly to 188%. So if you train your model you shouldn't just look at compute-optimal training but try to anticipate how much the model will probably be used for inference, to minimize total compute cost. Basically, the less inference you expect to do with your model, the closer the training should be to Chinchilla-optimality.

sho_hn 3y ago

Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

preommr 3y ago
> Branding is eating the world. Not realizing that it always has is why it's still effective. History is written by the victors and victors tend to have the best marketting.
- deleted 3y ago
  [deleted]
mumblemumble 3y ago
It's inescapable. 40 years ago, when articles went into print publications, you'd just get your paper into a key print journal and then trust that everyone who gets it would at least look through the article headlines and read the abstracts of articles that seemed relevant to them. And it was manageable because you'd only have a few new issues rolling in per month. But arXiv had an average of 167 CS papers being submitted per day in 2021. An academic who wants to keep their career alive needs to resort to every trick in the book to be heard above that din.
- V__ 3y ago
  Isn't this more a problem of not good enough curation? There are more papers now than ever, but the signal is probably getting worse and worse.
  - mumblemumble 3y ago
    How does one, practically speaking, curate article publication in a world with arXiv? Curation kind of requires gatekeepers, and good gatekeeping is a full-time job that needs to be paid for. In the days of print journals, there was both a motivation, because you can only print so many articles per month due to materials costs, and a steady revenue stream to pay for it, because everyone has to pay for the journals to get any access to articles at all. The Internet's been eroding both of those factors for a good quarter century now.

dauertewigkeit 3y ago

At some point you run into the problem where titles become useless because there are 100 papers on the same exact topic with very slight variation in the title. At this point people use surnames and date to cite papers. But then the title can become something catchy that will give you more visibility.

scrubs 3y ago

Agree -- owl talk of Winnie-the-pooh childhood fame --- sucks.

xpe 3y ago

Getting cited is probably the largest individual incentive. To various degrees, authors also want to get their ideas and reputation “out there”. To that end, authors want their work to be (i) skimmed (the abstract at least); (ii) shared; (iii) read. A catchy title almost always helps (right?); there doesn’t seem to be significant downside to catchy titles. So how do we get more desirable system behavior? It seems we have a collective action problem.

tough 3y ago

I asked GPT > https://arxiv.org/abs/2111.05803 https://arxiv.org/abs/2111.05803 make a good useful title that's not silly for this paper and wht's about I dunno if it even read the article lols ==== Title: “Deep Reinforcement Learning for Multi-Agent Navigation in Complex Environments” This paper proposes a deep reinforcement learning approach for multi-agent navigation in complex environments. The proposed method uses a centralized critic and decentralized actor architecture, where each agent has its own policy network and learns from its local observations, while the critic network takes in the global state information to provide a centralized value estimate. The method is evaluated on a variety of benchmark environments and achieves state-of-the-art performance compared to existing methods.

Bootvis 3y ago
ChatGPT doesn’t read URL’s on the fly but it will pretend it does.
- tough 3y ago
  lol

ajb 3y ago

There's quite a long history of these titles, even before ML: "Sometime’ is Sometimes ‘Not Never’" - Lamport "Reflections on trusting trust" - Thompson "On the cruelty of really teaching computer science" - Dijkstra I'm sure there are more

abhgh 3y ago

Your comment reminded me this talk [1] by David Duvenaud where he discusses his work on Neural ODEs, introduced in the paper [2] titled "Neural Ordinary Differential Equations" (this was one of the Best Papers in Neurips 2018). Apparently the original name of the paper was "Training Infinitesimally-Layered Neural Networks by Backpropagating through Black-box ODE Solvers". I found that to be much better description for the paper. It seems he thinks so too, but eventually the catch-phrase-y title won out. In the video [1], this bit is at 00:48-01:22. PS: [1] is a good talk in general where he discusses some of the limitations of the paper and things that could have been done better. [1] https://www.youtube.com/watch?v=YZ-_E7A3V2w https://www.youtube.com/watch?v=YZ-_E7A3V2w [2] https://papers.nips.cc/paper_files/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf https://papers.nips.cc/paper_files/paper/2018/file/69386f6bb...

nextaccountic 3y ago

This also happens with security vulnerabilities as well (Heartbleed, POODLE, GHOST, Shellshock, FREAK, CRIME, SMACK)