5 ms·

This is a pretty cool research area, but isn't this implementation basically just a Javadoc search engine? A basic keyword search of the javadoc descriptions wo

This is a pretty cool research area, but isn't this implementation basically just a Javadoc search engine? A basic keyword search of the javadoc descriptions would return similar results in most cases, wouldn't it?

nl 10y ago
Section 5.2 compares it to search techniques. It's a bit more than twice as good as the next best method.
- simongray 10y ago
  Ok. I definitely see the value then.
- daveguy 10y ago
  The technique is a bit more complex than simple text search. It uses actual click-through data to determine quality of the search result as it applies to the query about an API. The SWIM (synthesizing what I mean) paper can be found here: https://arxiv.org/abs/1511.08497 https://arxiv.org/abs/1511.08497 Also of note from section 5.2: SWIM uses Bing clickthrough data to build the model. Using a better (or simply more used) search engine like google search would likely improve the SWIM results. EDIT: The method they use to compare the methods is BLEU which stands for Bi-Lingual Evaluation Understudy and was developed for automated machine translation evaluation. Apparently CS authors no longer bother with expanding acronyms the first time they are used. Paper is here: http://www.aclweb.org/anthology/P02-1040.pdf http://www.aclweb.org/anthology/P02-1040.pdf EDIT2: Also, for the BLEU comparison they compare the computer generated API sequence to a human-written API sequence. However, they give no details on who or how the human-written sequence is developed. Are the researchers coming up with their own API sequence? Are they using mechanical turk? Interns? There could be significant bias depending on how these human-written sequences are generated.
  - zodiac 10y ago
    > Apparently CS authors no longer bother with expanding acronyms the first time they are used. What does CS stand for? :p But yeah, I might expect BLEU to be unexpanded in an NLP paper maybe, but not here
    - daveguy 10y ago
      > What does CS stand for? Haha. Well played. :) I just think it's good practice in general, no matter how common, to use the full name when the acronym is introduced. Especially if it is a method used in the paper.
  - nl 10y ago
    They also test a Lucene based model, which is much worse on Top1, but similar to SWIM for Top 3 and Top 10. Also, BLEU is a fairly widely used metric. I assumed they referenced the paper though?

nl 10y ago

Section 5.2 compares it to search techniques. It's a bit more than twice as good as the next best method.

simongray 10y ago
Ok. I definitely see the value then.
daveguy 10y ago
The technique is a bit more complex than simple text search. It uses actual click-through data to determine quality of the search result as it applies to the query about an API. The SWIM (synthesizing what I mean) paper can be found here: https://arxiv.org/abs/1511.08497 https://arxiv.org/abs/1511.08497 Also of note from section 5.2: SWIM uses Bing clickthrough data to build the model. Using a better (or simply more used) search engine like google search would likely improve the SWIM results. EDIT: The method they use to compare the methods is BLEU which stands for Bi-Lingual Evaluation Understudy and was developed for automated machine translation evaluation. Apparently CS authors no longer bother with expanding acronyms the first time they are used. Paper is here: http://www.aclweb.org/anthology/P02-1040.pdf http://www.aclweb.org/anthology/P02-1040.pdf EDIT2: Also, for the BLEU comparison they compare the computer generated API sequence to a human-written API sequence. However, they give no details on who or how the human-written sequence is developed. Are the researchers coming up with their own API sequence? Are they using mechanical turk? Interns? There could be significant bias depending on how these human-written sequences are generated.
- zodiac 10y ago
  > Apparently CS authors no longer bother with expanding acronyms the first time they are used. What does CS stand for? :p But yeah, I might expect BLEU to be unexpanded in an NLP paper maybe, but not here
  - daveguy 10y ago
    > What does CS stand for? Haha. Well played. :) I just think it's good practice in general, no matter how common, to use the full name when the acronym is introduced. Especially if it is a method used in the paper.
- nl 10y ago
  They also test a Lucene based model, which is much worse on Top1, but similar to SWIM for Top 3 and Top 10. Also, BLEU is a fairly widely used metric. I assumed they referenced the paper though?