3 ms·

thanks for the dataset release! So many questions. Have you thought about any approaches to assess quality/accuracy of comments by using the data itself? One c

thanks for the dataset release! So many questions. Have you thought about any approaches to assess quality/accuracy of comments by using the data itself? One could assume a distribution of accuracy and look for correlated metrics on the comment? Other information such as author or project could be helpful if present, by simplying regularizing comment quality over author/project? If GitHub would be a primary beneficiary of any techniques developed with this dataset, would they volunteer any compute resources toward running models?

mallamanis 7y ago
[I'm one of the Microsoft Research people who worked on this] Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method. Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source. [1] https://arxiv.org/abs/1806.04616 https://arxiv.org/abs/1806.04616