Towards Standardizing and Improving Classification of Bug-Fix Commits

Sarim Zafar, Muhammad Zubair Malik, Gursimran Singh Walia

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Scopus citations

Abstract

Background: Open source software repositories like GitHub are mined to gain useful empirical software engineering insights and answer critical research questions. However, the present state of the art mining approaches suffers from high error rate in the labeling of data that is used for such analysis. This is particularly true when labels are automatically generated from the commit message, and seriously undermines the results of these studies. Aim: Our goal is to label commit comments with high accuracy automatically. In this work, we focus on classifying a commit as a 'Bug-Fix commit' or not. Method: Traditionally, researchers have utilized keyword-based approaches to identify bug fix commits that leads to a significant increase in the error rate. We present an alternative methodology leveraging a deep neural network model called Bidirectional Encoder Representations from Transformers (BERT) that can understand the context of the commit message. We provide the rules for semantic interpretation of commit comments. We construct a hand-labeled dataset from real GitHub commits according to these rules and fine-tune BERT for classification. Results: Our initial evaluation shows that our approach significantly reduces the error rate, with up to 10% relative improvement in classification over keyword-based approaches. Future Direction: We plan on extending our dataset to cover more corner cases and reduce programming language specific biases. We also plan on refining the semantic rules. In this work, we have only considered a simple binary classification problem (Bug-Fix or not), which we plan to extend to other classes and extend the approach to consider multiclass problems. Conclusion: The rules, data, and the model proposed in this paper have the potential to be used by people analyzing open source repositories to improve the labeling of data used in their analysis.

Original languageEnglish (US)
Title of host publicationProceedings - 13th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2019
PublisherIEEE Computer Society
ISBN (Electronic)9781728129686
DOIs
StatePublished - Sep 2019
Externally publishedYes
Event13th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2019 - Porto de Galinhas, Pernambuco, Brazil
Duration: Sep 19 2019Sep 20 2019

Publication series

NameInternational Symposium on Empirical Software Engineering and Measurement
Volume2019-Septemer
ISSN (Print)1949-3770
ISSN (Electronic)1949-3789

Conference

Conference13th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2019
Country/TerritoryBrazil
CityPorto de Galinhas, Pernambuco
Period9/19/199/20/19

Keywords

  • Human Factors
  • Mining Software Repositories
  • Predictive Models
  • Software Maintenance

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Fingerprint

Dive into the research topics of 'Towards Standardizing and Improving Classification of Bug-Fix Commits'. Together they form a unique fingerprint.

Cite this