Overview

Software development processes should consider the impact of its policies in the quality of the software product when designed. Factors like salary scales, promotion policies and recognition mechanisms have direct influence in the way software artifacts are produced, hence they have impact in its quality.

An instance of this problem is observed in the bug reporting process, where testers are encouraged to increase the reported priority in order to maximize the number of fixes delivered. This situation — called Priority Inflation by practitioners — is analyzed using game theory. Game theory uses mathematical models to study conflict and cooperation, and and we use them in this paper to identify inflation causes and to propose a solution to this problem.

We built a simulation model of bug reporting as a part of our empirical game-theoretic analysis, finding that what we call an Unsupervised Prioritization process it is not incentive compatible since at equilibrium the Heurisitic Dishonest strategy has a probability of 1.0. We also evaluated our proposed bug reporting process —based on inflation penalties and reporter reputation— finding that with only a 3% penalty factor we can move equilibrium towards a profile where the euristic Honest strategy has a probability of 1.0.


The Priority Inflation Problem

We designed a survey to assess the magnitude of the priority inflation problem. Through convenience sampling we were able to reach 152 software engineers, whose answers were collected using the Google Forms platform.

Priority Deflation

Priority Deflation

14.97% of the respondents are working on projects
where the bug report priorities are frequently
understated, while 64.63% of them work on projects
where this scenario happens occasionally

Priority Inflation

Priority Inflation

24.66% of the participants are working on projects
where priority inflation is frequent while
64.38% of them participate on a project
where this happens occasionally. .

Dishonesty Impact

Dishonesty Impact

About 31.33% of them mention that understated/
overstated priorities have a significant impact
on their daily duties, while 50% of
them believe the impact is minimum.


Game-Theoretic Process Improvement

Empirical Game-Theoretic Analysis

The priority inflation game is played by bug reporters, who are in charge of discovering and reporting the bugs that are present on a specific version of a software system. When reporting a bug, the reporters are required to assign a priority label to a bug report, which is related to the urgency for a fix for this bug to be delivered. The bug reporter objective is to maximize its reporter score, which is a function of the bugs reported that got fixed and its ground-truth priority.

Formal game-theoretic analysis faces tractability problems when dealing with interactions among several agents with a large number of strategies available to them. To address these issues, empirical game-theoretic analysis relies in game reduction techniques and simulation to allow a rigorous analysis. The empirical game-theoretic approach used in this project is described in the diagram above.

Simulating Bug Reporting

Empirical Game-Theoretic Analysis

The pay-off function of a bug reporter playing the Priority Inflation Game depends on the number of reported bugs that got fixed as well as its ground-truth priority. Pay-off calculation is critical for equilibrium analysis as we need the pay-off value for each player on each strategy profile. We will rely on a simulation model —illustrated above—of the bug reporting process for obtaining the number of bug reports fixed per player per strategy profile.


New Mechanisms for Bug Reporting

After defining the internals of our empirical game, we perform the equilibrium analysis of three bug reporting processes: Two of them are taken from existing practices in the industry and one is proposed by the authors.

The Unsupervised Prioritization Mechanism

Unsupervised PrioritizationWhat we call a unsupervised prioritization process is a reporting process where one team has assigned the bug reporting and bug prioritization responsibilities while a different one is in charge of bug fixing.
By applying the procedure described before, we have found an equilibrium in pure strategies were the probability of the Heuristic Dishonest strategy is 1.0 and 0.0 for the other 6 strategies in our catalog. The dominance analysis indicates that this is an equilibrium on dominant strategies.

A Gatekeeper-Based Reporting process

Unsupervised PrioritizationIn a gatekeeper-based process, the bug reporting and bug prioritization responsibilities are assigned to different teams.
The first gatekeeper configuration explored considers that gatekeepers can detect 90% of mislabeled priorities: The equilibrium profile obtained through the process described previously has a probability of 1.0 for the Empirical Inflator strategy while 0.0 for the other 6 strategies.

A Throttling Mechanism

Throttling MechanismWe propose a mechanism — which we call the throttling bug reporting process — that maintains the decentralized prioritization attribute of unsupervised prioritization Reporting but incorporates a priority assessment phase like in a gatekeeper-based process:

When we applied a penalty of 3% in case of dishonesty, the number of equilibrium is only one where the Heuristic Honest strategy has a probability of 1.0. We obtained the same result when the penalty value was set to 5%.


A Support Tool for Throttling Bug Reporting

Issues
Reporters

As part of this project we built a software tool to support the teams that adopt a throttling bug reporting process, since according to our analysis is the best option to address Priority Inflation. Our tool — called JIRA Inflation Tracker — is a Chrome extension for supporting teams that are using JIRA as their bug tracking system.