JOIN
Get Time
long_comps_topcoder  Problem Statement
Contest: Bugzilla ROUND
Problem: Bugzilla

Problem Statement

    

Competition Phase: October 5, 21:00 EDT, 2018 - October 26, 21:00 EDT, 2018

Prizes

1-st: $8 000

2-nd: $5 000

3-rd: $3 000

4-th: $2 000

5-th: $1 000

This match is TCO19 eligible. Everybody who scores above 500k in the system test will get TCO19 points according to these rules.

Problem

Bugzilla is a bug tracking system from Mozilla. It is used to develop and maintain hundreds of software products, consisting of thousands of components. As new bugs are submitted into the system, their triaging (categorization by products and software components) becomes time and resource consuming, as many bug reporters are not closely familiar with internal division of software they use into products and component. Thus, members of Mozilla development team are assigned to be gatekeepers, who categorize and prioritize incoming bugs. This manual bug triaging consumes time of experienced engineers, which may be spent better, otherwise.

In this Marathon Match we give you real data from the bug tracking system, and ask to implement an algorithm for automatic bug triaging. Given details of a submitted bug, your algorithm will deduce the software component that bug belongs to, and the confidence score of this deduction.

The competitor pack contains all materials necessary for this challenge. In its data folder you will find:

  • bugs-training.csv - The training dataset. It includes all triaged bugs reported from July 6, 2012 to July 6, 2016; also, for each software component present in the ground truth, the first 30 bugs reported from July 6, 2012 are included into the training dataset, even if some of them were reported after July 6, 2016. The dataset contains the following columns: bug_id, short_desc (short description), creation_ts (timestamp of bug submission), reporter (ID of reporting user), rep_platform (reporter's platform), op_sys (reporter's OS), assigned_to (ID of user assigned to fix the bug), version (branch/version of the codebase the bug relates to), component_id (ID of the software component the bug belongs to).
  • bugs-testing.csv - The testing dataset. It includes 153127 randomly sorted bugs, reported between July 6, 2016 and July 6, 2018, that have not been included into the training dataset. Your algorithm will triage these bugs, and you will submit the outcomes as solution of the challenge, as explained below. The first 15312 bugs (10% of the set) will be considered for provisional scoring; the remaining 137815 bugs (90% of the set) will be used for the final system test. This dataset contains the same columns as the training one, with the last three excluded (assigned_to, version, and component_id).
  • longdescs.csv - This file contains long descriptions of bugs, both from testing and training datasets. Two columns: bug_id and long_desc
  • components.csv - Description of all software components: name, description, product_id, id, is_active (whether the component is active as of July 6, 2018), scoring_weight (weight of bugs belonging to this component in the challenge scoring; see Scoring section below for details)
  • products.csv - Description of all software products: name, description, id, classification_id
  • classifications.csv - Broad categories of software products: id, name, description
  • profiles.csv - Brief info on registered users: user_id, creation_ts (registration timestamp)
  • versions.csv - Versions and branches of software products: value (name tag), product_id, id
  • bugs-local-testing.csv and bugs-local-ground-truth.csv are sample input and ground truth files than can be used with the scoring code provided in the js-harness folder of competitor package. They include bugs reported from July 6, 2014, to July 6, 2016, excluding the first 30 bugs in each component (counting from July 6, 2012). Note that bugs in these files are ordered by reporting time, while in the real testing data the bugs to triage are sorted randomly.

Given the input dataset bugs-testing.csv, for each bug your algorithm will produce two values: component_id (predicted ID of the software component corresponding of the bug), and confidence_score (estimated confidence score of your prediction, which should be a positive number in 0.0 - 1.0 range; a higher confidence score means a higher confidence in the component ID prediction). The output of your algorithm should be written into a CSV file, with the header line reading component_id,confidence_score, followed by comma-separated values of component_id and confidence_score; one line per bug, in the same order of bugs, as in the input file. You will upload resulting CSV file to a file sharing service, and submit to the Topcoder system the code returning direct download URL of your solution:

public class Bugzilla {
  public String getAnswerURL() {
    return "DIRECT/URL/OF/YOUR/SOLUTION";
  }
}


Scoring

In a few words: to score your predictions, we’ll sort them by decrease of the confidence score; then we go along the predictions, from the highest to the lowest confidence score, and we sum-up scoring_weights of the bug categories for all correctly categorized bugs, reducing each element of the sum proportionally to the sum of scoring_weights for incorrectly categorized bugs seen so far. Finally, we normalize the result so that 1 000 000 is the maximum possible score, and 500 000 is the threshold to be eligible for a prize.

The formal definition of the scoring algorithm:

  1. For each bug bi ∈ B, where B is the test dataset containing N bugs in total (the first 10% of bugs from the testing dataset during provisional scoring; the other 90% of the dataset during the final system test), the tested algorithm will generate a pair of predicted component ID cbi and the corresponding confidence score εbi.
  2. Let gbi to be the correct component ID (ground truth) for the bug bi, and wbi=w(gbi)the score weight for the bugs (score weight of the bug exclusively depends on the ground truth component ID for the bug, and scoring weight for each component is provided in the scoring_weight column of components.csv file).
  3. Define abi=1 if cbi=gbi, abi=0 otherwise (i.e. it is 1 if a prediction is correct, and 0 if it is incorrect).
  4. Sort the triplets (abibi,wbi) in the order of εbi decrease, and re-index them so that aii,wifor i=1,2,...,N correspond to predictions done by the scored algorithm, in the order of confidence decrease.
  5. Define the sum of all weighting scores as:

  6. Define the sum of weighing scores for all incorrect component ID predictions in the range from 1 to n as:

  7. Define the score as:



    This way each correct component prediction is scored proportionally to the its scoring weight, reduced by a factor depending on the number of mistakes done in predictions with higher confidence scores. The score is normalized so that 1 000 000 is the maximum achievable score. An algorithm that classifies bugs with 50% accuracy, and reports random confidence scores, scores about ~ 375 000 on the local testing dataset. A similar algorithm with 50% accuracy, but with 80% accuracy in estimation of confidence scores, scores ~450 000. Here are sample visualization of these two cases:



    Each blue and red pixel on these images correspond to a correct and wrong predictions of a bug’s component ID. These predictions are sorted so that the top-left pixel corresponds to the prediction with the highest confidence, and the confidence decreases from left-to-right, then from top-to-bottom.

Eligibility

  • Mozilla employees and their immediate families are not eligible to participate in this challenge.
  • Softvision employees can participate in the challenge, but only if they submit tools created specifically for this challenge.

Additionally, to be eligible for a prize you must:

  • Qualify in the prize zone in the system scoring phase. The minimal score to be eligible for a prize is 500 000.
  • Provide your algorithm, within 7 days after announcement of results, along with any assets/materials necessary to use and train it, and a write-up explaining the training, using, and theory behind your approach;
  • Your algorithm must rely exclusively on the data provided in the competitor package; you may use pre-trained language models, if necessary;
  • Your solution must be MPL2-compatible (does not rely on any third-party libraries which can prevent the licensing of your deliverable under Mozilla Public License 2.0);
  • To claim the first prize, the winner is expected to re-train his algorithm on the latest bug data that will be provided after the challenge (performance of the re-trained algorithm will not affect the final scores and ranking);
 

Definition

    
Class:Bugzilla
Method:getAnswerURL
Parameters:
Returns:String
Method signature:String getAnswerURL()
(be sure your method is public)
    
 

Examples

0)
    
Seed: E

This problem statement is the exclusive and proprietary property of TopCoder, Inc. Any unauthorized use or reproduction of this information without the prior written consent of TopCoder, Inc. is strictly prohibited. (c)2010, TopCoder, Inc. All rights reserved.