[2/3] Resolving Code Review Comments with Machine Learning (AI column)

Continue. story forward whitepaper Google explains the evolution of the system. V1) It all started with generating suggests changes after sending comments code reviewers V2) The tool evolved to suggest code changes to reviewers while writing comments. V2 + IDE integration) Based on user feedback, the team implemented better integration with the IDE, including reviewing conflicting changes. (3-way-merges)

As a result, the current process of this feature and funnel

• Reviewer writes a comment - on the fly model generates code for fix
• The reviewer can accept this code or not, if it accepts, then the author of MR will receive a comment with this auto suggest, if it rejects, then just a comment
• Next, the author of the code during the review can shake the auto-accept button

The goal of the whole work and the metric for optimization was formulated as follows:

A primary goal for any assistance tool is to increase productivity. One metric we use to gauge the positive impact of our assistant on productivity is acceptance rate, the fraction of all code-review comments that are resolved by the assistant; this measures, out of all (non-automated) comments left by human reviewers, what fraction received an ML-suggested edit that the author accepted and applied directly to their changelist.

Each stage looks more interesting than just one. 7.5Percentage of auto receptions suggest from all comments.

Stage -- (%) of total -- (%) of previous step Incoming comments -- 100.0% -- 100.0% Confident predictions -- 49.0% -- 49.0% Accepted by reviewer -- 33.1% -- 63.6% Previewed by authora -- 10.7% -- 34.5% Applied by author -- 7.5% -- 69.5%

Well, the preview step in this statue is not as significant as in the first version of how they made this feature, there is such annotation under the plate.

The concept of author preview is less significant in V2. The author automatically sees a small preview and can “click-to-view” full suggested edits. This full view either shows the “Apply“ button or informs about an edit that requires a three-way merge. Almost all not-applied previews in V2 denote an edit that required a three-way merge to be applied.

High-quality feedback on this feature sounded like this.

Early feedback about the assistant in internal message boards is enthusiastic, including characterizations such as “sorcery!”, “magic!”, “impressive”. Although the new version V2, in which suggested edits are presented as the reviewer is typing a comment, has only been deployed to 100% of the population for a relatively limited time, we have received delighted reports demonstrating that just the location and the initial sentiment of the reviewer’s comment can lead to helpful suggested edits, for both parties involved.

In addition to the results, the authors described how they tuned the quality of the system through model tuning and data tuning. - Model tuning included: fine-tuning DIDACTR models, size tuning of the number of parameters in the model, reducing precision, tuning hyperparameters, tuning for programming languages and final preview for reviewers, which allowed even the niche to cut off precision - Data tuning included: offline dataset for evaluation limited to single comment changes, training on "done" comments, training on synthetic problems

In general, it turned out an interesting whitepaper with a description of the approach of the guys in Google and interesting practical results. In last post Interesting pictures from this article.

#Software #AI #Engineering #Process #DevEx