[1/3] BitsAI-CR: Automated Code Review via LLM in Practice (AI column)

Interesting. paper ByteDance, the creator of TikTok on the topic of automated code review using GenAI. Interestingly, I’ve been looking at Google’s approach to this question in three parts of Resolving Code Review Comments with Machine Learning. 1, 2 and 3. But the guys at ByteDance have a slightly different approach: they made a system in which LLM writes MR. (merge requests) Comments in code review, and the guys from Google wrote the comment itself engineer, and LLM made code suggest, which offered a fix, which corresponds to this comment. But let’s take a look at the problems ByteDance has been trying to solve with BitsAI-CR: Get rid of the time-consuming bottleneck in the form of traditional code review Eliminate errors and inconsistencies in comments during code review

In general, the topic of using LLM for code review is fruitful, but there are problems that the authors took into account and tried to eliminate. (By the way, there are more links in the list 50 Other scientific articles, a dozen of which are about code review)

1. Low accuracy and reliability: Many LLM solutions generate inaccurate or irrelevant comments while wasting developer time.
2. Insufficient practicality: Current systems often do not provide the useful feedback that developers actually use.
3. Lack of continual improvement mechanism: Most solutions do not learn from user feedback and do not evolve over time.

To solve these problems, the authors came up with a two-part architecture that combines a high-precision comment generation pipeline to review and data flywheel. This approach not only identifies potential problems in the code with high accuracy, but also triggers a cycle of continuous improvement based on real feedback from developers. It is interesting that the taxonomy of code review rules is used to generate comments, where there are categories of security vulnerability, code defect, maintainability and readability, performance issue.

Let’s take a look at how each of the two components of the model works.

Review Comment Generation Pipeline Pipeline operates in four consecutive stages: 1. Context PreparationAt this stage, input code changes are structured for further analysis. Changes are broken down into segments by header hunks, complete feature definitions are added, giving the LLM enough context for qualitative analysis. 2. RuleChecker: this is a pre-trained LLM, trained on a broad taxonomy of 219 rules of review. RuleChecker identifies potential problems in each code segment, classifying errors by types: vulnerabilities, performance problems, violations of coding standards, etc. Initial comments on the review. 3. ReviewFilterThis verification layer is a key quality control mechanism that filters out RuleChecker false positives and hallucinations. ReviewFilter is also a trained LLM, using three patterns of reasoning: Direct Conclusion, Reasoning-First, Conclusion-First. After the tests, Conclusion-First proved to be the most effective – it best balances accuracy and performance. 4. Comment AggregationAt this stage, similar comments are grouped using cosine similarity so as not to overload developers with duplicate feedback. Each group retains one representative.

The second part of the model, as well as the approach to learning and the results of implementation, we will consider in next post.

#Software #AI #Engineering #Process #DevEx