[2/3] BitsAI-CR: Automated Code Review via LLM in Practice (AI column)

Continue. consideration An interesting whitepaper about code review from the second part of the model, namely the mechanism data flywheel

Data flywheel is a systematic approach to continuous improvement through: 1. Annotation Feedback IntegrationBitsAI-CR collects and uses user feedback to further educate and improve datasets. 2. Outdated Rate MeasurementA new metric, Outdated Rate, measures the percentage of lines of code changed after BitsAI-CR tagged them. This gives an objective assessment of how often developers actually accept the system's proposals. 3. Dynamic Rule AdjustmentThe system constantly adjusts review rules based on accuracy and Outdated Rate, removing those that generate low-value comments. (Low Outdated Rate with high accuracy). Together, these components create a feedback loop that step by step improves the quality of code review based on real developer behavior.

The authors applied an advanced BitsAI-CR learning and optimization strategy that includes several key elements: Doubao-Pro-32K was chosen as the base model.0828 (Own LLM ByteDance)due to data security and privacy requirements. Sequence size selected 8192 token, since 99The percentage of review examples falls within this limit.

• LoRA technique was used to train RuleChecker and ReviewFilter (Low-Rank Adaptation)
• They also studied on the basis of a detailed taxonomy of code review rules, which Structured framework for identifying and classifying problems in code Systemic collection and markup of data for training Clear criteria for assessing system quality This approach proved to be effective: BitsAI-CR, trained in taxonomy, achieved accuracy 57.03% whereas the non-taxonomy version (random human review) just 16.83%.

To evaluate the model, the authors of the new metric - Outdated Rate, which measures the proportion of lines of code changed after BitsAI-CR comments. This allows:

1. Automatically assess the real impact of comments in practice
2. Understand how system proposals are actually implemented by developers This approach closes the drawbacks of traditional precision metrics, which require manual markup and do not reflect real business benefits.

When implemented in production system showed impressive technical results

1. precision (Precision): Without taxonomy and a two-stage architecture, accuracy was about 25%. After the introduction of these elements, the accuracy of RuleChecker has increased. 27.9percentage 62.6% and ReviewFilter with 35.6percentage 75.0%. This proves the effectiveness of the two-stage architecture in reducing false positives.
2. Outdated Rate: For the Go language (primary) Outdated Rate has grown to 26.7percent 18 weeks of optimization. For comparison, in humans, this indicator varies between 35-46%. The gradual approach to the human level is a testament to the practical value of BitsAI-CR.

In addition, BitsAI-CR quickly became part of the development process.

1. AudienceMore: 12 000 Weekly Active Users (WAU)above 210 000 Weekly Page Views (WPV) The system is integrated into the workflow.
2. Retention: Retention in week two -- 61.64%, after eight weeks, about 48%. This is the first published retention benchmark for such code intelligence tools.
3. User evaluation: In the poll. (N=137) and expert interviews (N=12) 74.5% (102/137) Users have confirmed the benefits and effectiveness of BitsAI-CR. All experts noted the benefits of BitsAI-CR, pointing to wishes for speed, customization and support of languages.

The authors outlined clear plans for the development of BitsAI-CR:

1. Expand language support to all programming languages, not just the five main languages.
2. Improved contextual understanding. BitsAI-CR analyzes code at the function level with limited context. In the plans - cross-file analysis (cross-file review)To find problems related to architecture and dependencies between files.
3. Continuous development and improvement of the system

#Software #AI #Engineering #Process #DevEx