Moving Faster and Reducing Risk: Using LLMs in Release Deployment (AI column)

Interesting. lighting talk DPE Summit 2024 Rui Abreu from Meta on improving the reliability of releases with LLMs (Meta is recognized as an extremist organization and its activities are banned in Russia).

The highlights of this presentation are presented below.

1. The Problem of Scale in Release Engineering On the scale of Meta, it is impossible to centrally decide what to release in production. Responsibility is transferred to developers who write and revise code. 2. The Productivity TriangleFor kids, productivity is a balance between three dimensions: speed. (velocity)reliability (reliability) legibility (readability). Optimizing one parameter affects the others. 3. Diff risk assessment system They developed models to predict the likelihood that diff will cause a major failure affecting users. 4. The evolution of code freeze practices The guys have moved from completely blocking changes to intelligent gateways based on risk assessment. 5. Multi-level gated There are four types of restrictions: Green (limitlessly)weekends (5%)average risk (10%) high-risk (50% of the riskiest diffs). 6. Comparison of approaches to ML - checked models based on logistic regression, BERT-based models (StarBERT) generative LLMs for risk prediction. 7. Excellence of risk-aligned LLM Risk-aligned iDiffLlama-13B models won 8. Simplification of feature engineering For LLM models, only three input parameters were required: the name of the changes, the test plan and the code changes themselves. (It is interesting that the summary of changes in the model are not served, as they allow manipulation.). 9. Practical applications As a result of the implementation of these risk models, the guys were able to come to the recommendations of auditors for risky diffs, automatic approval of low-risk changes, and overall improvement of the release process. 10. Balance of speed and quality It is possible to "unfreeze" some diffs during code freeze, without blocking all changes to code freeze. 100This increases the productivity of engineers.

As a result, it’s an interesting approach to improve quality through flexible release risk management. I wonder how it really works at Meta and whether it’s rolled out to all products/projects.

#AI #Software #Engineering #Architecture #Agents #Math #Software #ML