Just-in-Time Catching Test Generation at Meta (Category Engineering)

Researchers banned in Russia Meta wrote interesting whitepaper About a new class of autotests - regression catching tests. Standard. hardening They must be insured against future regressions. New catching tests You should drop on a specific incoming diff and pass on the parent version. The idea is not to wait for the bug to pop up later, but to try to catch it at the time of a review or CI, before the code is injected. Interestingly, the tests generated by the authors should detect regressions not based on implicit oracle. (Responding to obvious problems such as crashes)on the basis of a common oracle (general oracle)It represents correct behavior, often poorly defined and only partially known. The authors distinguish between weak catch and strong catch: Weak catch is just the one that falls on the current audit Strong catch means that the test does indicate (true positive) Real error in expected behavior (general oracle-based) Strictly weak catch is false positive, which indicates a problem with the test set The main task is to learn how to determine that weak catch is actually strong catch. This is interesting because LLM is not used as another unit test generator for the sake of coverage, but as part of an early regression interception system. In Meta, this circuit is run on high-risk diffs. (PR analogue in GitHub) It runs on hundreds of millions of lines of code, and it targets heavy, expensive bugs, not cosmetic failures.

The main results of the article are present directly in the abstract. (first paragraph of any article). In fact, the goal was to beat the development slowdown due to false positive tests. The authors analyzed 22,126 Tests generated and showed that Code-change-aware methods improve the generation of candidates catching problems in 4 compared to the usual hardening tests and 20 Compared to Accidentally Falling Tests To combat false positives, the authors used rule-based and LLM-based assessors. These approaches reduce the burden on people 70% Inferential statistical analysis shows that human-accepted code changes have significantly more false positives, and human-rejected changes have more true positives.

• Authors report 41 candidate catches on engineer, 8 of which are true positives, and 4 They could have caused a lot of problems if they hadn’t been caught.

False alarms, according to the authors, usually closed in minutes and almost did not hit the developer velocity. The authors do not go too far: they directly write that the sample is too small to draw too strong scientific conclusions about the share of heavy bugs. But they note that this approach is scalable, industry-friendly, and prevents major problems.

The main conclusion for me is this: this is not a paper about “magical automation of QA”, but a paper about a working engineering circuit. LLM + mutation testing + cheap human-in-the-loop already provide a way to argue with the incoming diff and ask the author a very useful question: “Are you sure you wanted to change behavior this way?” In the next series, the authors plan to add a better recovery of change intent + take a richer diff context and come to a more confident separation of weak catch from strong catch.

#AI #QA #Engineering #Whitepaper #Software #LLM #DevOps