Skip to content
#ML

Precision and recall on fingers (Category ML)

#ML #PopularScience #AI #Engineering #Software

Just yesterday, we discussed the whitepaper about the ByteDance approach to code review. (1, 2 and 3)They gave preference to precision and sacrificed recall when designing their solution. Precision (precision) recall (fullness) These are two important metrics for assessing the quality of classification models, especially when classes are unevenly distributed or when it is important to minimize certain errors. But what does that mean in plain language? Let's talk.

Precision (precision) Of all the objects that the model identified as positive, how many are actually positive? The formula looks like this. **Precision = TP / (TP + FP)**where

  • TP (True Positive) correctly defined positive objects,
  • FP (False Positive) Wrongly defined as positive objects The easiest way to explain the example of Vasi, that guards the sheep and shouts "Wolf" when he thinks that the wolf has come. Precision shows how often when Vasya shouted "Wolf!" the wolf actually came. If Vasya often confuses a dog with a wolf and raises an alarm in vain, precision will be low.

Recall (fullness) Of all the positive objects that exist, how many have the model found? The formula for recall is similar, but not entirely. **Recall = TP / (TP+FN)**where

  • FN (False Negative) Positive objects that the model missed Let's continue with Vasya's example of sheep. Recall shows how attentive Vasya is: of all the cases when a wolf did come, how many times did Vasya notice it and shout "Wolf!"? If Vasya often sleeps and misses the wolf, recall will be low.

There is usually a trade-off between these metrics: if you raise one, the other can go down. For example, if the model is “reinsured” and marks few objects as positive (only when sure)Precision is high and recall is low. If it marks almost everything as positive, recall will be high and precision low.

In the end, we're doing it. Vasya often shouts "Wolf!" without even being sure - Precision is low: many false alarms - Recall is high: almost always notices Vasya rarely screams, only if he is absolutely sure - Precision is high: almost no mistake - Recall is low: often misses the wolf

As a result, in the case of an assistant in code review, it is not profitable to shout “Wolves” too often, since in this case, engineers will simply stop paying attention to the help of the assistant’s code review:)

#ML #PopularScience #AI #Engineering #Software