All the Segmentation Metrics

Segmentation challenges expose us to a lot of metrics and I mean a lot.

So I have decided to make a list of as many as I can and try to explain and illustrate some of these (starting from simple ones and moving to more useful ones).

Let's go!

Note 1: if you have a better suggestion for a banner image, please share it in the comments. :p

Note 2: the title is of course misleading, I won't discuss all the segmentation metrics, I might miss one or two. ;)

Semantic segmentation targets

Before we can start, we have to define what we mean by semantic segmentation.

In semantic segmentation tasks, we predict a mask, i.e. where the object of interest is present.

Let's assume we only have one type of object to predict thus it is a binary task.

We can thus assign the following mapping:

1. 0 for the background class.
2. 1 for the object of interest.

For example, in the hubmap competition, glomuerli) are the 1 class masks.

Now that we know what we are predicting, we can move to the metrics.

Let's start with the easiest one: accuracy.

Accuracy

This isn't the most clever one (will see why soon) but it is for sure the simplest so we should start with it.

How does it work?

We check all the image pixels and assign a score of 1 if the pixel is correctly predicted, 0 otherwise.

Thus, this can mathematically be represented by the indicator function with the following formula:

\(Accuracy(A, B) = \frac{1}{n}\sum_{i=0}^{n}\mathbb{1}_{(A_{i} = B_{i})}\)

i.e. if all the \(A_{i}\)'s and all the \(B_{i}\)'s are equal, the accuracy is equal to 1.

Next, to illustrate the accuracy metric and how it works (and doesn't in some cases), let's see an example.

For that, I will generate an image with half background, half object of interest pixels. Here is the code in the next cell:

Thus, if we have a model that predicts everything as a background pixels (i.e. y = 0 for any input), we will get an accuracy of 0.5 or 50%.

That's a good illustration of the accuracy metric.

What if we have less object pixels?

In fact, that's what happens 99% of the time: the backgournd is the vast majority of the image and the object to predict represents only a tiny percentage of the whole (let's say between few percents to 10s of percents).

Let's take a more realistic example then and see what happens.

Here you can see that we have only few object pixels (in white). Around 10 * 20 + 20 * 20 = 600 pixels out of a total of 256 * 256 = 65536.

That's 0.90%, i.e. less than 1%.

If we now have the same model predict all background pixels, we will get an accuracy of

(65536 - 600) / 65536 = 0.9908 or 99.1%

That's a very high accuracy for a very bad model. How to explain this discrepancy?

Well, as in other machine learning tasks, the problem here comes from the fact that the dataset is highly imbalanced. Let's compute this imbalance:

1. Background class 0: (65536 - 600) / 65536 = 0.9908
2. Object class 1: 600 / 65536 = 0.0092

What's a better metric for these types of unbalanced images?

Well, as for other imbalanced datasets, we can use similar concepts but slightly adapted for computer vision. Let's explore these.

Precision and Recall for Computer Vision

Without much surprise, we will use the two concepts of precision and recall slightly adapted to computer vision.

Before that, a quick reminder for those new to these two fundamental concepts.

Even before that, let's start with the concept of confusion matrix.

Confusion Matrix

To have a confusion matrix, we need some predictions and some ground truths. As said in the beginning, we will only focus on binary perdictions for now but notice that the concept extends easily to multiclasses.

Let's generate a confusion matrix using scikit-learn:

We can see that we have 4 quadrants:

• A: 0 vs B: 0
• A: 1 vs B: 0
• A: 0 vs B: 1
• A: 1 vs B: 1

If A are the true labels and B are the predicted labels and if 1 is the positive class (object of interest), then we get the following:

• A: 0 vs B: 0 => true negative tha will be shortened to TN
• A: 1 vs B: 0 => false negative that will be shortened to FP
• A: 0 vs B: 1 => false positive that will be shortened to FN
• A: 1 vs B: 1 => true positive that will be shortened to TP

We can place these labels in the confusion matrix.

Now that we have these 4 numbers (TP, FP, TN, FN) from the confusion matrix, we can build some useful metrics. Let's start with precision.

Precision

As its name indicates, precision is how precise the model is, i.e. it is a measure of the quality of the predictions we make.

To compute it, we need to have access to the true positives and the predicted positives (both true and false):

\(P := \frac{TP}{PP} = \frac{TP}{TP + FP}\)

For example, if we make one single pixel prediction and it is correct, the precision is 1. So one way to "game" this metric is to predict very few pixels from the object class, i.e. the less predictions one makes, the less errors.

Let's go back to the second image with few object pixels and compute the precision of the model:

In order to avoid the pitfalls of the precision, we will introduce another metric.

Recall

Now the problem with predicting only one pixel from the object class even if it is correct is that we are missing most of the object pixels, i.e. the recall is very low.

To compute it, we need to have access to the true positives and all the positive elements (both correct and uncorrectly predicted):

\(R := \frac{TP}{P} = \frac{TP}{TP + FN}\)

If precision can be compared to the quality of the model, recall is then akin to quantity, i.e. how many of the true labels have been found.

So, on the unbalanced image example, the more pixels we predict as the true class, the better will be the recall.

As you can see, there is a balance to be found between precision and recall.

\(F_{1}\) score

The \(F_{1}\) score is a "harmonious" balance between precision and recall (you will soon learn why I have used such an adjective).

As we have seen in the previous two sections, using only precision or recall isn't enough.

One idea is to use both the metrics and combine their strengths.

So the next immediate question that comes to mind is: how do we combine these two metrics?

One simple solution is to use the arithmetic mean. However, this mean suffers from extreme values.

For example, if the model has a precision of 1 and a recall of 0, the arithmetic mean will be 0.5.

Another better solution (will see why at the end), as outlined in this SO thread, is to use some measure theory.

Let's base our selection on the following six elements (which are the desired properties of a composite measure) adapted from the SO anwser:

1. Connectedness (two pairs can be ordered) and transitivity (i.e. if e1 >= e2 and e2 >= e3 then e1 >= e3)

2. Independence: two components contribute their effects independently to the effectiveness.

3. Thomsen condition: given that at a constant recall (precision respectivaly), we find a difference in effectiveness for two values of precision (recall)

then this difference cannot be removed or reversed by changing the constant value.

4. Restricted solvability.

5. Each component is essential: variation in one while leaving the other constant gives a variation in effectiveness.

6. Archimedean property for each component: tt merely ensures that the intervals on a component are comparable.

These desired properties give us an effectivness measure \(E\) as follows:

\(E = 1 - \left(\frac{\alpha}{P} + \frac{1-\alpha}{R}\right)^{-1}\)

where \(\alpha\) is between 0 and 1, \(P\) is the precision, and \(R\) is the recall.

In turn, this effectivness measure is linked to the \(F_\beta\) score by:

\(F_\beta = 1 - E\)

where \(\alpha = \frac{1}{1 + \beta^2}\)

Taking \(\beta = 1\), we get the F1 score. This gives: \(\alpha = \frac{1}{1 + 1} = \frac{1}{2}\)

\(F_{1} = \left(\frac{0.5}{P} + \frac{0.5}{R}\right)^{-1}\)

This is the formula of the harmonic mean (which is said to be the "correct" average when having ratios, which is the case for precision and recall) of \(P\) and \(R\).

Back to the first example of precision 1 and recall of 0: this time, the harmonic mean is 0. Much better!

Let's see why this formula works?

For that, we need to compute the derivative of the effectivness with regards to \(P\) and \(R\). Let's denote by \(D\) the denomniator.

Then:

\(\frac{\partial E}{\partial P} = -\frac{\alpha}{P^{2}D^{2}}\)

and

\(\frac{\partial E}{\partial R} = -\frac{(1-\alpha)}{R^{2}D^{2}}\)

Thus, to have a good effectivness, both derivatives should be equal. This happens exactly when:

\(\frac{R}{P} = \sqrt{\frac{1-\alpha}{\alpha}}\)

or in other words, using the forumla for \(\beta\), when \(R = \beta P\).

More technical details can be found here and here.

As a final thing about the \(F_{1}\) score, let's see how the formula becomes when using \(TP\), \(FP\), and \(FN\) (these come from the definition of R and P of course):

\(F_{1} := \left(\frac{0.5}{P} + \frac{0.5}{R}\right)^{-1} = \frac{2PR}{R + P}\)

\(F_{1} = \frac{2\frac{TP}{TP + FP}\frac{TP}{TP + FN}}{\frac{TP}{TP + FP} + \frac{TP}{TP + FN}}\)

After simplifications (one can factor a \(TP\) from both numerator and denominator for example), we get:

\(F_{1} = \frac{2 TP}{2 TP + FP + FN}\)

In the formula, we can see one limitation of the \(F_{1}\) score: true negatives aren't used directly. One often cited alternative to the \(F_{1}\) score is the Mathhews correlation coefficient. This metric takes into account the true negatives.

For more details, check this great blog post introducing the notion of \(F_{1}\) score.

Now that the \(F_{1}\) score has been explained, we can move again to segmentation metrics.

From F1 score to Dice score

Dice score, also known as Dice-Sørensen coefficient (it has many more names), is another name for the F1 score that you will find a lot more on the context of computer vision.

Here is how it is defined when applied to two sets of pixels \(A\) and \(B\), where \(A\) is the set of true pixels and \(B\) the set of predicted ones:

\(DSC(A, B) := \frac{2 |A \cap B|}{|A| + |B|}\)

Let's see how it is equal to the F1 score by computing it (we will use the simplifed version from above):

\(F_{1}(A, B) = \frac{2 TP(A, B) }{2 TP(A, B) + FP(A, B) + FN(A, B)}\)

As you can see, these are the same by noticing that:

\(TP(A, B) = |A \cap B|\)

that:

\(TP(A, B) + FP(A, B) = |B|\)

and that:

\(FN(A, B) + TP (A, B) = |A|\)

Alright, now that we have established the link between Dice and F1, let's move to other often encountered metrics: Jaccard and IoU.

What about Jaccard and IoU?

These are other metrics that come often in the context of Dice score and are closely related as we will see below.

So what are the differences?

Let's start by defining the Jaccard metric to see how it links to the Dice one:

\(J(A,B) := {{|A \cap B|}\over{|A \cup B|}} = {{|A \cap B|}\over{|A| + |B| - |A \cap B|}}\)

where A and B are the mask pixels: for example A is the ground truth ad B is the predicted mask.

Here is an illustration:

From: https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/

Jaccard metric, also known as the Jaccard index is another name for IoU metric, i.e. intersection over union which is definitely more explicit.

Now that these are defined, what is the link with the Dice metric?

As a reminder, here is how the Dice metric is defined:

\(DSC(A, B) = \frac{2 |A \cap B|}{|A| + |B|}\)

From the two formula (Jaccard vs Dice), we can establish that (the trick is to get \(\frac{|A| + |B|}{|A \cap B|}\) from the Jaccard coefficient):

\(J(A,B)^{-1} + 1 = 2DSC(A, B)^{-1}\)

then

\(DSC(A, B) = \frac{2 J(A, B)}{1 + J(A, B)}\)

Notice that if \(J\) is close to 0, \(D\) is around twice \(J\) (we will check this emperically on the next section).

Since one moves from one metric to another using an increasing metric, we can use one or the other.

For more details, check the following StackExchange thread.

Enough with theory, time for some application and what better than using the current competition's dataset.

Application: one simple model and metrics.

Alright, time to apply some of the new things we have explored.

For that, we will be using the fastai framework and train for few epochs and for only one fold.

Let's start by setting a simple semantic segmentation mdoel using fastai. To be more precise, I will be using SemTorch (more suitable for segmentation).

For the curious, here is a more complete notebook using the SemTorch library.

Notice also that some of the code is inspired from this original work by lafoss.

Also, the dataset is a resized one (to 256 x 256).

Finally, before we start, notice that the model here is just an illustration of the segmentation metrics and you should spend more time to improve it so don't be suprised to see low scores. If you seen any bug, please let me know in the comments. :)

Let's start by installing the missing libraries.

To check that our dataset pipeline is working, let's plot some of the images and masks as an overaly.

As for the loss, we will use a symmetric version of Lovasz (more on this loss in the final section):

As stated in the start of this section, we will only use one fold to make this notebook run faster and we will takee a batch size small enough so that the notebook's RAM is happpy.

Time to get the segmentation pipeline. For that, get_segmentation_learner from SemTorch comes in handy.

We can now run the lr_find method to get a good starting learning rate.

The model is learning but the loss is still quite high and the metrics aren't that good (maybe there is a bug?).

A second thing to notice is that the Dice metric is around twice the Jaccard metric. This is in concordance with what we have established therotically (the values are low so the approximation should hold) in the section above which is reassuring.

That's it with the main parts of this lengthy notebook. If you are curious, read the next sections with the following bonus sections:

1. instance segmentation
2. multiclass segmentation
3. metrics libraries
4. metrics vs losses
5. GIoU, DIoU, and CIoU

Bonus 1: instance segmentation

Instance segmentation is a variation of semantic segmentation where, instead of predicting "stuff", we predict "things". For example: cats, humans, etc.

This is a very "hand-wavy" way to introduce instance segmentation. A better definition could be found in the following paperswithcode link:

Instance segmentation is the task of detecting and delineating each distinct object of interest appearing in an image.

Let's check how the instance segmentation metrics work and how they are linked (or not) to the semantic segmentation ones?

mAP

As you have guesseed it from the title above, a metric that is often used for instance segmentation is mAP, short for mean average precision. Let's explore how this metric works.

Before we start, notice that we will only explore the definition used at Pascal VOC (and more precisely for the latest version 2012). Other very close definition exist, for example there is one for the COCO dataset.

First, we need to understand why there are two means: mean for the first m in the acronym and average for the A (in AP).

Let's start with the easiest bit: the mean m. It is taken over the different classes. In the binary case, there is no mean to take since we have only one class (the background doesn't count).

Now, the more interesting part: the average precision. For that, we will use the graph above.

For those familiar with machine learning, you might have recognized a similar graph: the ROC (short for receiver operating curve).

First things first, how do we get different metric values to plot a curve in the first place?

To start, we need to have a segmetnation score for each instance.

For example, we can use IoU as the segmentation metric (as seen in the previous sections) and we get a value between 0 and 1 per class. From there, we need to get a binary score: is the instance correctly classified or not?

One possible way to do it is to use a threshold: if the IoU is > threshold, it is 1 (the object belongs to the class), otherwise it is 0.

For example, if we have an IoU of 0.89 for a cat class and we set the threshold to 0.9, then we will predict that the mask isn't a cat despite having a high IoU and this will be a false negative. If the threshold was 0.8, the prediction would have been a true positive. Having a small threshold means less false negatives but a threshold too low means more false positives so there is a balance as usual.

We then do this processing for all the different instances of the different images, for one class, and one threshold.

Now, how do we compute precision and recall?

For that, we first need to rank all the predictions from largest IoU score to lowest (for a given threshold of course).

Let's suppose we have the following table of values for a threshold of 0.5 and an image containing 3 cats (it is a binary instance segmentation):

The TP column (true positive) is computed using the threshold. For example, the first row has an IoU of 0.7 > 0.5 so it is a true positive.

Since we have ranked the items from best to worst IoU, the precision and recall we will compute will be cumulative values as we go down the table. Let's add these two columns to our previous table:

Let's go through each row to understand how precision and recall are computed:

• First row: we only have one prediction so far, so precision is 1. We have found one of the three cats so far, so recall of 0.33.
• Second row: we have two predictions now and both are true positives, so a precision of 1.
• Third row: we now have three predictions and only two are correct, so a precision of 0.66. The recall is still at 0.66.

One remarkable thing about this computation is that precision can go down when we move down the table but the recall will always increase (since the number of true positives can only increase).

This computation gives us the blue curve and we understand better how the "zigzags" can emerge.

Next, we interpolate the blue curve to get the orange one: instead of keeping dropping precision values, we use the maximum precision to the right of the current recall when a dip happens.

From there, we can compute average precision that is defiend as being the area under the orange curve.

Notice that for previous versions of Pascal VOC, instead of directly computing the area, an 11 points sampling is done first.

If we go back to the cats example, we get a score of 0.33.

That's it for mAP!

The tricky part in this definition/computation, is that there are two levels of confusion matrix: one at the pixel level to first get the IoU score and then a second one at the instance level to get the curve.

One final thing to know is that for the COCO mAP, instead of using a single threshold of 0.5, it uses instead different thresholds and takes an additional average over these.

Bonus 2: multiclass segmentation

One other question to ask is: how to move from binary semantic segmentation to multiclass segmentation?

There are few things we need to change:

• the metric computation of course.
• the dataset processing, especially the mask.
• the choice of the loss

As for the metric computation, let's assume it is IoU. A simple extension is to compute this metric for each class then take the mean of everything, this gives the mIoU. Often, it is better to exclude the background class from the mIoU computation.

As for the loss, if the different classes are balanced, we can use categorical cross entropy. If the classes are unbalanced, we can use the focal loss for example.

Finally, for the mask, we can encode each class as an integer from 0 to n-1 where n is the number of classes. For example, if we have 3 classes, let's say background, car, and pedestrian, we can have the following mapping:

• 0 => background
• 1 => car
• 2 => pedestrian

The mask can look like this (we will reuse the same code from the first part but adapted):

Nice. That's enough with multiclass segmentation, let's move to metrics libraries.

Bonus 3: metrics libraries

In addition to the metrics we have shown in this notebook, there are of course many more. Instead of implementing everything from scratch, you can rely on existing libraries and avoid reinventing the wheel.

Here are some of the metrics libraries that can be used:

• pytorch: well, the easiest way is to implement it yourself from scratch.
• scikit-learn (of course)
• pytorch-toolbelt: has many useful metrics including segmentation ones.
• TorchMetrics: a new library that has been extracted from Pytorch lightning.

Bonus 4: metrics vs losses

There is one obvious way to define the loss from a metric that is:

loss = 1 - metric

Why does that work?

A metric is often better if it is higher and a loss is better the lower it is. Notice the use of often since some metrics are better when lower (for example MAE). If the metric is better when lower, you can define it as exactly the loss.

However this approach has a big limitation: the metric isn't necessary differentiable so won't be suitable as a loss.

Thus, a loss needs to be differentiable (well at least in the autograd sense of Pytorch).

To illustrate, here are some segmentation losses:

• Jaccard loss: this is linked to the Jaccard metric via the trick above (or some variant). Here is an implementation (from here):

def jaccard_distance_loss(y_true, y_pred, smooth=100):
    """
    Jaccard = (|X & Y|)/ (|X|+ |Y| - |X & Y|)
            = sum(|A*B|)/(sum(|A|)+sum(|B|)-sum(|A*B|))
    
    The jaccard distance loss is usefull for unbalanced datasets. This has been
    shifted so it converges on 0 and is smoothed to avoid exploding or disapearing
    gradient.
    
    Ref: https://en.wikipedia.org/wiki/Jaccard_index
    
    @url: https://gist.github.com/wassname/17cbfe0b68148d129a3ddaa227696496
    @author: wassname
    """
    intersection= (y_true * y_pred).abs().sum(dim=-1)
    sum_ = torch.sum(y_true.abs() + y_pred.abs(), dim=-1)
    jac = (intersection + smooth) / (sum_ - intersection + smooth)
    return (1 - jac) * smooth

• Binary cross entropy in the case of binary semantic segmentation when the classes are balanced (or categorical cross entropy for multiclasses).

Lovász-Softmax loss

Lovász-Softmax (Berman et al., CVPR 2018) is a surrogate for the discrete Jaccard loss, designed to optimize mean IoU in semantic segmentation. It uses soft class probabilities during training; taking an argmax first would remove the useful gradients.

For pixel \(i\) and class \(c\), let \(p_i(c)\) be the softmax probability and \(y_i\) the true class. The error for that class is:

Sort these errors from largest to smallest, then weight them by the successive changes in Jaccard loss when pixels in that order are marked incorrect. The weighted sum is the Lovász extension, written \(\overline{\Delta_{J_c}}\). Averaging across a set of classes \(\mathcal{C}\) gives:

The authors' implementation expects probabilities of shape [batch, classes, height, width] and integer labels of shape [batch, height, width]. Apply softmax to the logits first. By default, it averages only over classes present in the ground truth; ignored pixels and per-image versus per-batch averaging are explicit options. For binary segmentation, the companion Lovász hinge takes raw logits instead. Neither loss is itself an evaluation metric.

Focal loss

Focal loss

Lovász-Softmax targets mean IoU directly, while focal loss downweights easy examples. Both can be useful for imbalanced segmentation tasks.

Bonus 5: GIoU, DIoU, and CIoU

Why generalize IoU?

Two predicted boxes can both miss the target completely, even though one is much closer. Both receive an IoU of zero. For non-overlapping bounding boxes, the loss \(1 - IoU\) is flat, so its gradient gives no direction to improve their positions.

Generalized Intersection over Union (GIoU), introduced by Rezatofighi et al. at CVPR 2019, adds a penalty for unused space around the boxes.

Definition and intuition

Let \(A\) and \(B\) be two axis-aligned boxes, \(U = A \cup B\) their union, and \(C\) the smallest axis-aligned rectangle enclosing both boxes. Here \(|\cdot|\) denotes area, calculated from continuous coordinates without a pixel-counting +1.

The second term measures the fraction of the enclosing rectangle left empty by the union. A large gap lowers the score, even when the intersection is empty. The authors' interactive explanation illustrates this enclosing region.

For valid boxes with positive area:

• GIoU is at most IoU. Its upper bound is 1 for identical boxes; it approaches -1 as boxes become very far apart relative to their size.
• A negative score is valid: the enclosure penalty exceeds the overlap score.
• If the union fills \(C\), the penalty is zero and GIoU equals IoU. This includes one box contained in the other.
• The corresponding training loss is \(\mathcal{L}_{GIoU} = 1 - \mathrm{GIoU}\). It is zero for identical boxes and approaches 2 in the worst case. Its enclosure term can provide gradients while boxes are disjoint; min/max operations are handled with subgradients at their kinks.

A small example

Use coordinates (xmin, ymin, xmax, ymax) and fix the target at \(A=(0,0,2,2)\), with area 4. Compare two predictions of the same size:

IoU cannot distinguish the two misses. GIoU improves from about -0.333 to -0.200 as the prediction moves closer, and the loss falls from about 1.333 to 1.200.

Here is a scalar implementation to reproduce these scores. It rejects zero-area or reversed boxes rather than silently assigning them a score. This uses ordinary Python numbers for illustration; training requires tensor operations that preserve autograd.

def box_giou(a, b):
    """GIoU for two positive-area (xmin, ymin, xmax, ymax) boxes."""
    for box in (a, b):
        if not (box[0] < box[2] and box[1] < box[3]):
            raise ValueError("Each box must have positive width and height")

    def area(box):
        return (box[2] - box[0]) * (box[3] - box[1])

    intersection = (
        max(0, min(a[2], b[2]) - max(a[0], b[0]))
        * max(0, min(a[3], b[3]) - max(a[1], b[1]))
    )
    union = area(a) + area(b) - intersection
    enclosure = (
        (max(a[2], b[2]) - min(a[0], b[0]))
        * (max(a[3], b[3]) - min(a[1], b[1]))
    )
    return intersection / union - (enclosure - union) / enclosure


target = (0, 0, 2, 2)
for prediction in [(4, 0, 6, 2), (3, 0, 5, 2), target]:
    print(f"{box_giou(target, prediction):.3f}")
# -0.333
# -0.200
# 1.000

How do DIoU and CIoU differ?

GIoU's enclosure penalty vanishes for nested boxes, even when their centers differ. Zheng et al. (AAAI 2020) introduce two other penalties:

Here \(b_A,b_B\) are the centers and \(\rho\) is Euclidean distance. For box widths \(w_A,w_B\) and heights \(h_A,h_B\), the CIoU aspect-ratio term and its weight are:

At an exact match, the aspect-ratio penalty is defined as zero to avoid the \(0/0\) expression. Each variant gives a loss through \(1 - \mathrm{score}\).

Back to segmentation: these formulas concern bounding-box geometry, useful for the box-regression branch of an instance-segmentation model. Converting a mask into a box loses its boundaries, holes, and fine detail. For semantic masks, continue evaluating pixel-level IoU or Dice; box GIoU does not replace them.

Before you go

That's if for this this very long notebook.

I hope you have enjoyed reading about segmentation metrics and are now a bit less confused (at least I am after writing and searching for all these things).

See you around for another notebook!

Other resources

Some additional links to explore these topics further:

• Loss function library for Keras and PyTorch
• Segmentation in PyTorch using convenient tools
• AP and mAP for object detection
• Handling imbalanced datasets in machine learning

More on mean average precision (mAP):

• Why is mAP used for instance segmentation?
• Evaluating image segmentation models
• Understanding mean average precision
• Evaluation metrics for object detection and segmentation
• Calculating mAP for the Pascal VOC challenge
• mAP for object detection