Variational Autoencoders for Anomaly Detection: A Gentle Introduction

Variational Autoencoders (VAEs) have become one of my favorite tools for unsupervised anomaly detection. In this post I want to give an intuitive walkthrough of how they work and why the math matters — with proper equations along the way.

The core idea¶

A VAE learns a probabilistic mapping between a data space \(\mathcal{X}\) and a latent space \(\mathcal{Z}\). Given an input \(x\), the encoder produces a distribution \(q_\phi(z \mid x)\) over latent codes, and the decoder maps a latent code \(z\) back to a distribution \(p_\theta(x \mid z)\) over reconstructions.

When trained on normal data only, the VAE learns to reconstruct healthy inputs well. At test time, anomalous regions produce large reconstruction errors — and that's our anomaly signal.

The ELBO¶

The objective is the Evidence Lower Bound (ELBO):

The first term encourages good reconstruction; the second keeps the learned posterior close to a prior \(p(z)\), typically \(\mathcal{N}(0, I)\).

The reparameterization trick¶

To backpropagate through the sampling step \(z \sim q_\phi(z \mid x)\), we use the reparameterization trick:

This turns a stochastic node into a deterministic function of \(\phi\) and an auxiliary noise variable \(\epsilon\), making gradients well-defined.

Anomaly scoring¶

The simplest anomaly score is the per-pixel reconstruction error:

But in my work I showed that gradients of the loss w.r.t. the input are even more informative¹:

This "score-based" localization highlights which pixels the model is struggling with — giving us spatially precise anomaly maps without pixel-level supervision.

A minimal PyTorch snippet¶

import torch
import torch.nn as nn

class VAE(nn.Module):
    def __init__(self, dim_in=784, dim_hidden=128, dim_latent=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(dim_in, dim_hidden),
            nn.ReLU(),
        )
        self.fc_mu = nn.Linear(dim_hidden, dim_latent)
        self.fc_logvar = nn.Linear(dim_hidden, dim_latent)
        self.decoder = nn.Sequential(
            nn.Linear(dim_latent, dim_hidden),
            nn.ReLU(),
            nn.Linear(dim_hidden, dim_in),
            nn.Sigmoid(),
        )

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def forward(self, x):
        h = self.encoder(x)
        mu, logvar = self.fc_mu(h), self.fc_logvar(h)
        z = self.reparameterize(mu, logvar)
        return self.decoder(z), mu, logvar

The KL divergence for a Gaussian posterior has a nice closed form:

Context-encoding extension¶

In my Context-encoding VAE (ceVAE), we mask parts of the input and force the model to infer them from context. The anomaly score becomes:

where \(x_{\setminus k}\) denotes the input with patch \(k\) removed. This forces the model to learn semantic relationships rather than copying.

Why this matters¶

The key insight across all of these approaches: reconstruction alone isn't enough. You need the model to learn meaningful structure so that anomalies violate semantic expectations, not just pixel statistics. That's what gradients and context-encoding give you.

::: tip Want to try this yourself? Check out batchgenerators for data augmentation pipelines that pair well with VAE training on medical images. :::