VICReg
VICReg, short for Variance-Invariance-Covariance Regularization, is a non-contrastive self-supervised learning method that learns representations by balancing three pressures: make paired views agree, preserve variation across a batch, and reduce redundancy across embedding dimensions.
Definition
VICReg is a self-supervised objective introduced by Bardes, Ponce, and LeCun. It trains encoders on different augmented views of the same input without requiring explicit negative examples.
Its design directly addresses representation collapse by requiring embeddings to keep nontrivial variance while still becoming invariant to chosen augmentations.
The Three Terms
Variance. Each embedding dimension should vary enough across the batch. This prevents the encoder from outputting constant vectors.
Invariance. Embeddings from different views of the same sample should be close. This teaches the model which transformations should not change the representation.
Covariance. Different embedding dimensions should not become redundant. The covariance penalty discourages dimensions from carrying the same information.
Why It Matters
VICReg made collapse avoidance explicit and interpretable. Instead of depending on negative examples, stop-gradient tricks, or architectural asymmetries, it stated the pressures directly: keep information, align views, reduce redundancy.
That makes it important in the path from contrastive learning toward JEPA-style systems. It shows one route to stable representation learning without reconstructing pixels and without requiring a large negative set.
Governance Questions
VICReg itself is a training objective, not a deployed product. The governance question is what downstream systems do with the representations it helps produce.
Operators should ask which augmentations define "same," what information is discarded, whether the embedding space is stable across populations or domains, and whether downstream thresholds create hidden classification rules.
Related Pages
- Barlow Twins
- Contrastive Learning
- Siamese Networks
- JEPA and World Models
- DINO Self-Supervised Vision
- BYOL
- CLIP
- Embeddings and Vector Representations
- Active Learning
Sources
- Adrien Bardes, Jean Ponce, and Yann LeCun, "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning", arXiv, 2021.
- Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, and Yann LeCun, "An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization", arXiv, 2023.