Barlow Twins
Barlow Twins is a self-supervised visual representation method that trains two shared-weight encoders on distorted views of the same image. Its objective makes corresponding embedding dimensions agree while reducing redundancy between different dimensions.
Definition
Barlow Twins is a self-supervised learning method introduced by Zbontar, Jing, Misra, LeCun, and Deny in 2021. It belongs to the family of joint-embedding methods: two augmented views of the same sample are encoded, then the training objective shapes the relationship between their embeddings.
The name refers to neuroscientist H. Barlow's redundancy-reduction principle: useful sensory representations should reduce redundant coding.
Objective
The method computes a cross-correlation matrix between the embedding dimensions produced by the two network branches. It pushes the matrix toward the identity matrix.
Diagonal terms should be close to one, meaning corresponding dimensions from the two views agree. Off-diagonal terms should be close to zero, meaning different dimensions are not redundant copies of one another.
Collapse Avoidance
Many joint-embedding objectives risk representation collapse, where the encoder outputs the same vector for every input. Barlow Twins avoids collapse by combining invariance across views with redundancy reduction across dimensions.
Unlike many contrastive methods, it does not require explicit negative examples. That made it part of a broader shift toward non-contrastive self-supervised representation learning.
Why It Matters
Barlow Twins helped clarify how self-supervised systems might learn useful visual representations without labels and without negative samples. It also influenced the conceptual path toward VICReg and JEPA-style representation learning.
In the JEPA/world-model lineage, Barlow Twins matters because it addresses a foundational problem: how to make representations informative rather than collapsed, redundant, or merely reconstructive.
Related Pages
- Siamese Networks
- Contrastive Learning
- VICReg
- JEPA and World Models
- Yann LeCun
- DINO Self-Supervised Vision
- BYOL
- CLIP
- Embeddings and Vector Representations
- Active Learning
Sources
- Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny, "Barlow Twins: Self-Supervised Learning via Redundancy Reduction", arXiv, 2021.
- H. B. Barlow, "Possible Principles Underlying the Transformations of Sensory Messages", 1961.