Researchers propose attention-free AI architecture using Grassmann flows
A 2025 arXiv paper challenges the necessity of self-attention in sequence modeling, proposing an alternative architecture based on Grassmann flows. The authors argue standard attention is mathematically opaque and introduce a causal layer that processes local token pairs via geometric subspaces. Early results show these attention-free models achieve perplexities within 10–15% of comparable Transformers on Wikitext-2, suggesting a viable, interpretable path for future neural network design.
Key points
- Researchers published a 2025 arXiv paper arguing that explicit self-attention is not strictly necessary for strong performance in sequence modeling tasks.
- The paper introduces a causal Grassmann layer that encodes local token pairs as two-dimensional subspaces on a Grassmann manifold using Plucker coordinates, avoiding standard attention matrices.
- On the Wikitext-2 benchmark, attention-free models with 13 to 18 million parameters achieved validation perplexities within 10 to 15 percent of size-matched Transformer models.
- The authors contend that standard multi-head attention acts as a mathematically opaque tensor lifting process, whereas their method keeps core computations on a finite-dimensional manifold.
- The proposed architecture aims to improve interpretability by constraining information propagation through controlled deformations of low-rank subspaces over multi-scale local windows.
- Preliminary results on the SNLI natural language inference task indicate that a Grassmann-Plucker head offers a competitive alternative to traditional attention mechanisms.
A new study published on arXiv in 2025 challenges the foundational role of self-attention in modern artificial intelligence. The authors argue that standard multi-head attention, while effective, functions as a form of tensor lifting that maps hidden vectors into high-dimensional spaces of pairwise interactions. This mechanism, they contend, is mathematically opaque, making it difficult to describe model behavior through explicit invariants after multiple layers of processing.
To address this, the researchers propose an attention-free architecture based on Grassmann flows. Instead of constructing large attention matrices, their Causal Grassmann layer linearly reduces token states and encodes local token pairs as two-dimensional subspaces on a Grassmann manifold. These geometric features are then fused back into hidden states through gated mixing. This approach ensures that information propagates via controlled deformations of low-rank subspaces, keeping the core computation within a finite-dimensional manifold rather than an unstructured tensor space.
Preliminary benchmarks demonstrate the viability of this alternative. On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieved validation perplexities within approximately 10 to 15 percent of size-matched Transformers. Additionally, the paper reports results on the SNLI natural language inference task, where a Grassmann-Plucker head was evaluated. These findings suggest that attention-free architectures can deliver competitive performance while potentially offering greater mathematical transparency and interpretability for future sequence modeling applications.
Sources
The WireByte editorial team synthesises technology news from multiple primary sources, verifies the facts, and links every source. Articles are produced with AI assistance and reviewed under our editorial policy.
