llms-attention
![Brian Piercy](data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%230A66C2"><path d="M20.447 20.452H16.89v-5.569c0-1.328-.027-3.037-1.852-3.037-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85 3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433a1.982 1.982 0 1 1 0-3.964 1.982 1.982 0 0 1 0 3.964zm1.782 13.019H3.555V9h3.564v11.452zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0h.003z"/></svg>)
DeepSeek V4 runs a million-token context at a quarter of V3.2's compute and a tenth of its KV cache. The trick is two new attention kernels it shipped, CSA and HCA. Here's how they work: At a million tokens, two things explode: the KV cache grows linearly with the sequence, and attention compute grows quadratically. DeepSeek's MLA (from V2) attacked this by compressing the head dimension, shrinking each token's KV footprint. DSA (V3.2) added a sparse indexer that reads only the top-k tokens per query. But both left a wall. MLA still stores every token. DSA's indexer still scans every token just to score it. At a million tokens the sequence length itself is the enemy. V4's move is to compress along the sequence dimension instead of the head dimension. CSA, Compressed Sparse Attention, does it in two steps. First it pools every 4 tokens into one KV entry with a learned, data-dependent weighting and overlapping windows, so block boundaries blur. That alone cuts the cache 4x. Then it runs DSA's lightning indexer over the compressed entries, keeps the top 1024, and adds a 128-token sliding window for local detail. The clever part is the compounding. CSA reuses DSA's indexer unchanged, but now runs it over a sequence that's already 4x shorter. The indexer's quadratic scan, the exact thing that bottlenecked V3.2 at long context, shrinks with the compression. Compress, then sparsify, and the two wins multiply. HCA, Heavily Compressed Attention, is the same operator with the knobs cranked: pool 128 tokens into one entry, no overlap, then attend densely over all of them. At a million tokens that's only about 8000 entries, so dense attention is cheap and you can drop the indexer entirely. The detail I love: HCA is literally CSA with the top-k budget set to keep everything. vLLM ships it as a sparse kernel with k=8192, which just means every entry. CSA finds the specific clause in the contract. HCA senses the topic shifted three pages ago. So V4 interleaves them through its 61 layers and lets the model learn which view each layer needs. Top-k selection isn't differentiable, so DeepSeek never backpropagates through it. The indexer is trained by a separate loss, detached from the main graph, that distills the full dense attention distribution into the indexer's scores. The model learns from the language loss, the indexer learns from the distillation loss, and they never share a gradient. And the indexer scores with a ReLU, not a softmax, because you only need a ranking, not a distribution. You never exponentiate, which is exactly what lets the whole indexer run in 4-bit. At a million tokens V4-Pro activates more parameters than V3.2 (49B vs 37B) and still runs attention at 27% of the cost. CSA's working set stays near 1100 entries no matter how long the context gets. MLA compressed the width of a token. DSA sparsified the reads. And V4 is the first to attack both axes at once!
From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs
AMD's 3D V-Cache CPUs deliver a huge boost versus the Non-X3D part in AI benchmarks, showcasing why they are best suited for RAG pipelines.
Researchers at Tsinghua University and Z.ai built IndexCache to eliminate redundant computation in sparse attention models like DeepSeek and GLM. The training-free technique cuts 75% of indexer operations, delivering up to 1.82x faster time-to-first-token at 200K context lengths and at least a 20% reduction in deployment costs for long-context workloads.
Google published a research blog post on Tuesday about a new compression algorithm for AI models. Within hours, memory stocks were falling. Micron dropped 3 per cent, Western Digital ...
From MHA and GQA to MLA, sparse attention, and hybrid architectures
A Blog post by Ksenia Se on Hugging Face
How a Key-Value (KV) cache reduces Transformer inference time by trading memory for computation
Compute costs scale with the square of the input size. That’s not great.
A deep dive into absolute, relative, and rotary positional embeddings with code examples
Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large language models and long-context applications. FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now used by most libraries to accelerate Transformer training and inference. This has contributed to a massive increase in LLM context length in the last two years, from 2-4K (GPT-3, OPT) to 128K (GPT-4), or even 1M (Llama 3). However, despite its success, FlashAttention has yet to take advantage of new capabilities in modern hardware, with FlashAttention-2 achieving only 35% utilization of theoretical max FLOPs on the H100 GPU. In this blogpost, we describe three main techniques to speed up attention on Hopper GPUs: exploiting asynchrony of the Tensor Cores and TMA to (1) overlap overall computation and data movement via warp-specialization and (2) interleave block-wise matmul and softmax operations, and (3) incoherent processing that leverages hardware support for FP8 low-precision.
Deep learning architectures have revolutionized the field of artificial intelligence, offering innovative solutions for complex problems across various domains, including computer vision, natural language processing, speech recognition, and generative models. This article explores some of the most influential deep learning architectures: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Transformers, and Encoder-Decoder architectures, highlighting their unique features, applications, and how they compare against each other. Convolutional Neural Networks (CNNs) CNNs are specialized deep neural networks for processing data with a grid-like topology, such as images. A CNN automatically detects the important features without any human supervision.
Large language models do better at solving problems when they show their work. Researchers are beginning to understand why.
The Math and the Code Behind Attention Layers in Computer Vision
We will deep dive into understanding how transformer model work like BERT(Non-mathematical Explanation of course!). system design to use the transformer to build a Sentiment Analysis
When we talk about using different ways to share information, it's like picking the one that fits what you need! Words, pictures, and mixes of both have
Large Language Models (LLMs) have gained significant prominence in modern machine learning, largely due to the attention mechanism. This mechanism employs a sequence-to-sequence mapping to construct context-aware token representations. Traditionally, attention relies on the softmax function (SoftmaxAttn) to generate token representations as data-dependent convex combinations of values. However, despite its widespread adoption and effectiveness, SoftmaxAttn faces several challenges. One key issue is the tendency of the softmax function to concentrate attention on a limited number of features, potentially overlooking other informative aspects of the input data. Also, the application of SoftmaxAttn necessitates a row-wise reduction along the input sequence length,