llms-deepseek
![Brian Piercy](data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%230A66C2"><path d="M20.447 20.452H16.89v-5.569c0-1.328-.027-3.037-1.852-3.037-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85 3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433a1.982 1.982 0 1 1 0-3.964 1.982 1.982 0 0 1 0 3.964zm1.782 13.019H3.555V9h3.564v11.452zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0h.003z"/></svg>)
DSpark can make decoding faster, but acceptance quality still determines how much speed the system actually realizes.
DeepSeek V4 runs a million-token context at a quarter of V3.2's compute and a tenth of its KV cache. The trick is two new attention kernels it shipped, CSA and HCA. Here's how they work: At a million tokens, two things explode: the KV cache grows linearly with the sequence, and attention compute grows quadratically. DeepSeek's MLA (from V2) attacked this by compressing the head dimension, shrinking each token's KV footprint. DSA (V3.2) added a sparse indexer that reads only the top-k tokens per query. But both left a wall. MLA still stores every token. DSA's indexer still scans every token just to score it. At a million tokens the sequence length itself is the enemy. V4's move is to compress along the sequence dimension instead of the head dimension. CSA, Compressed Sparse Attention, does it in two steps. First it pools every 4 tokens into one KV entry with a learned, data-dependent weighting and overlapping windows, so block boundaries blur. That alone cuts the cache 4x. Then it runs DSA's lightning indexer over the compressed entries, keeps the top 1024, and adds a 128-token sliding window for local detail. The clever part is the compounding. CSA reuses DSA's indexer unchanged, but now runs it over a sequence that's already 4x shorter. The indexer's quadratic scan, the exact thing that bottlenecked V3.2 at long context, shrinks with the compression. Compress, then sparsify, and the two wins multiply. HCA, Heavily Compressed Attention, is the same operator with the knobs cranked: pool 128 tokens into one entry, no overlap, then attend densely over all of them. At a million tokens that's only about 8000 entries, so dense attention is cheap and you can drop the indexer entirely. The detail I love: HCA is literally CSA with the top-k budget set to keep everything. vLLM ships it as a sparse kernel with k=8192, which just means every entry. CSA finds the specific clause in the contract. HCA senses the topic shifted three pages ago. So V4 interleaves them through its 61 layers and lets the model learn which view each layer needs. Top-k selection isn't differentiable, so DeepSeek never backpropagates through it. The indexer is trained by a separate loss, detached from the main graph, that distills the full dense attention distribution into the indexer's scores. The model learns from the language loss, the indexer learns from the distillation loss, and they never share a gradient. And the indexer scores with a ReLU, not a softmax, because you only need a ranking, not a distribution. You never exponentiate, which is exactly what lets the whole indexer run in 4-bit. At a million tokens V4-Pro activates more parameters than V3.2 (49B vs 37B) and still runs attention at 27% of the cost. CSA's working set stays near 1100 entries no matter how long the context gets. MLA compressed the width of a token. DSA sparsified the reads. And V4 is the first to attack both axes at once!
Two Open-Weight Models. Eight Chip Families. One Frontier Co-Engineered for Non-Nvidia Silicon. The Stack Was the Moat. Now It Has a Fork.
🚀 DeepSeek-V4 Preview is officially live & open-sourced! Welcome to the era of cost-effective 1M context length.
The promotional discount runs until 5 May 2026. Even at full price, V4-Pro already undercuts GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro on per-token costs. The move is a direct challenge to the pricing strategy of US AI providers at a moment when the Trump administration has accused Chinese firms of distilling American AI […]
The company says its cost-efficient new V4 model is competitive with top closed-source models from OpenAI and Google DeepMind.
Chinese AI lab DeepSeek’s last model release was V3.2 (and V3.2 Speciale) last December. They just dropped the first of their hotly anticipated V4 series in the shape of two …
In January 2025, DeepSeek, a Chinese AI startup, launched R1, an AI model that rivaled top-tier LLMs from OpenAI and Anthropic. Built at a fraction of the
SemiAnalysis is hiring an analyst in New York City for Core Research, our world class research product for the finance industry. Please apply here It’s been a bit over 150 days since the launc…
DeepSeek is set to drop another model pretty soon, as details about their next DeepSeek R2 model have surfaced on the internet
Thanks to KiwiCo for sponsoring today’s video! Go to https://www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first monthly club crate or for...