llms-inference
![Brian Piercy](data:image/svg+xml;utf8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%230A66C2"><path d="M20.447 20.452H16.89v-5.569c0-1.328-.027-3.037-1.852-3.037-1.853 0-2.136 1.445-2.136 2.939v5.667H9.351V9h3.414v1.561h.046c.477-.9 1.637-1.85 3.37-1.85 3.601 0 4.267 2.37 4.267 5.455v6.286zM5.337 7.433a1.982 1.982 0 1 1 0-3.964 1.982 1.982 0 0 1 0 3.964zm1.782 13.019H3.555V9h3.564v11.452zM22.225 0H1.771C.792 0 0 .774 0 1.729v20.542C0 23.227.792 24 1.771 24h20.451C23.2 24 24 23.227 24 22.271V1.729C24 .774 23.2 0 22.222 0h.003z"/></svg>)
DSpark can make decoding faster, but acceptance quality still determines how much speed the system actually realizes.
OpenAI and Broadcom introduce Jalapeño, a custom AI chip built for LLM inference to improve performance, efficiency, and scale across AI systems.
The Practical Guide to LLM Inference on Consumer Hardware A practical walk through quantization, KV caches, Flash Attention, Mamba, MoE, and the runtimes that tie it all together — without the …
Maximum Likelihood Estimation is the quiet engine underneath almost everything in ML, DL, and RL. Most people learn it once in a stats class and move on. They shouldn't. It's not just a parameter-fitting trick — it's the philosophical foundation of why our models work at all. The core idea is almost trivial. You have data. You have a model with parameters θ. Ask: what value of θ makes the data you actually observed as probable as possible? Formally: maximize log p(data | θ). That's it. Everything else is an implementation detail. In classical ML, MLE gives you linear regression for free. Assume Gaussian noise on your outputs → maximizing likelihood gives you least squares. Assume Bernoulli outputs → you get logistic regression. The loss functions you memorized aren't arbitrary. They're MLE in disguise. In deep learning, cross-entropy loss is MLE. When you train a classifier, you're maximizing the log-likelihood of the true labels under your model's output distribution. Every gradient step is the model asking: how do I make the observed data more probable? Extend this to generative models: → VAEs maximise a lower bound on the log-likelihood (the ELBO) → Diffusion models maximise a reweighted likelihood over denoising steps → Autoregressive LLMs do pure MLE — next-token prediction is maximising p(xₜ | x₁,...,xₜ₋₁) over billions of tokens The entire pre-training recipe of GPT is MLE on text. In reinforcement learning, MLE shows up in a less obvious place. Policy gradient methods like REINFORCE compute: ∇θ log π(a|s) · R That score function — ∇θ log π — is the gradient of the log-likelihood of your policy's action. You're doing MLE, but weighted by reward. Actions that led to high reward get their likelihood pushed up. Actions that didn't, get pushed down. Even modern RLHF (the technique behind ChatGPT's fine-tuning) is built on this: fit a reward model via MLE, then optimise policy likelihood against that reward signal. The deeper point: MLE has a dual identity. Maximising log-likelihood is identical to minimising KL divergence between your model distribution and the true data distribution. MLE ≡ minimise KL(p_data ‖ p_model) This is why it generalises everywhere. You're always doing the same thing: pulling your model's distribution toward the data-generating process, as efficiently as information theory allows. The limitations are real — MLE can overfit, it treats all errors equally, it breaks under distribution shift. That's why we have regularisation (MAP estimation), robust losses, and distributional RL. Image Source: https://share.google/AbqFE6BjUlehRPhzx
Estimate the Average Treatment Effect in Python using DoWhy with regression and propensity score matching on real data.
Learn how to build and test causal inference models in Python using the DoWhy library.
why did NVIDIA cancel the Rubin CPX and replace it with a Groq LPU? LLM inference has two phases. prefill processes the whole prompt at once and is compute-bound. decode generates one token at a time and is memory-bound, reading the full model weights and KV cache for every token. since 2024 the standard approach has been splitting these onto separate GPU pools so prefill and decode stop interfering with each other. NVIDIA's first hardware answer was the Rubin CPX, announced September 2025. a dedicated prefill chip with 128 GB of GDDR7 memory, over 50% cheaper per GB than HBM, under 800W, more chips per rack. prefill is compute-bound, so save the expensive HBM for decode. the savings were real, but regular Rubin GPUs also handle prefill well enough. six months later at GTC 2026, NVIDIA shelved the CPX and partnered with Groq on a completely different split. Groq's split happens inside each layer. every transformer layer has an attention block that reads the KV cache and a feed-forward block that reads only its own weights. the KV cache is tens of gigabytes and is stored in HBM on the GPU. the feed-forward weights are smaller and read the same way every token. that second property is the reason SRAM works here. the Groq LP30 has 500 MB of on-chip SRAM at 150 TB/s, about 20x the memory bandwidth of a B200's HBM3e. run feed-forward on the LP30, keep attention on the GPU, and pass intermediate results between them at every layer. NVIDIA calls it Attention-FFN Disaggregation. the LPX rack packs 256 LP30 chips for 128 GB of SRAM and 40 PB/s aggregate bandwidth, with 640 TB/s of chip-to-chip interconnect for the per-layer transfer. NVIDIA claims 35x more inference throughput per megawatt than GB200 NVL72. so the short answer: the CPX saved cost on prefill, where regular GPUs were already fine. AFD delivers 20x bandwidth on decode by splitting inside the layer, where the real bottleneck sits. different axis, bigger win. wrote up the full details on the blog with diagrams for each approach, plus the AWS + Cerebras split where Trainium handles prefill and a wafer-scale CS-3.
Disaggregated inference started as a software technique for splitting prefill and decode onto separate GPU pools. NVIDIA, Groq, Cerebras, and AWS are now taking it further with chips purpose-built for each phase.
Researchers at Tsinghua University and Z.ai built IndexCache to eliminate redundant computation in sparse attention models like DeepSeek and GLM. The training-free technique cuts 75% of indexer operations, delivering up to 1.82x faster time-to-first-token at 200K context lengths and at least a 20% reduction in deployment costs for long-context workloads.
Taalas is replacing programmable GPUs with hardwired AI chips to achieve 17,000 tokens per second for ubiquitous inference
A new technical paper titled “Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling” was published by researchers at Uppsala University. Abstract “Energy consumption dictates the cost and environmental impact of deploying Large Language Models. This paper investigates the impact of on-chip SRAM size and operating frequency on the energy efficiency and performance of... » read more
What is Artificial Intelligence AI Inference? A Technical Deep Dive and Top 9 AI Inference Providers (2025 Edition)
Startup Groq has developed an machine learning processor that it claims blows GPUs away in large language model workloads – 10x faster than an Nvidia GPU at 10 percent of the cost, and needing a tenth of the electricity. Update: Groq model compilation time and time from access to getting it up and running clarified. […]
Faster than Nvidia? Dissecting the economics
Free books, lectures, blogs, papers, and more for a causal inference crash course
Since writing this post back in 2018, I have extended this to a 4-part series on causal inference: * ➡️️ Part 1: Intro to causal inference and do-calculus [https://www.inference.vc/untitled] * Part 2: Illustrating Interventions with a Toy Example [https://www.inference.vc/causal-inference-2-illustrating-interventions-in-a-toy-example/] * Part 3: Counterfactuals [https://www.inference.
Large Language Models (LLMs) have become a cornerstone in artificial intelligence, powering everything from chatbots and virtual assistants to advanced text generation and translation systems. Despite their prowess, one of the most pressing challenges associated with these models is the high cost of inference. This cost includes computational resources, time, energy consumption, and hardware wear. Optimizing these costs is paramount for businesses and researchers aiming to scale their AI operations without breaking the bank. Here are ten proven strategies to reduce LLM inference costs while maintaining performance and accuracy: Quantization Quantization is a technique that decreases the precision of model