Building a GPU Compiler from Scratch
Lowering PyTorch Graphs to CUDA via Multi-Level IRs
pytorch
wafer-ai/gpu-perf-engineering-resources: A curriculum for learning about gpu performance engineering, from scratch to what the frontier AI…
A curriculum for learning about gpu performance engineering, from scratch to what the frontier AI labs do - wafer-ai/gpu-perf-engineering-resources
DiffusionGemma Explained
by Timothy Gao
The Most Complete Guide to PyTorch for Data Scientists
Pytorch is OG
Meet SymTorch: A PyTorch Library that Translates Deep Learning Models into Human-Readable Equations - MarkTechPost
Cambridge Researchers Introduce SymTorch: A PyTorch Library that Translates Deep Learning Models into Human-Readable Equations
Building Transformer Models from Scratch with PyTorch (10-day Mini-Course)
You've likely used ChatGPT, Gemini, or Grok, which demonstrate how large language models can exhibit human-like intelligence. While creating a clone of these large language models at home is unrealistic and unnecessary, understanding how they work helps demystify their…
Machine Learning Mastery
Making developers awesome at machine learning.
PyTorch in One Hour: From Tensors to Training Neural Networks on Multiple GPUs
A curated introduction to PyTorch that gets you up to speed in about an hour.
Meta Introduces KernelLLM: An 8B LLM that Translates PyTorch Modules into Efficient Triton GPU Kernels
Introducing the New PyTorch Landscape: Your Guide to the PyTorch Ecosystem
We’re excited to reveal our brand new PyTorch Landscape. The PyTorch Landscape helps researchers, developers, and organizations easily locate useful, curated, community-built tools that augment the PyTorch core framework.
Why PyTorch Gets All the Love
PyTorch has emerged as a top choice for researchers and developers due to its relative ease of use and continuing improvement in performance.
PyTorch 101, Understanding Graphs, Automatic Differentiation and Autograd
In this article, we dive into how PyTorch’s Autograd engine performs automatic differentiation.
Triton Kernel Compilation Stages
The Triton open-source programming language and compiler offers a high-level, python-based approach to create efficient GPU code. In this blog, we highlight the underlying details of how a triton program is compiled and the intermediate representations. For an introduction to…
PyTorch 2.5 Released: Advancing Machine Learning Efficiency and Scalability
The PyTorch community has continuously been at the forefront of advancing machine learning frameworks to meet the growing needs of researchers, data scientists, and AI engineers worldwide. With the latest PyTorch 2.5 release, the team aims to address several challenges faced by…
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-preci
Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large language models and long-context applications. FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now…
This Is How To Optimize PyTorch for Faster Model Training
These six tips will help you significantly accelerate your model training.
Comprehensive Guide to Datasets and Dataloaders in PyTorch
The full guide to creating custom datasets and dataloaders for different models in PyTorch
Understanding Principal Component Analysis in PyTorch
Built-in function vs. numerical methods
How AMD May Get Across the CUDA Moat
When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically...
Inside the Matrix: Visualizing Matrix Multiplication, Attention and Beyond
Use 3D to visualize matrix multiplication expressions, attention heads with real weights, and more.
PyTorch Model Performance Analysis and Optimization — Part 6
How to Identify and Analyze Performance Issues in the Backward Pass with PyTorch Profiler, PyTorch Hooks, and TensorBoard
Accelerated CPU Inference with PyTorch Inductor using torch.compile
Story at a Glance
Variational Autoencoder (VAE) with Discrete Distribution using Gumbel Softm
Theory and PyTorch Implementation
Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorc
This article provides a series of techniques that can lower memory consumption in PyTorch (when training vision transformers and LLMs) by approximately 20x without sacrificing modeling performance and prediction accuracy.
PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever
We are excited to announce the release of PyTorch® 2.0 which we highlighted during the PyTorch Conference on 12/2/22! PyTorch 2.0 offers the same eager-mode development and user experience, while fundamentally changing and supercharging how PyTorch operates at compiler level…
Some Techniques To Make Your PyTorch Models Train Faster
This blog post outlines techniques for improving the training performance of your PyTorch model without compromising its accuracy. To do so, we will wrap a P...
Comparing Different Automatic Image Augmentation Methods in PyTorch
Data augmentation is a key tool in reducing overfitting, whether it's for images or text. This article compares three Auto Image Data Augmentation techniques...
Torch and Torchvision C installation and debugging on Linux
Example debugging RoIAlign from Torchvision
Why TensorFlow for Python is dying a slow death
Many developers who use Python for machine learning are now switching to PyTorch. Find out why and what the future could hold for TensorFlow.
lucidrains/vit-pytorch: Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single…
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch - lucidrains/vit-pytorch
Artificial Intelligence for Geospatial Analysis with Pytorch’s TorchGeo (pa
An end-to-end deep learning geospatial segmentation project using Pytorch and TorchGeo packages
YOLOv5 PyTorch Tutorial
The YOLO algorithm offers high detection speed and performance through its one-forward propagation capability. In this tutorial, we will focus on YOLOv5.
How to Accelerate your PyTorch GPU Training with XLA
The Power of PyTorch/XLA and how Amazon SageMaker Training Compiler Simplifies its use
fastai/fastbook: The fastai book, published as Jupyter Notebooks
The fastai book, published as Jupyter Notebooks.
How Computational Graphs are Executed in PyTorch
Welcome to the last entry into understanding the autograd engine of PyTorch series! If you haven’t read parts 1 & 2 check them now to understand how PyTorch creates the computational graph for the backward pass!
How to Build an Image-Captioning Model in Pytorch
A detailed step-by-step explanation of how to build an image-captioning model in Pytorch
Taking Datasets, DataLoaders, and PyTorch’s New DataPipes for a Spin
The PyTorch team recently announced TorchData, a prototype library focused on implementing composable and reusable data loading utilities for PyTorch. I hone...
A Comprehensive Guide to Image Augmentation using Pytorch
A way to increase the amount of data and make the model more robust
Introducing TorchRec, a library for modern production recommendation system
We are excited to announce TorchRec, a PyTorch domain library for Recommendation Systems. This new library provides common sparsity and parallelism primitives, enabling researchers to build state-of-the-art personalization models and deploy them in production.
Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 2.4.0+cu121 documentation
Engineering Trade-Offs in Automatic Differentiation: from TensorFlow and Py
To understand the differences between automatic differentiation libraries, let’s talk about the engineering trade-offs that were made. I would personally say that none of these libraries are “better” than another, they simply all make engineering trade-offs based on the domains…
PyTorch vs TensorFlow in 2023
Should you use PyTorch vs TensorFlow in 2023? This guide walks through the major pros and cons of PyTorch vs TensorFlow, and how you can pick the right framework.
An Introduction to PyTorch Lightning
PyTorch Lightning has opened many new possibilities in deep learning and machine learning with a high level interface that makes it quicker to work with PyTorch.
The torch.linalg module: Accelerated Linear Algebra with Autograd in PyTorch
Linear algebra is essential to deep learning and scientific computing, and it’s always been a core part of PyTorch. PyTorch 1.9 extends PyTorch’s support for linear algebra operations with the torch.linalg module. This module, documented here, has 26 operators, including faster…
Introduction - Hugging Face NLP Course
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
29 Pytorch Snippets to Speed Up Your Machine Learning Cycle
From creating tensors to writing neural networks
Pytorchvideo a deep learning library for video understanding
An Introduction to PyTorch Lightning
Word on the street is that PyTorch lightning is a much better version of normal PyTorch. But what could it possibly have that it brought such consensus in our world? Well, it helps researchers scale…
Beginner guide to Variational Autoencoders (VAE) with PyTorch Lightning (Part 2)
This blog post is part of a mini-series that talks about the different aspects of building a PyTorch Deep Learning project using Variational Autoencoders. In Part 1, we looked at the variational…
Using PyTorch + NumPy? You're making a mistake.
A bug that plagues thousands of open-source ML projects.