A curriculum for learning about gpu performance engineering, from scratch to what the frontier AI labs do - wafer-ai/gpu-perf-engineering-resources
by Timothy Gao
Pytorch is OG
Cambridge Researchers Introduce SymTorch: A PyTorch Library that Translates Deep Learning Models into Human-Readable Equations
You've likely used ChatGPT, Gemini, or Grok, which demonstrate how large language models can exhibit human-like intelligence. While creating a clone of these large language models at home is unrealistic and unnecessary, understanding how they work helps demystify their…
Making developers awesome at machine learning.
A curated introduction to PyTorch that gets you up to speed in about an hour.
We’re excited to reveal our brand new PyTorch Landscape. The PyTorch Landscape helps researchers, developers, and organizations easily locate useful, curated, community-built tools that augment the PyTorch core framework.
PyTorch has emerged as a top choice for researchers and developers due to its relative ease of use and continuing improvement in performance.
In this article, we dive into how PyTorch’s Autograd engine performs automatic differentiation.
The Triton open-source programming language and compiler offers a high-level, python-based approach to create efficient GPU code. In this blog, we highlight the underlying details of how a triton program is compiled and the intermediate representations. For an introduction to…
The PyTorch community has continuously been at the forefront of advancing machine learning frameworks to meet the growing needs of researchers, data scientists, and AI engineers worldwide. With the latest PyTorch 2.5 release, the team aims to address several challenges faced by…
Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large language models and long-context applications. FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now…
These six tips will help you significantly accelerate your model training.
The full guide to creating custom datasets and dataloaders for different models in PyTorch
Built-in function vs. numerical methods
When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically...
Use 3D to visualize matrix multiplication expressions, attention heads with real weights, and more.
How to Identify and Analyze Performance Issues in the Backward Pass with PyTorch Profiler, PyTorch Hooks, and TensorBoard
Story at a Glance
Theory and PyTorch Implementation
This article provides a series of techniques that can lower memory consumption in PyTorch (when training vision transformers and LLMs) by approximately 20x without sacrificing modeling performance and prediction accuracy.
We are excited to announce the release of PyTorch® 2.0 which we highlighted during the PyTorch Conference on 12/2/22! PyTorch 2.0 offers the same eager-mode development and user experience, while fundamentally changing and supercharging how PyTorch operates at compiler level…
This blog post outlines techniques for improving the training performance of your PyTorch model without compromising its accuracy. To do so, we will wrap a P...
Data augmentation is a key tool in reducing overfitting, whether it's for images or text. This article compares three Auto Image Data Augmentation techniques...
Example debugging RoIAlign from Torchvision
Many developers who use Python for machine learning are now switching to PyTorch. Find out why and what the future could hold for TensorFlow.
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch - lucidrains/vit-pytorch
An end-to-end deep learning geospatial segmentation project using Pytorch and TorchGeo packages
The YOLO algorithm offers high detection speed and performance through its one-forward propagation capability. In this tutorial, we will focus on YOLOv5.
The Power of PyTorch/XLA and how Amazon SageMaker Training Compiler Simplifies its use
The fastai book, published as Jupyter Notebooks.
Welcome to the last entry into understanding the autograd engine of PyTorch series! If you haven’t read parts 1 & 2 check them now to understand how PyTorch creates the computational graph for the backward pass!
A detailed step-by-step explanation of how to build an image-captioning model in Pytorch
The PyTorch team recently announced TorchData, a prototype library focused on implementing composable and reusable data loading utilities for PyTorch. I hone...
A way to increase the amount of data and make the model more robust
We are excited to announce TorchRec, a PyTorch domain library for Recommendation Systems. This new library provides common sparsity and parallelism primitives, enabling researchers to build state-of-the-art personalization models and deploy them in production.
To understand the differences between automatic differentiation libraries, let’s talk about the engineering trade-offs that were made. I would personally say that none of these libraries are “better” than another, they simply all make engineering trade-offs based on the domains…
Should you use PyTorch vs TensorFlow in 2023? This guide walks through the major pros and cons of PyTorch vs TensorFlow, and how you can pick the right framework.
PyTorch Lightning has opened many new possibilities in deep learning and machine learning with a high level interface that makes it quicker to work with PyTorch.
Linear algebra is essential to deep learning and scientific computing, and it’s always been a core part of PyTorch. PyTorch 1.9 extends PyTorch’s support for linear algebra operations with the torch.linalg module. This module, documented here, has 26 operators, including faster…
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
From creating tensors to writing neural networks
Word on the street is that PyTorch lightning is a much better version of normal PyTorch. But what could it possibly have that it brought such consensus in our world? Well, it helps researchers scale…
This blog post is part of a mini-series that talks about the different aspects of building a PyTorch Deep Learning project using Variational Autoencoders. In Part 1, we looked at the variational…
A bug that plagues thousands of open-source ML projects.
A library for state-of-the-art self-supervised learning from images