3 Ways to Speed Up Model Training Without More GPUs

In this article, you will learn three proven ways to speed up model training by optimizing precision, memory, and data flow — without adding any new GPUs.

Topics we will cover include:

• How mixed precision and memory techniques boost throughput safely
• Using gradient accumulation to train with larger “virtual” batches
• Sharding and offloading with ZeRO to fit bigger models on existing hardware

Let’s not waste any more time.

3 Ways to Speed Up Model Training Without More GPUs
Image by Editor
 

Introduction

Training large models can be painfully slow, and the first instinct is often to ask for more GPUs. But extra hardware isn’t always an option. There are issues that stand in the way, such as budgets and cloud limits. The good news is that there are ways to make training significantly faster without adding a single GPU.

Speeding up training isn’t only about raw compute power; it’s about using what you already have more efficiently. A significant amount of time is wasted on memory swaps, idle GPUs, and unoptimized data pipelines. By improving how your code and hardware communicate, you can cut hours or even days from training runs.

Method 1: Mixed Precision and Memory Optimizations

One of the easiest ways to speed up training without new GPUs is to use mixed precision. Modern GPUs are designed to handle half-precision (FP16) or bfloat16 math much faster than standard 32-bit floats. By storing and computing in smaller data types, you reduce memory use and bandwidth, allowing more data to fit on the GPU at once, which means that the operations complete faster.

The core idea is simple:

• Use lower precision (FP16 or BF16) for most operations
• Keep critical parts (like loss scaling and a few accumulations) in full precision (FP32) to maintain stability

When done correctly, mixed precision often delivers 1.5 – 2 times faster training with little to no drop in accuracy. It’s supported natively in PyTorch, TensorFlow, and JAX, and most NVIDIA, AMD, and Apple GPUs now have hardware acceleration for it.

Here’s a PyTorch example that enables automatic mixed precision:

Why this works:

• autocast() automatically chooses FP16 or FP32 per operation
• GradScaler() prevents underflow by dynamically adjusting the loss scale
• The GPU executes faster because it moves and computes fewer bytes per operation

You can also activate it globally with PyTorch’s Automatic Mixed Precision (AMP) or Apex library for legacy setups. For newer devices (A100, H100, RTX 40 series), bfloat16 (BF16) is often more stable than FP16.
Memory optimizations go hand-in-hand with mixed precision. Two common tricks are:

• Gradient checkpointing: save only key activations and recompute others during backpropagation, trading compute for memory
• Activation offloading: temporarily move rarely used tensors to CPU memory

These can be enabled in PyTorch with:

or configured automatically using DeepSpeed, Hugging Face Accelerate, or bitsandbytes.

When to use it:

• If your model fits tightly on GPU memory, or your batch size is small
• You’re using a recent GPU (RTX 20-series or newer)
• You can tolerate minor numeric variation during training

It is typically expected to gain 30–100% faster training and up to 50% less memory use, depending on model size and hardware.

Method 2: Gradient Accumulation and Effective Batch Size Tricks

Sometimes the biggest barrier to faster training isn’t compute, it’s GPU memory. You might want to train with large batches to improve gradient stability, but your GPU runs out of memory long before you reach that size.

Gradient accumulation solves this neatly. Instead of processing one massive batch at once, you split it into smaller micro-batches. You run forward and backward passes for each micro-batch, accumulate the gradients, and only update the model weights after several iterations. This lets you simulate large-batch training using the same hardware.

Here’s what that looks like in PyTorch:

How it works:

• The loss is divided by the number of accumulation steps to maintain balanced gradients
• Gradients are stored in memory between steps, rather than being cleared
• After accum_steps mini-batches, the optimizer performs a single update

This simple change allows you to use a virtual batch size up to four or eight times larger, improving stability and potentially convergence speed, without exceeding GPU memory.

Why it matters:

• Larger effective batches reduce noise in gradient updates, improving convergence for complex models
• You can combine this with mixed precision for additional gains
• It’s especially effective when memory, not compute, is your limiting factor

When to use it:

• You hit “out of memory” errors with large batches
• You want the benefits of larger batches without changing hardware
• Your data loader or augmentation pipeline can keep up with multiple mini-steps per update

Method 3: Smart Offloading and Sharded Training (ZeRO)

As models grow, GPU memory becomes the main bottleneck long before compute does. You might have the raw power to train a model, but not enough memory to hold all its parameters, gradients, and optimizer states at once. That’s where smart offloading and sharded training come in.

The idea is to split and distribute memory use intelligently, rather than replicating everything on each GPU. Frameworks like DeepSpeed and Hugging Face Accelerate implement this through techniques such as ZeRO (Zero Redundancy Optimizer).

How ZeRO Works

Normally, every GPU in a multi-GPU setup holds a full copy of: Model parameters, Gradients, and Optimizer states. That’s incredibly wasteful, especially for large models. ZeRO breaks this duplication by sharding those states across devices:

• ZeRO Stage 1: shards optimizer states
• ZeRO Stage 2: shards optimizer states and gradients
• ZeRO Stage 3: shards everything, including model parameters

Each GPU now holds only a fraction of the total memory footprint, but they still cooperate to compute full updates. This enables models that are significantly larger than the memory capacity of a single GPU to train efficiently.

Simple Example (DeepSpeed)

Below is a basic DeepSpeed configuration snippet that enables ZeRO optimization:

Then in your script:

What it does:

• Enables mixed precision (fp16) for faster compute
• Activates ZeRO Stage 2, sharding optimizer states and gradients across devices
• Offloads unused tensors to CPU memory when GPU memory is tight

When to Use It

• You’re training a large model (hundreds of millions or billions of parameters)
• You run out of GPU memory even with mixed precision
• You’re using multiple GPUs or distributed nodes

Bonus Tips

The three main methods above—mixed precision, gradient accumulation, and ZeRO offloading—deliver most of the performance gains you can achieve without adding hardware. But there are smaller, often overlooked optimizations that can make a noticeable difference, especially when combined with the main ones.

Let’s look at a few that work in nearly every training setup.

1. Optimize Your Data Pipeline

GPU utilization often drops because the model finishes computing before the next batch is ready to be processed. The fix is to parallelize and prefetch your data.

In PyTorch, you can boost data throughput by adjusting the DataLoader:

• num_workers uses multiple CPU threads for loading
• pin_memory=True speeds up host-to-GPU transfers
• prefetch_factor ensures batches are ready before the GPU asks for them

If you’re working with large datasets, store them in formats optimized for sequential reads like WebDataset, TFRecord, or Parquet instead of plain images or text files.

2. Profile Before You Optimize

Before applying advanced techniques, find out where your training loop actually spends time. Frameworks provide built-in profilers:

You’ll often discover that your biggest bottleneck isn’t the GPU, but something like data augmentation, logging, or a slow loss computation. Fixing that yields instant speedups without any algorithmic change.

3. Use Early Stopping and Curriculum Learning

Not all samples contribute equally throughout training. Early stopping prevents unnecessary epochs once performance plateaus. Curriculum learning starts training with simpler examples, then introduces harder ones, helping models converge faster.

This small pattern can save hours of training on large datasets with minimal impact on accuracy.

4. Monitor Memory and Utilization Regularly

Knowing how much memory your model actually uses helps you balance batch size, accumulation, and offloading. In PyTorch, you can log GPU memory statistics with:

Monitoring utilities like nvidia-smi, GPUtil, or Weights & Biases system metrics help catch underutilized GPUs early.

5. Combine Techniques Intelligently

The biggest wins come from stacking these strategies:

• Mixed precision + gradient accumulation = faster and more stable training
• ZeRO offloading + data pipeline optimization = larger models without memory errors
• Early stopping + profiling = fewer wasted epochs

When to Use Each Method

To make it easier to decide which approach fits your setup, here’s a summary table comparing the three main techniques covered so far, along with their expected benefits, best-fit scenarios, and trade-offs.

Method	Best For	How It Helps	Typical Speed Gain	Memory Impact	Complexity	Key Tools / Docs
Mixed Precision & Memory Optimizations	Any model that fits tightly in GPU memory	Uses lower precision (FP16/BF16) and lighter tensors to reduce compute and transfer overhead	1.5 – 2× faster training	30–50% less memory	Low	PyTorch AMP, NVIDIA Apex
Gradient Accumulation & Effective Batch Size	Models limited by GPU memory but needing large batch sizes	Simulates large-batch training by accumulating gradients across smaller batches	Improves convergence stability; indirect speed gain via fewer restarts	Moderate extra memory (temporary gradients)	Low – Medium	DeepSpeed Docs, PyTorch Forum
Smart Offloading & Sharded Training (ZeRO)	Very large models that don’t fit in GPU memory	Shards optimizer states, gradients, and parameters across devices or CPU	10–30% throughput gain; trains 2–4× larger models	Frees up most GPU memory	Medium – High	DeepSpeed ZeRO, Hugging Face Accelerate

Here is some advice on how to choose quickly:

• If you want instant results: Start with mixed precision. It’s stable, simple, and built into every major framework
• If memory limits your batch size: Add gradient accumulation. It’s lightweight and easy to integrate
• If your model still doesn’t fit: Use ZeRO or offloading to shard memory and train bigger models on the same hardware

Wrapping Up

Training speed isn’t just about how many GPUs you have; it’s about how effectively you utilize them. The three methods covered in this article are the most practical and widely adopted ways to train faster without upgrading hardware.
Each of these techniques can deliver real gains on its own, but their true strength lies in combining them. Mixed precision often pairs naturally with gradient accumulation, and ZeRO integrates well with both. Together, they can double your effective speed, improve stability, and extend the life of your hardware setup.

Before applying these methods, always profile and benchmark your training loop. Every model and dataset behaves differently, so measure first, optimize second.

References