Reproducible Randomness in Deep Learning: Why Seeding Matters (With NumPy & PyTorch)

In machine learning and deep learning, we often use random operations like:

• Random weight initialization in neural networks
• Random sampling of mini-batches
• Data shuffling before training

Since these operations involve randomness, we can get slightly different results every time we run the same code. This makes debugging and comparisons difficult!

👉 Solution? Use seeding to control randomness and ensure reproducibility.

What is a Random Seed?

A random seed is a number that initializes the random number generator.

• If we set the same seed every time, we get the same random results.
• Without setting a seed, results will be different every time.

Example Without Seeding (NumPy)

import numpy as np

print(np.random.rand(3))  # Random numbers
print(np.random.rand(3))  # Different random numbers

❌ Every time you run this, you’ll get different outputs.

Example With Seeding (NumPy)

np.random.seed(42)  # Set seed
print(np.random.rand(3))  # Fixed random numbers
np.random.seed(42)  # Reset seed
print(np.random.rand(3))  # Same numbers as before!

✅ Now the random numbers are identical each time.

Why is Reproducibility Important in Deep Learning?

✅ Debugging: Ensures that errors are reproducible.
✅ Experimentation: Allows fair comparisons between model versions.
✅ Consistency: Training with the same settings should produce the same results.

🔹 Setting a Random Seed in PyTorch

In PyTorch, we need to set the seed for multiple random generators (CPU, CUDA, NumPy).

import torch

# Set PyTorch random seed
torch.manual_seed(42)

# Set NumPy seed for consistency
np.random.seed(42)

# Generate some random numbers
print(torch.rand(3))  # Fixed random numbers

✅ This ensures that PyTorch operations will be reproducible.

Ensuring Reproducibility in GPU (CUDA)

If using a GPU, set the CUDA seed as well:

torch.cuda.manual_seed(42)
torch.cuda.manual_seed_all(42)  # If using multiple GPUs
torch.backends.cudnn.deterministic = True  # Ensures deterministic behavior
torch.backends.cudnn.benchmark = False  # Slower but more reproducible

👉 Why?

• GPUs perform operations in parallel, which can introduce non-deterministic behavior.
• Setting these options ensures consistent results.

Example: Training a Model with Seeding**

Let’s train a simple PyTorch model with and without a seed to see the difference.

import torch.nn as nn
import torch.optim as optim

def train_model():
    torch.manual_seed(42)  # Set seed for reproducibility
    np.random.seed(42)  # Set NumPy seed

    model = nn.Linear(2, 1)
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    loss_fn = nn.MSELoss()

    X = torch.randn(10, 2)
    y = torch.randn(10, 1)

    for epoch in range(5):
        optimizer.zero_grad()
        preds = model(X)
        loss = loss_fn(preds, y)
        loss.backward()
        optimizer.step()

        print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

print("Training Run 1:")
train_model()

print("\nTraining Run 2:")
train_model()  # Same loss values since seed is fixed!

✅ Both runs produce identical results because we set the random seed.

Key Takeaways

Conclusion**

• Randomness affects training in deep learning (e.g., random weight initialization, data shuffling).
• Setting a seed ensures results are reproducible across runs.
• In PyTorch, set both CPU & GPU seeds for full control.