Reading data more efficiently from a csv file in python

by

Summary

The slowdown came from line‑by‑line CSV parsing and Python‑level loops that repeatedly allocate lists, convert values, and reshape data for every MNIST row. The neural network wasn’t the bottleneck — the data‑loading pipeline was.

Root Cause

The primary root cause was Python‑level iteration over every element in the dataset. This created several expensive operations:

  • Repeated list allocations for every row
  • Per‑element Python loops instead of vectorized NumPy operations
  • Unnecessary dtype inflation (float128 is extremely slow and unnecessary for MNIST)
  • Transformations done row‑by‑row instead of batching
  • No caching or preloading of the dataset

Why This Happens in Real Systems

Real systems often degrade when:

  • Data ingestion is written in a Python loop instead of vectorized operations
  • Developers assume the model is slow, but the I/O pipeline dominates runtime
  • CSV is used instead of binary formats (NumPy .npy, .npz, or PyTorch tensors)
  • Dtypes are chosen without considering memory bandwidth
  • Transformations are done repeatedly instead of once at load time

Real-World Impact

Inefficient data loading causes:

  • Massive training slowdowns (hours lost per epoch)
  • GPU/CPU underutilization because the model waits for data
  • Higher memory pressure from oversized dtypes
  • Inconsistent training throughput due to Python’s GIL and loop overhead

Example or Code (if necessary and relevant)

A vectorized, efficient MNIST CSV loader:

import numpy as np

# Load using float32 (fast, sufficient for ML)
file = np.loadtxt("mnist.csv", delimiter=",", dtype=np.float32)

# Normalize entire array at once
file /= 255.0

# Split labels and images
labels = file[:, 0].astype(int)
images = file[:, 1:].reshape(-1, 784, 1)

A vectorized one‑hot encoder:

one_hot = np.eye(10)[labels].reshape(-1, 10, 1)

How Senior Engineers Fix It

Senior engineers eliminate Python loops and restructure the pipeline:

  • Use vectorized NumPy operations instead of per‑element loops
  • Switch to float32 (industry standard for ML)
  • Load once, preprocess once, reuse many times
  • Convert CSV to .npy or .npz for instant loading
  • Batch transformations instead of doing them inside the training loop
  • Profile the pipeline to confirm the bottleneck is I/O, not the model

Why Juniors Miss It

Juniors often miss this because:

  • They assume the neural network is the slow part, not the data loader
  • They rely on intuitive Python loops instead of vectorized operations
  • They don’t yet recognize that dtype choice affects performance
  • They treat CSV as a normal format for ML, unaware that binary formats are 10–100× faster
  • They rarely profile code, so bottlenecks remain hidden

If you want, I can also generate a version of this postmortem tailored for your internal engineering wiki or convert the example into a reusable data‑loader module.

Leave a Comment