Split Recommender Ratings Before Aggregation to Avoid Leakage

by

Summary

Splitting the rating matrix into training and testing sets is a fundamental step before building recommender models. The goal is to keep the original distribution of users, movies, and ratings while preventing information leakage. This postmortem explains why the original notebook’s data preparation missed a proper split, the consequences, and how senior engineers reliably perform the split.

Root Cause

  • The notebook builds aggregated movie features (movieDict) before any split, mixing raw ratings from the entire dataset.
  • No explicit train/test split is performed on the ratings DataFrame.
  • Consequently, the model would be evaluated on data it has already seen, inflating performance metrics.

Why This Happens in Real Systems

  • Convenient prototyping: developers often explore data and compute aggregates first, then forget to redo them after splitting.
  • Assumption that aggregation is “static” and does not need to respect the split.
  • Lack of a standard pipeline that enforces split‑first, transform‑later semantics.

Real-World Impact

  • Data leakage leads to overly optimistic RMSE/MAE scores.
  • Production models may under‑perform when exposed to truly unseen users or movies.
  • Business decisions based on inflated metrics can cause budget overruns and loss of stakeholder trust.
  • Re‑engineering effort later to redesign the pipeline and re‑train models from scratch.

Example or Code (if necessary and relevant)

from sklearn.model_selection import train_test_split

# 1️⃣ Split the raw ratings into train / test *before* any aggregation
train_ratings, test_ratings = train_test_split(
    ratings,
    test_size=0.2,
    random_state=42,
    stratify=ratings['user_id']   # keep user distribution roughly equal
)

# 2️⃣ Build movie statistics **only** on the training set
movie_props = (
    train_ratings.groupby('movie_id')
    .agg(rating_size=('rating', 'size'), rating_mean=('rating', 'mean'))
    .reset_index()
)

# 3️⃣ Normalise the size column
min_sz, max_sz = movie_props['rating_size'].min(), movie_props['rating_size'].max()
movie_props['norm_size'] = (movie_props['rating_size'] - min_sz) / (max_sz - min_sz)

# 4️⃣ Construct the movie dictionary for *training* only
movie_dict = {}
with open('ml-100k/u.item', encoding="ISO-8859-1") as f:
    for line in f:
        fields = line.rstrip('\n').split('|')
        movie_id = int(fields[0])
        name = fields[1]
        genres = list(map(int, fields[5:25]))
        if movie_id in movie_props['movie_id'].values:
            row = movie_props[movie_props['movie_id'] == movie_id].iloc[0]
            movie_dict[movie_id] = (
                name,
                np.array(genres),
                row['norm_size'],
                row['rating_mean']
            )

How Senior Engineers Fix It

  • Pipeline first: always split raw data before any feature engineering.
  • Use scikit‑learn’s train_test_split (or StratifiedKFold) with a fixed random_state.
  • Keep separate feature stores for training and testing to avoid accidental reuse.
  • Validate the split by checking distribution similarity (e.g., rating histogram, user/movie counts).
  • Automate the workflow with scripts or DAGs so the split step cannot be omitted.
  • Write unit tests that assert no rows from train_ratings appear in test_ratings.

Why Juniors Miss It

  • They view aggregation as a purely exploratory step and forget it influences model inputs.
  • Lack of exposure to data leakage concepts in early coursework.
  • Tendency to work interactively in notebooks, where the order of cells can hide the mistake.
  • Often skip stratification, leading to imbalanced user/movie representation in the test set.

Leave a Comment