sklearn’s FactorAnalysis varimax orthogonal rotation increases correlation of factors

by

Summary

The issue at hand is that Scikit-Learn’s FactorAnalysis with varimax orthogonal rotation is resulting in correlated factors, which contradicts the assumption that the factors should be uncorrelated. This is observed in both the provided example and the user’s own dataset.

Root Cause

The root cause of this issue is due to the following reasons:

  • Varimax rotation is an orthogonal rotation method that aims to simplify the factor structure by maximizing the variance of the squared loadings of a factor on all the variables.
  • Scikit-Learn’s implementation of FactorAnalysis with varimax rotation may not be correctly ensuring that the resulting factors are uncorrelated.
  • Numerical instability may also be a contributing factor, particularly when dealing with large datasets.

Why This Happens in Real Systems

This issue can occur in real systems due to:

  • Insufficient understanding of the underlying assumptions and limitations of FactorAnalysis and varimax rotation.
  • Inadequate data preprocessing, which can lead to numerical instability and correlated factors.
  • Inappropriate choice of parameters, such as the number of components or the rotation method.

Real-World Impact

The real-world impact of this issue includes:

  • Inaccurate results and misinterpretation of the factor analysis output.
  • Overfitting or underfitting of the model, leading to poor performance on new data.
  • Loss of trust in the model and its results, particularly if the correlations between factors are not properly addressed.

Example or Code

import numpy as np
import matplotlib.pyplot as plt
from numpy.random import RandomState
from sklearn import decomposition
from sklearn.datasets import fetch_olivetti_faces

rng = RandomState(0)
faces, _ = fetch_olivetti_faces(return_X_y=True, shuffle=True, random_state=rng)
n_samples, n_features = faces.shape

faces_centered = faces - faces.mean(axis=0)
faces_centered -= faces_centered.mean(axis=1).reshape(n_samples, -1)

n_components = 20
for svd in ["lapack", "randomized"]:
    for rotation in ["varimax", None]:
        fa_estimator = decomposition.FactorAnalysis(
            n_components=n_components, max_iter=20, svd_method=svd, rotation=rotation
        )
        factors = fa_estimator.fit_transform(faces_centered)
        corr = np.corrcoef(factors, rowvar=False)
        corr[np.diag_indices_from(corr)] = np.nan
        plt.imshow(corr)
        plt.colorbar()
        plt.title("SVD method: %s, rotation: %s" % (svd, rotation))
        plt.show()

How Senior Engineers Fix It

Senior engineers can fix this issue by:

  • Carefully reviewing the documentation and implementation of Scikit-Learn’s FactorAnalysis and varimax rotation.
  • Verifying the assumptions and limitations of the model and its parameters.
  • Implementing additional data preprocessing steps, such as normalization or feature scaling, to ensure numerical stability.
  • Selecting alternative rotation methods or models that better suit the specific problem and dataset.

Why Juniors Miss It

Junior engineers may miss this issue due to:

  • Lack of experience with FactorAnalysis and varimax rotation.
  • Insufficient understanding of the underlying assumptions and limitations of the model.
  • Inadequate attention to detail, particularly when it comes to data preprocessing and parameter selection.
  • Overreliance on default parameters and settings, without properly verifying their suitability for the specific problem and dataset.