SLURM_CONF environment variable is not affecting jobs started with sbatch

by

Summary

The SLURM_CONF environment variable is not affecting jobs started with sbatch as expected. The goal is to temporarily modify the weights of nodes in an HPC cluster to control which nodes are used for a specific job. Despite setting the SLURM_CONF environment variable to point to a modified slurm.conf file, the changes are not being applied when submitting a job with sbatch.

Root Cause

The root cause of this issue is a misunderstanding of how the SLURM_CONF environment variable is used by sbatch. The SLURM_CONF environment variable only affects the slurm daemon, not the sbatch command directly. When sbatch is used to submit a job, it uses the existing slurm configuration, which is not updated by changing the SLURM_CONF environment variable.

Why This Happens in Real Systems

This issue occurs in real systems due to the following reasons:

  • The SLURM_CONF environment variable is not propagated to the sbatch command
  • The slurm daemon is not restarted after modifying the SLURM_CONF environment variable
  • The sbatch command uses a cached version of the slurm configuration

Real-World Impact

The real-world impact of this issue is:

  • Inability to dynamically modify node weights for job scheduling
  • Reduced control over node allocation for specific jobs
  • Potential for inefficient use of resources in the HPC cluster

Example or Code

#!/bin/bash
# Set the SLURM_CONF environment variable
export SLURM_CONF=/path/to/modified/slurm.conf

# Submit a job with sbatch
sbatch --job-name=WeightTest --output=output.txt --error=error.txt --nodes=3 test_script.sh

How Senior Engineers Fix It

Senior engineers fix this issue by:

  • Restarting the slurm daemon after modifying the SLURM_CONF environment variable
  • Using the –slurm-conf option with sbatch to specify the modified slurm.conf file
  • Ensuring that the SLURM_CONF environment variable is propagated to the sbatch command

Why Juniors Miss It

Juniors may miss this issue due to:

  • Lack of understanding of how the SLURM_CONF environment variable is used by sbatch
  • Insufficient knowledge of slurm configuration and job scheduling
  • Failure to test and verify the changes to the SLURM_CONF environment variable