Why Llama 2 7B ARC‑Easy scores vary across evaluation protocols

by

Summary

Llama 2 7B’s ARC‑Easy scores vary because subtle differences in evaluation protocols—prompt formatting, few‑shot selection, tokenization, and metric calculation—lead to measurable swings. Understanding these variables explains why the official report shows ~75 % accuracy while some quantization papers report 69 % or even 53 % normalized accuracy.

Root Cause

  • Prompt template mismatch – whitespace, system messages, or “Answer: ” suffix affect token probabilities.
  • Few‑shot example handling – number, order, and truncation of examples differ across harnesses.
  • Tokenizer version / BPE merge table – quantized models sometimes use a modified tokenizer that changes token ids.
  • Metric definition – acc (raw top‑1) vs. acc_norm (normalized for optional “none of the above”) are not interchangeable.
  • Random seed / shuffling – evaluation scripts may shuffle the test set, producing different sampled subsets.

Why This Happens in Real Systems

  • Production pipelines reuse code that evolves independently; a downstream team may copy a harness without pinning the exact version.
  • Quantization pipelines often replace the original tokenizer to improve speed, unintentionally altering token boundaries.
  • Metric libraries diverge (e.g., lm-evaluation-harness vs. custom scripts) and implement “normalized accuracy” differently.
  • Hardware‑specific optimizations (FP16 vs. INT8) can cause slight numerical differences that affect tie‑breaking in top‑k selections.

Real-World Impact

  • Misleading model comparisons – teams may think a quantized model is dramatically worse when the gap is just a metric artifact.
  • Incorrect budgeting – lower reported accuracy can lead to over‑provisioning of compute resources.
  • Customer trust erosion – published benchmark regressions erode confidence in release notes.
  • Research reproducibility crisis – papers that do not disclose exact evaluation settings become non‑reproducible.

Example or Code (if necessary and relevant)

from lm_eval import evaluator, tasks

# Load exact Llama 2 7B checkpoint and official tokenizer
model = evaluator.load_model("meta-llama/Llama-2-7b-hf")
task = tasks.get_task("arc_easy")

# Use the official prompt template
preds = evaluator.evaluate(
    model,
    task,
    limit=0,
    fewshot=0,
    default_prompt="Question: {question}\nAnswer:",
    bootstrap_iters=1000,
)
print(preds["arc_easy"]["acc"], preds["arc_easy"]["acc_norm"])

How Senior Engineers Fix It

  • Pin the entire evaluation stack (model repo, tokenizer version, lm‑eval‑harness commit).
  • Create a reproducible Docker/Singularity image that includes the exact prompt template file.
  • Automate metric sanity checks: run both acc and acc_norm on a known baseline (e.g., GPT‑Neo) to verify consistency.
  • Document every hyper‑parameter (few‑shot count, seed, truncation length) in the repo’s README and CI logs.
  • Add regression tests that compare current run results against stored “golden” JSON files.

Why Juniors Miss It

  • Assume “accuracy” is universal and ignore subtle metric definitions.
  • Copy‑paste evaluation scripts without reviewing version constraints or prompt strings.
  • Overlook tokenizer dependencies when swapping to a quantized model.
  • Skip deterministic seeding, leading to non‑repeatable results that appear as performance drops.

Leave a Comment