dd Copies Thin LVM Volumes Misaligns Data%

by

Summary

Raw dd of a thin logical volume copies logical block contents but cannot replicate LVM thin-pool metadata. Because thin provisioning maintains physical-to-logical mappings in a hidden metadata sub-device, copying the visible volume /dev/pool3/vol without its pool metadata causes the destination to reconstruct its own allocation map. The result is a volume that mounts and looks correct to the filesystem layer while reporting a different Data% and diverging from the source at the physical block layer.

Root Cause

  • Thin provisioning splits control from data. Every thin volume is fronted by a pool that stores block-mapping tables and refcounts in a separate metadata LV (typically *_tmeta). dd reads only the virtualized user LV, completely omitting the allocation bitmap that governs Data%.
  • conv=sparse skips zero blocks, suppressing allocation. When dd encounters all-zero logical blocks, it seeks past them on the output side. Thin pools provision physical chunks only on write, so skipped blocks remain unallocated on the destination even if the source thin pool had provisioned them from earlier writes, snapshots, or discarded-but-not-trimmed filesystem blocks. This directly skews Data%.
  • Logical size does not equal physical footprint. A thin volume exposes its nominal size to the guest operating system, but the backing thin pool decides which chunks are physically occupied. Raw block copying recreates neither the pool geometry nor its metadata state.

Why This Happens in Real Systems

  • Engineers often treat block devices as opaque files, assuming dd performs a perfect physical clone.
  • Thin-pool metadata lives on hidden LVs that are absent from standard /dev/<vg>/<name> user workflows, making them invisible to routine device-level operations.
  • Network compression (gzip) is frequently paired with conv=sparse to save bandwidth, amplifying the divergence by eliding zero-filled writes that would otherwise allocate destination chunks.
  • Snapshot discard histories and fstrim behavior vary between hosts, producing source-side provisioned blocks that read as zeros and simply disappear on the destination when sparse conversion is active.

Real-World Impact

  • Capacity accounting mismatch. Destination pools may show significantly different Data% values than their sources, breaking threshold-based alerting and automated expansion logic.
  • Sudden out-of-space events. A destination that under-reports Data% can later explode when live writes allocate chunks for the first time, causing unexpected thin-pool exhaustion and I/O errors.
  • Unreliable DR posture. Block-level copies that omit thin metadata produce bootable but architecturally divergent volumes, invalidating snapshot chains and making future incremental block syncs impossible.
  • Performance inconsistency. Sparse destination volumes incur thin-pool allocation latency on first write, introducing noisy latency not present on the source.

Example or Code (if necessary and relevant)

The original pipeline copies logical content but silently strips thin-pool allocation parity because it neither transfers pool metadata nor guarantees every block reaches the destination.

dd if=/dev/pool3/vol bs=4M conv=sparse,noerror,sync status=progress | gzip | ssh root@172.16.12.183 'gzip -d | dd of=/dev/pool3/vol bs=4M'

To obtain matching Data%, first align the source filesystem so provisioned blocks reflect live data. Then remove conv=sparse and use notrunc on the destination to preserve volume geometry.

fstrim /mnt/vol
dd if=/dev/pool3/vol bs=4M conv=noerror,sync status=progress | gzip | ssh root@172.16.12.183 'gzip -d | dd of=/dev/pool3/vol bs=4M conv=notrunc'

If true block identity is required, you must copy the entire thin pool, including its metadata and data devices, while the pool is offline. For most production migrations, the superior strategy is to avoid raw thin-volume dd entirely and replicate at the filesystem layer.

rsync -aHAX --numeric-ids --inplace --delete /mnt/vol/ /mnt/dest/

How Senior Engineers Fix It

  • Eliminate conv=sparse from thin-LV copies. Sparse conversion conflates logical zeros with physical unallocation. Seniors ensure every logical block is written so the destination thin pool mirrors the source allocation pattern.
  • Pre-trim the source. Before copying, run fstrim on the mounted source filesystem so the source pool’s provisioned blocks match live data. This collapses the delta between logical and physical usage, making the resulting copy deterministic.
  • Replicate the metadata volume when true identity matters. If the destination must be an exact clone (including snapshot relationships and discard histories), seniors deactivate the thin pool and copy both its _tmeta and _tdata devices, or use thin_dump / thin_restore for metadata portability.
  • Prefer filesystem-level or native replication. Instead of raw dd, seniors use rsync, tar, application-level sync, or storage-native sends (for example, ZFS send/recv, or DRBD). These tools copy meaningful data, not abandoned block allocations.

Why Juniors Miss It

  • Block-device abstraction gap. Juniors see /dev/pool3/vol as a simple file-like object and assume dd performs a perfect physical clone. They do not realize that thin LVs are virtualized constructs with out-of-band metadata.
  • Overreliance on copy-paste dd recipes. Common online snippets include conv=sparse for speed; juniors rarely question whether that flag is safe for the underlying storage backend.
  • Failure to distinguish logical and physical usage. df reports 10 GB used, so juniors assume 10 GB was copied. They do not check lvs -o data_percent and therefore never notice the divergence.
  • Not knowing _tmeta exists. Hidden thin-pool metadata devices are absent from day-to-day device listings; juniors simply do not know to account for them.

Leave a Comment