How to resolve “No such device or file” error when opening files on a pfs-formatted disk with custom fio’s pfs ioengine (while pfs tools works)?

by

Summary

The failure to open files on a pfs‑formatted disk through a custom fio ioengine occurs because the fio engine is invoking pfs_open() without initializing the PFS environment, without mounting a PFS instance, and without providing a valid PFS work directory. The official pfs CLI tools succeed because they implicitly perform these initialization steps, while fio does not.

Root Cause

The underlying cause is that PFS requires a mounted PFS instance and a valid working directory, but the fio ioengine calls pfs_open() directly on a raw device path such as /dev/nvme0n1/test, which is not a valid PFS path.

Key failures include:

  • No PFS environment initialization (pfs_init(), pfs_mount())
  • No PFS work directory (the log shows: “work_dir is empty”)
  • Using raw block device paths instead of PFS namespace paths
  • Assuming O_CREAT works on PFS devices (it does not unless the PFS namespace is mounted and writable)
  • fio’s working directory may not exist, causing getwd() to fail inside PFS

Why This Happens in Real Systems

Real distributed or userspace filesystems (like PFS, CephFS, Lustre, FUSE-based FS) require:

  • A mounted namespace before any file operations
  • A userspace context (thread-local or global)
  • A valid working directory for relative paths
  • Filesystem-specific open semantics that differ from POSIX

When these assumptions are violated:

  • The filesystem returns ENODEV (No such device) because the path does not belong to a mounted PFS instance.
  • The filesystem returns ENOENT (No such file or directory) because the working directory is not set.

Real-World Impact

When fio bypasses the filesystem’s expected initialization:

  • Benchmarks fail, producing misleading performance results.
  • Writes silently go to the wrong device or fail entirely.
  • PFS metadata becomes inconsistent if the engine attempts partial initialization.
  • Debugging becomes difficult because fio hides the filesystem context.

Example or Code (if necessary and relevant)

A minimal PFS-aware fio engine must:

  • Call pfs_init()
  • Call pfs_mount()
  • Set a valid work directory
  • Use PFS paths (not raw device paths) 
    pfs_init();
pfs_mount("/dev/nvme0n1", "/mnt/pfs0", 0);
chdir("/mnt/pfs0");
int fd = pfs_open("test", O_RDWR | O_CREAT, 0644);

How Senior Engineers Fix It

Senior engineers resolve this by ensuring full PFS lifecycle management inside the ioengine:

  • Initialize PFS before any I/O
    • pfs_init()
    • pfs_mount()
  • Use a valid PFS mountpoint, not /dev/nvme0n1/test
  • Set fio’s working directory to the PFS mountpoint
  • Validate that fio’s --filename is a PFS path, not a block device
  • Ensure the fio job file creates the directory before running
  • Propagate PFS errors correctly so fio reports the real failure

They also add:

  • Logging around mount and open calls
  • Assertions that the PFS environment is active
  • Cleanup logic (pfs_umount(), pfs_fini())

Why Juniors Miss It

Juniors often assume:

  • PFS behaves like a normal filesystem (it does not)
  • pfs_open() works like open() (it requires a mounted namespace)
  • fio will create directories automatically (it won’t)
  • Raw device paths are valid PFS paths (they are not)
  • The PFS CLI tools and the PFS API behave identically (the CLI performs hidden initialization)

They also miss the significance of the log line:

  • “work_dir is empty” → PFS cannot resolve relative paths
  • “No such device” → PFS instance not mounted

If you want, I can generate a corrected fio ioengine implementation that properly initializes and mounts PFS before running I/O.

Leave a Comment