Converting unchunked HDF5 to OME-Zarr with Dask
Summary This postmortem analyzes why Dask workers show steadily increasing memory usage during an HDF5 → OME‑Zarr conversion when reading large, unchunked HDF5 slabs and writing many small Zarr chunks. Although the early tasks appear lightweight, later tasks accumulate hidden memory pressure from Python object retention, task graph expansion, filesystem latency, and Dask’s own scheduling … Read more