ADR-0001: Dask threaded scheduler for pipeline execution#
Field |
Value |
|---|---|
Kind |
ADR |
Status |
Accepted |
Decided |
historical |
Deciders |
Antoine Berchet |
Supersedes |
— |
Superseded by |
— |
Context#
CIF used to build its pipeline as a home-made dictionary-based tree, walked and executed by hand rather than through a real DAG (directed acyclic graph) engine. That tree became very slow to build as the number of transforms grew — construction cost scaled with the pipeline, not just its execution. A DAG became necessary both to accelerate tree building and to parallelize the pipeline’s embarrassingly parallel tasks, in particular in the GPU context: heavy lifting Fortran kernels run on the GPU while the CPU would otherwise sit idle waiting on them. Dask was chosen as the orchestrator.
With Dask in place, process-based execution (multiprocessing / a worker pool) was tried and turned out to be impractical in CIF’s context: tasks pass xarray / NetCDF-backed objects between stages, and Dask’s process-based schedulers must serialise (pickle) task inputs and outputs to move them across worker processes. Open NetCDF file handles — and related netCDF4 / HDF5 objects — are not reliably picklable, so process-based execution repeatedly broke when such objects crossed task boundaries. Threading, which shares one address space and needs no such serialisation, was the obvious choice.
Decision#
We use Dask’s threaded scheduler for CIF’s pipeline execution. A dask_mode config
option (threads / synchronous / processes) still exists and processes is wired up
with its own worker-logging initialisation, but threads is the scheduler CIF is designed
and validated around; processes remains available rather than actively supported.
Consequences#
Tasks share one process and one address space: no per-task serialisation of large arrays, and no attempt to pickle NetCDF handles.
Parallelism is bounded by the GIL for pure-Python sections. This is acceptable because the heavy work — numpy kernels, NetCDF / HDF5 I/O, GPU-offloaded Fortran — releases the GIL.
Tasks and plugins must be thread-safe and must not rely on process isolation or per-process global state; shared module/global state is visible across concurrent tasks.
True multi-process / distributed scaling would require an explicit serialisation boundary (e.g. pass file paths, not open handles) to work around the pickling issue that made pooling impractical here. That is a one-way-door change and would need a new RFC, not an edit to this ADR.