# ADR-0001: Dask threaded scheduler for pipeline execution

| Field         | Value       |
|---------------|-------------|
| Kind          | ADR         |
| Status        | Accepted    |
| Decided       | historical  |
| Deciders      | Antoine Berchet |
| Supersedes    | —           |
| Superseded by | —           |

## Context

CIF used to build its pipeline as a home-made dictionary-based tree, walked and executed by
hand rather than through a real DAG (directed acyclic graph) engine. That tree became very
slow to build as the number of transforms grew — construction cost scaled with the pipeline,
not just its execution. A DAG became necessary both to accelerate tree building and to
parallelize the pipeline's embarrassingly parallel tasks, in particular in the GPU context:
heavy lifting Fortran kernels run on the GPU while the CPU would otherwise sit idle waiting
on them. Dask was chosen as the orchestrator.

With Dask in place, process-based execution (multiprocessing / a worker pool) was tried and
turned out to be impractical in CIF's context: tasks pass xarray / NetCDF-backed objects
between stages, and Dask's process-based schedulers must serialise (pickle) task inputs and
outputs to move them across worker processes. Open NetCDF file handles — and related netCDF4
/ HDF5 objects — are not reliably picklable, so process-based execution repeatedly broke when
such objects crossed task boundaries. Threading, which shares one address space and needs no
such serialisation, was the obvious choice.

## Decision

We use Dask's **threaded** scheduler for CIF's pipeline execution. A `dask_mode` config
option (`threads` / `synchronous` / `processes`) still exists and `processes` is wired up
with its own worker-logging initialisation, but `threads` is the scheduler CIF is designed
and validated around; `processes` remains available rather than actively supported.

## Consequences

- Tasks share one process and one address space: no per-task serialisation of large arrays,
  and no attempt to pickle NetCDF handles.
- Parallelism is bounded by the GIL for pure-Python sections. This is acceptable because the
  heavy work — numpy kernels, NetCDF / HDF5 I/O, GPU-offloaded Fortran — releases the GIL.
- Tasks and plugins must be **thread-safe** and must **not** rely on process isolation or
  per-process global state; shared module/global state is visible across concurrent tasks.
- True multi-process / distributed scaling would require an explicit serialisation boundary
  (e.g. pass file paths, not open handles) to work around the pickling issue that made pooling
  impractical here. That is a one-way-door change and would need a new **RFC**, not an edit to
  this ADR.