.. _dask_threads:

Dask threads and subprocess count
=================================

Several tasks and subtasks have config options ``daskThreads`` and
``subprocessCount`` used to control threading within a subtask::

  # The number of threads dask is allowed to spawn for each task/subtask.
  # Decrease this number if tasks/subtasks are running out of available threads
  daskThreads = 2

  # The number of subprocesses that each task/subtask gets counted as
  # occupying. Increase this number if tasks/subtasks are running out of
  # memory, so that fewer tasks will be allowed to run at once
  subprocessCount = 1

Dask threads
------------

Dask and xarray support thread-parallel operations on data sets.  They also
support chunk-wise operation on data sets that can't fit in memory.  These
capabilities are very powerful but also difficult to configure for general
cases.  Dask is also not desigend by default with the idea that multiple tasks,
each with multiple dask threads, might operate simultaneously.  As a result,
it is possible to spawn huge numbers of dask threads in MPAS-Analysis that both
slow down analysis and lead to errors when the node runs out of threads
completely.

To prevent this, many tasks or subtasks that use dask threading take the number
of execution threads from a config option, typically in the config section for
the parent task.  Typically, the number of ``daskThreads`` should be around
the same as the number of cores on a node divided by the number of tasks
that will run simultaneiously.  Since the number of running tasks is controlled
by ``subprocessCount``, see below, this number might differ from
``parallelTaskCount``.

Subprocess count
----------------

Tasks or subtasks that use dask threading may consume too much memory or use
too many threads to "count" as a single task.  That is, it might not be safe to
run with ``parallelTaskCount`` simultaneious instances of the task/subtask and
it would be better if it occupied the slot of multiple tasks in the pool of
tasks.  MPAS-Analysis will treat a dask-based task or subtask as occupying
the number of task slots given by the ``subprocessCount`` option.  For example,
if ``parallelTaskCount = 8`` and ``subprocessCount = 2``, up to 4 tasks or
subtasks would be allowed to run simultaneiously.